I feel like retrieval approaches, despite how good your ranking mechanisms are, end up biasing LLMs into what a heuristics-based system with way worse understanding of the domain says. I guess is a trade off between speed and quality, but letting the LLM explore without a bias (based on my personal experience) seems to generally provide much better results in terms of quality for the final output.
With the caveat that I am not the average user, my most philosophically-pleasing system would be an LLM which generates and iterates "normal" search parameters. Then the user has the option to view, tweak, and and--just maybe--learn the underlying query language.
The trade-off between speed and quality is everything though. I've found that semantic searching everything that might be relevant and dump the top N into the model is way faster at getting a workable result.
Grep is the fastest tool for many things, but for production RAG it cannot search with meaning, different languages or typos like hybrid vector search can. And it won't scale on terabytes of documents across distributed servers if you want to say under 1000ms.
That's more-or-less what RAG is; a tool to surface relevant data so that AI can make informed decisions. Efficient retrieval is important so that AI doesn't need to ingest the whole corpus for each query.
all true for a single grep invocation, however a capable LLM can drive grep using similar terms (`grep (red|maroon|scarlet|crimson)`, moving the measure of similarity from embedding/indexing time to inference time. There's benefits to that.
but a good combination of prompt, harness and model will easily close that gap. a more advanced (but equivalent) form of grep could also offer fuzzy matching/levenshtein distance function to augment that.
It will, but it does so by burning tokens. I'm sure you've also seen the output of an overly eager grep by an LLM.
I think we're verging on the point where people will care more about the most token-efficient ways to do a task than the lowest human effort version. Humans are expensive, humans using half as many tokens are cheaper.
Retrieval is a whole subject of study. For well-structured plaintext datasets, basic filtering may be as good as semantic and lexical searches. Takes measurement to know what you need
You definitely should use grep wherever you can (i.e. simple, unranked term searches) to 1) exclude things you are definitely not care about 2) to include things you definitely do care about. The reason is that this is cheap and easy and it reduces the overall results you still need to dig through. And because everything else is much more expensive this is going to help reduce the amount of compute needed from that. And results are cacheable as well which helps you with repeat searches for similar things.
For ranked searches, you can do more than one and combine their results. Reciprocal rank fusion (RRF) is a common approach to do that. A simple ranked lexical query and one or more vector searches. Can be combined to amplify results that perform well across multiple of these.
RRF is really easy: for each search result, take its position in each ranked list, add 60, and divide 1 by that number. Add up those values across all lists to get the final score. Results that rank highly in multiple lists get higher scores. The magical number of 60 dampens the effect of highly ranked results biasing the outcome to much. It's not going to be a good approach for all data sets. But it works surprisingly well and it's stupidly easy to implement.
For term search, expending some AI and other cleverness at indexing time to e.g. calculate categorizations is very helpful. If you can reduce your result set with some simple term searches, it might be small enough to pass to an LLM directly and skip ranked searches entirely. Using vector search as magic pixie dust is generally going to work a lot better if you can filter and combine it with a lexical search. Using e.g. a decision model at index time to calculate some categorizations and classifications is relatively cheap and easy relative to calculating vector embeddings.
Apple has a fascinating history of waiting and seeing how a market develops before entering with a better product and eventually dominating, the iPod and iPhone are two examples. It seems they may be building a strong position to pull that off again in the personal assistant AI category ChatGPT is trying to dominate.
I enjoy watching the slow burn approach Apple is deploying as they quietly gear up to offer some % of local inference on iPhones to provide most of the AI services consumers want. Local inference could enable a very low price point for AI services that Apple would leverage alongside their massive ecosystem advantages and privacy claims to compete very effectively for personal assistant AI subscriptions.
Agreed, but I wonder how that plays out without them entering the frontier model space. I just downloaded the new foundation models cli and ran some inference on my Mac, it wasn't great, but it was neat to see. I asked which model it was using and it said gpt 4o mini.
An on-device, 100% private AI assistant seems like it could be so big that it could be a legitimate threat to the frontier model providers they are partnering with right now, imho.
I think it'd be foolish to assume that they aren't already - they have a reputation of doing a lot more internally than ever comes to light. Smaller companies like mistral or thinking machines show that you don't even need that much resources to be semi-competitive, and comparatively Apple is a vertically integrated behemoth.
That said in a pinch they could also acquire / acquihire both of those companies without much thought.
Didn’t Apple sort of already enter this market early-on with their subpar Siri?
I’ve been an iPhone and MacBook Pro user for years, but each time when I set up a new Apple device the first thing I do is say no to anything Siri, and also turning off additional Siri related settings. To the best of my knowledge, Siri is worse than useless, and I haven’t regretted staying away from it.
Just yesterday I saw a talk from Spotify where they presented their own fast retrieval mechanism "Parallel Fusion Router" [1]. It's also in the "learned retrieval" paradigm - in their case they use a teacher LLM to train a much smaller router LLM how to retrieve, using an LLM-as-a-judge based reward function.
From my own experience, these learned methods from Apple, Spotify and others, are highly relevant to hyperscale only - when you are dealing with hundreds of millions of items to choose from (or more), and require super high relevance in the top-5. For all other use cases (up to a few million items) I found that just building a BM25 index, and giving the LLM a BM25 "search tool" and casting a "wide net" wins every single time compared to learned methods (though the latter are more publishable in recsys and icml).
> found that just building a BM25 index, and giving the LLM a BM25 "search tool" and casting a "wide net" wins every single time compared to learned methods
I've been saying this for a while now. Lexical is approximately the same thing as semantic when a frontier LLM is driving the search terms.
> I found that just building a BM25 index, and giving the LLM a BM25 "search tool" and casting a "wide net" wins every single time compared to learned methods
Funny, I did exactly this for my last RAG project. Works great for a small dataset.
Same here. Up to a few millions of documents, this is already great. Combine this with a few other simple techniques and extra metadata during ingestion, and you already have a system that practically useable.
I have tried this by ingesting all of cloudflare's highly technical complex documents (more than 3000 A4 sized pages) and then comparing results with that of cloudflare's own chatbot in their dashboard.
In all cases my system either matched, or performed marginally better than cloudflare's own bot.
I do use a few more techniques though, but, not RDF triplets, and definitely no custom trained neural network.
It seems, that the amount of information that "typically" an person would want to work with, can be handled with pretty much any technique by the compute that most of us have, at a performance level that is satisfying.. It's only when you start to increase the information volume, decrease the compute, or increase the performance expectations, that it is really impactful to start to find methods that really make the most of the deep structures of that information to query it. Complexity of the original information and what accuracy means for your use case also comes into play I suppose.
I've been experimenting with hybrid memory retrieval using SQLite FTS5/BM25 for lexical search and dense embeddings from Nomic Embed Text v1.5 (768D), all-MiniLM-L6-v2 (384D), and Qwen3-Embedding (0.6B), with NumPy cosine similarity, Matryoshka-style dimensional truncation (64D → 768D reranking), and optional graph-based candidate expansion. In one 40-query benchmark, combining retrieval channels improved candidate recall from 30/40 with compact vectors alone to 40/40, although Top-1 accuracy was only 33/40. Another metric I was looking at was number of final candidates found to filter. If its very high, you can get better accuracy, but at the expense of resource use, and it can also lead to worse accuracy if your tying to choose top result efficently.. it makes a needle in a needle stack.
I'm not sold; that's not to say I can't be swayed, but I've spent a lot of time building RAG solutions as a "one-size-fits-all" against codebases and I am not convinced embedding models or any of the myriad BERT implementations work well enough that they can beat out grep.
Most embedding/BERT models are far too small to truly grasp the nettles when an AI ask questions to them; and if you condition the AI to only search for really basic things, then... you're basically back to grep + regex.
You need to prune all manner of useless detritus that might gum up your matches like test files, documentation, and sidecars that semantically overlap or you're just going to match your tests. That's fine if you're building a document store for a business, but if you want a "one-and-done" solution for your codebase it's rarely good enough.
And if you do use a very large embedding model (which means it'll require running in the cloud as most company computers won't run it well), then... well, you're better off just using a language model as a subagent to scout ahead for you and elide the embedding model to begin with.
comandillos | 18 hours ago
WhitneyLand | 10 hours ago
RAG can take the intelligence of an LLM and dumb it down to old school Google search results.
Terr_ | 8 hours ago
refactor_master | 8 hours ago
brrrrrm | 18 hours ago
"why not use grep?"
goldenarm | 18 hours ago
dv_dt | 18 hours ago
aaroninsf | 18 hours ago
With a very large dataset of totally unnormalized data, semantic folding is a silver bullet.
nostrebored | 18 hours ago
itishappy | 17 hours ago
ozkatz | 17 hours ago
jdthedisciple | 17 hours ago
still doesn't include typo'ed variants for any of these
ozkatz | 15 hours ago
c-hendricks | 15 hours ago
everforward | 11 hours ago
I think we're verging on the point where people will care more about the most token-efficient ways to do a task than the lowest human effort version. Humans are expensive, humans using half as many tokens are cheaper.
mejutoco | 17 hours ago
elpakal | 17 hours ago
esafak | 7 hours ago
0o777 | 17 hours ago
https://github.com/lightonai/next-plaid/tree/main/colgrep
Foobar8568 | 4 hours ago
evantbyrne | 13 hours ago
eluru | 11 hours ago
jillesvangurp | 4 hours ago
For ranked searches, you can do more than one and combine their results. Reciprocal rank fusion (RRF) is a common approach to do that. A simple ranked lexical query and one or more vector searches. Can be combined to amplify results that perform well across multiple of these.
RRF is really easy: for each search result, take its position in each ranked list, add 60, and divide 1 by that number. Add up those values across all lists to get the final score. Results that rank highly in multiple lists get higher scores. The magical number of 60 dampens the effect of highly ranked results biasing the outcome to much. It's not going to be a good approach for all data sets. But it works surprisingly well and it's stupidly easy to implement.
For term search, expending some AI and other cleverness at indexing time to e.g. calculate categorizations is very helpful. If you can reduce your result set with some simple term searches, it might be small enough to pass to an LLM directly and skip ranked searches entirely. Using vector search as magic pixie dust is generally going to work a lot better if you can filter and combine it with a lexical search. Using e.g. a decision model at index time to calculate some categorizations and classifications is relatively cheap and easy relative to calculating vector embeddings.
lavendermon | 17 hours ago
I enjoy watching the slow burn approach Apple is deploying as they quietly gear up to offer some % of local inference on iPhones to provide most of the AI services consumers want. Local inference could enable a very low price point for AI services that Apple would leverage alongside their massive ecosystem advantages and privacy claims to compete very effectively for personal assistant AI subscriptions.
elpakal | 17 hours ago
An on-device, 100% private AI assistant seems like it could be so big that it could be a legitimate threat to the frontier model providers they are partnering with right now, imho.
andriy_koval | 17 hours ago
bet could be that this space will be commoditized: many good enough open weight models and multiple competing provides.
makmanalp | 17 hours ago
That said in a pinch they could also acquire / acquihire both of those companies without much thought.
QuantumNomad_ | 16 hours ago
Didn’t Apple sort of already enter this market early-on with their subpar Siri?
I’ve been an iPhone and MacBook Pro user for years, but each time when I set up a new Apple device the first thing I do is say no to anything Siri, and also turning off additional Siri related settings. To the best of my knowledge, Siri is worse than useless, and I haven’t regretted staying away from it.
joshheitzman | 11 hours ago
pants2 | 17 hours ago
armcat | 17 hours ago
From my own experience, these learned methods from Apple, Spotify and others, are highly relevant to hyperscale only - when you are dealing with hundreds of millions of items to choose from (or more), and require super high relevance in the top-5. For all other use cases (up to a few million items) I found that just building a BM25 index, and giving the LLM a BM25 "search tool" and casting a "wide net" wins every single time compared to learned methods (though the latter are more publishable in recsys and icml).
[1] https://research.atspotify.com/2025/9/you-say-search-i-say-r...
bob1029 | 12 hours ago
I've been saying this for a while now. Lexical is approximately the same thing as semantic when a frontier LLM is driving the search terms.
fcarraldo | 11 hours ago
Funny, I did exactly this for my last RAG project. Works great for a small dataset.
greesil | 8 hours ago
freakynit | 8 hours ago
I have tried this by ingesting all of cloudflare's highly technical complex documents (more than 3000 A4 sized pages) and then comparing results with that of cloudflare's own chatbot in their dashboard.
In all cases my system either matched, or performed marginally better than cloudflare's own bot.
I do use a few more techniques though, but, not RDF triplets, and definitely no custom trained neural network.
totetsu | 8 hours ago
I've been experimenting with hybrid memory retrieval using SQLite FTS5/BM25 for lexical search and dense embeddings from Nomic Embed Text v1.5 (768D), all-MiniLM-L6-v2 (384D), and Qwen3-Embedding (0.6B), with NumPy cosine similarity, Matryoshka-style dimensional truncation (64D → 768D reranking), and optional graph-based candidate expansion. In one 40-query benchmark, combining retrieval channels improved candidate recall from 30/40 with compact vectors alone to 40/40, although Top-1 accuracy was only 33/40. Another metric I was looking at was number of final candidates found to filter. If its very high, you can get better accuracy, but at the expense of resource use, and it can also lead to worse accuracy if your tying to choose top result efficently.. it makes a needle in a needle stack.
mickeyp | 17 hours ago
Most embedding/BERT models are far too small to truly grasp the nettles when an AI ask questions to them; and if you condition the AI to only search for really basic things, then... you're basically back to grep + regex.
You need to prune all manner of useless detritus that might gum up your matches like test files, documentation, and sidecars that semantically overlap or you're just going to match your tests. That's fine if you're building a document store for a business, but if you want a "one-and-done" solution for your codebase it's rarely good enough.
And if you do use a very large embedding model (which means it'll require running in the cloud as most company computers won't run it well), then... well, you're better off just using a language model as a subagent to scout ahead for you and elide the embedding model to begin with.
Foobar8568 | 6 hours ago