Why query understanding matters so much
Instacart's search box has to understand messy, shorthand queries like "bread no gluten" or "x large zip lock" and still return the right products. The team treats this layer as an "intent engine": if it misreads what you meant, every downstream system suffers. Traditional machine learning models handled common, well–structured searches, but they struggled with sparse data, vague queries, and the long tail of highly specific requests.
Over time, Instacart accumulated separate models for things like query classification and query rewrites. Each had its own data pipeline and infra, making the overall system complex and slow to evolve.
Why LLMs are a better backbone
Large language models offer something the legacy stack didn't: broad world knowledge and strong reasoning. A model that already knows the relationship between "Italian parsley", "flat parsley" and "curly parsley" needs far less bespoke feature engineering to perform well on grocery search.
Instacart's strategy is to use LLMs as a unified backbone instead of a collection of narrow models. They focus on enriching the model with Instacart–specific context and then compressing that knowledge into smaller, efficient models for real-time use.
The three–layer strategy for LLM–powered intent
1. Context-engineering with RAG. Data pipelines retrieve Instacart-specific information (conversion history, catalog details, taxonomies) and inject it into prompts. This grounds the LLM's answers in live business reality instead of generic web knowledge.
2. Post-processing guardrails. Validation layers check that the model's outputs align with Instacart's product taxonomy and filters out obvious hallucinations or off-topic suggestions.
3. Fine-tuning for deep expertise. For the hardest problems, they fine-tune smaller open-source models on proprietary data, baking domain knowledge directly into the weights.
Key lessons for enterprise LLM systems
• Context is the real moat. General-purpose LLMs are becoming commodities; proprietary context — engagement data, catalog metadata, operational constraints — is where defensibility lives.
• Start offline, then move to real-time. Prove value and generate high-quality labels with offline pipelines before investing in low-latency inference.
• Simplify the stack. A single LLM backbone can often replace a zoo of task-specific models, reducing maintenance overhead.
• Model quality isn't enough. You also need caching, autoscaling, latency tuning, and guardrails before an LLM system delivers consistent value in production.
Large language models offer something the legacy stack didn't: broad world knowledge and strong reasoning. A model that already knows the relationship between "Italian parsley", "flat parsley" and "curly parsley" needs far less bespoke feature engineering to perform well on grocery search.
Instacart's strategy is to use LLMs as a unified backbone instead of a collection of narrow models. They focus on enriching the model with Instacart–specific context and then compressing that knowledge into smaller, efficient models for real-time use.
The three–layer strategy for LLM–powered intent
1. Context-engineering with RAG. Data pipelines retrieve Instacart-specific information (conversion history, catalog details, taxonomies) and inject it into prompts. This grounds the LLM's answers in live business reality instead of generic web knowledge.
2. Post-processing guardrails. Validation layers check that the model's outputs align with Instacart's product taxonomy and filters out obvious hallucinations or off-topic suggestions.
3. Fine-tuning for deep expertise. For the hardest problems, they fine-tune smaller open-source models on proprietary data, baking domain knowledge directly into the weights.
Key lessons for enterprise LLM systems
• Context is the real moat. General-purpose LLMs are becoming commodities; proprietary context — engagement data, catalog metadata, operational constraints — is where defensibility lives.
• Start offline, then move to real-time. Prove value and generate high-quality labels with offline pipelines before investing in low-latency inference.
• Simplify the stack. A single LLM backbone can often replace a zoo of task-specific models, reducing maintenance overhead.
• Model quality isn't enough. You also need caching, autoscaling, latency tuning, and guardrails before an LLM system delivers consistent value in production.
• Context is the real moat. General-purpose LLMs are becoming commodities; proprietary context — engagement data, catalog metadata, operational constraints — is where defensibility lives.
• Start offline, then move to real-time. Prove value and generate high-quality labels with offline pipelines before investing in low-latency inference.
• Simplify the stack. A single LLM backbone can often replace a zoo of task-specific models, reducing maintenance overhead.
• Model quality isn't enough. You also need caching, autoscaling, latency tuning, and guardrails before an LLM system delivers consistent value in production.