The AI Developer Stack // Complete Landscape

Cost sensitive? Use Llama 4 or Mistral locally via Ollama (free) · or GPT-4o-mini / Claude Haiku for cheap API calls
Best quality? Claude 4 Opus or GPT-4o or o3 for complex tasks
Data privacy required? Self-host Llama 4 or Mistral — your data never leaves your servers
Long documents? Claude (200K+ context) or Gemini (1M+ context)
Multimodal (images/video/audio)? Gemini 2.0 or GPT-4o
Fine-tuning on your data? Open-weight models: Llama 4, Mistral, Qwen

Layer 2 SDKs are provider-specific — if you switch from OpenAI to Anthropic you rewrite your code. Layer 3 (model abstraction) gives you a unified interface so you can swap providers with one config change. Start with a layer 2 SDK, upgrade to layer 3 when you need flexibility.

Use layer 2 SDK if: you're prototyping, one provider is enough, minimal deps matter.
Use layer 3 abstraction if: you want to A/B test models, need failover, manage costs across providers, or your team uses multiple models.

1. Index time: Load documents → chunk text → embed with model → store vectors in DB (layer 6)
2. Query time: Embed user question → search DB for similar chunks → inject chunks into prompt → LLM answers grounded in your data

Result: The LLM "knows" your private documents without fine-tuning. Cost: ~$0.001 per query.

LangChain = better for complex chains, agent logic, wide integrations, and general LLM apps.
LlamaIndex = better for pure data indexing & retrieval tasks — it has richer document parsing, query routing, and evaluation tools.
Many production apps use both together.

Orchestration (layer 4) = predetermined steps. You define the flow. LLM fills in the blanks.
Example: always retrieve → always summarize → always format

Agents (layer 5) = the LLM decides the steps. It can loop, branch, retry, call different tools.
Example: "research this topic" → agent decides to search, then read papers, then write summary, then fact-check

1. Embed: Convert text → vector: "The cat sat on the mat" → [0.23, -0.11, 0.87, ...] (1536 numbers)
2. Store: Save vectors + original text in DB
3. Search: Embed query → find closest vectors (cosine similarity) → return original texts

Why it's powerful: "automobile" and "car" are near each other in vector space, even though the words differ. Traditional keyword search would miss this.

Function calling = defined per-request in your code, works with one model's API, not shareable.
MCP = define tools once in a server, any compatible client/AI can use them, reusable across teams and apps.

Think: function calling is a local script, MCP is a published API.

Latency — p50/p95/p99 per model and endpoint
Cost — token usage, spend per user/feature/day
Quality — human ratings, automated eval scores, thumbs up/down
Errors — rate limit hits, model refusals, timeouts
Prompts — version control which prompt produced which output

Langfuse = open-source (self-host = full data ownership), works with any framework/SDK, prompt management is a first-class feature, free tier generous.
LangSmith = better LangChain integration (automatic tracing), stronger dataset tooling, SaaS only.
Choose Langfuse if data privacy matters or you're not on LangChain.

Vercel AI SDK → production web app, React, TypeScript, streaming UI required
Streamlit / Gradio → internal tool, demo, data app, Python team, ship in hours
FastAPI + custom → full product, unique UX, need complete control
Flowise / Dify → non-engineer building an app, or rapid prototype without code
n8n → automating workflows that include AI steps (not primarily a chat UI)