Why pipeline costs are different
Production LLM systems rarely make one model call. They retrieve context, rerank, classify, call tools, critique answers, retry failed branches, and sometimes fan out across multiple agents. A single-call calculator hides the architecture that actually drives the bill.
Why pipeline costs are different
Pipeline cost grows with node count, calls per request, routing probability, retries, and cache effectiveness. A cheap model can become expensive when called eight times per user action. A more expensive model can be cheaper when it replaces several weaker critique or repair calls. Model choice is only one layer of the cost model.
Optimisations and their real savings
Caching, batching, smaller prompts, router gates, and cheaper critique models can all help, but only if they target the top cost driver. Prompt caching helps repeated long prefixes. Batch APIs help delayed workloads. Routing helps when expensive steps are needed for only a fraction of requests. The tool is designed to make those tradeoffs visible.
Assumptions and methodology
Pipeline cost is modelled as per-node token cost multiplied by calls per request, monthly volume, retry overhead, and cache or batch discounts.
The page keeps the interaction fast, accessible, and dependency-light while preserving the important planning behaviour from the implementation plan: editable inputs, visible outputs, no signup gate, structured data, internal links, citations, and clear caveats. Heavy runtime features such as Monaco, solc, exact tokenizers, or PDF export should remain code-split when added so the public page stays fast.
Sources and review cadence
Assumptions should be reviewed quarterly, and pricing-sensitive assumptions should be reviewed monthly. The public data files in the repo include last-reviewed or last-verified dates where the plan calls for them.
Need the architecture behind the number?
Use this tool to narrow the conversation, then bring the scenario into an architecture review if the decision affects budget, security, compliance, or production reliability.
Book a strategy callFrequently asked questions
How is pipeline cost different from token cost?
Token cost prices one model call. Pipeline cost prices the full chain: classifiers, retrieval prompts, generation, critique, retries, and parallel branches. The difference matters because a request that appears to use one answer can quietly trigger several model calls behind the scenes.
Why does fan-out matter so much?
Fan-out multiplies work. If one user request launches five parallel research calls and each call uses retrieval and summarization, cost scales with every branch. Even when latency improves, the bill rises unless routing or caching prevents unnecessary branches.
Does prompt caching really save 90 percent?
It can save a large share of repeated prefix cost for providers that support prompt caching, but only when the cached context is stable and reused. It does not reduce unique output tokens, tool traces, or one-off prompts. Real savings depend on cache hit rate.
When does Batch API help?
Batch pricing helps delayed, non-interactive workloads such as offline enrichment, evaluation runs, and nightly classification. It usually does not help chat, support, or agent workflows where users expect an immediate answer.
How do I estimate retry rate?
Start from logs if the pipeline exists. If not, model a conservative 5-15 percent retry rate for early production and raise it for brittle tools, strict schemas, or external APIs. Retries should include validation failures and repair loops, not just HTTP errors.
Can I model conditional branching?
Use calls-per-request and conditional probability together. A branch that runs 20 percent of the time with three calls has an expected multiplier of 0.6 calls per request. That expected-cost framing is usually enough for planning.