instructor-ai/instructor
120 source files scanned · 48 LLM call site(s) · ~0% of LLM spend looks recoverable (static heuristics · repo looks like a library)
Library/SDK layout detected (Python package "instructor" is an LLM SDK/framework): call sites are most likely pass-through wrappers where the caller chooses the model and prompt, so findings are downgraded one level and the estimate is discounted.
$4.8per 10k runs today (gpt-4o-mini, ~2000 prompt tokens)
$4.8per 10k runs with fixes applied
[](https://tokenzero.sh/audit/instructor-ai/instructor)
Models referenced
gpt-3.5-turbo ×20gpt-4o-mini ×12gpt-4-0613 ×3claude-3-haiku-20240307 ×2gpt-4-vision-preview ×2gpt-4 ×2gpt-4o-audio-preview ×1gpt-4-1106-preview ×1
Findings
| Severity | Where | Finding | Est. |
|---|---|---|---|
| low | examples/openai-audio/run.py:18 | gpt-4o used for a task whose prompt reads like extraction/classification Full-size pricing ($2.5/M in, $10/M out) on a constrained task. gpt-4o-mini costs ~94% less per call at this shape. Fix: Opt into TokenZero cascade routing (`X-TokenZero-Cascade: auto`) so this call is routed to gpt-4o-mini only when the classifier is confident (an opt-in shadow parity monitor is coming; nothing is promised), or switch the model here behind an eval. | 94% |
| info | examples/asyncio-benchmarks/run.py:369 | LLM call inside a loop Per-item calls repeat the full prompt prefix for every element. If items are independent and latency-tolerant, the provider Batch API is 50% cheaper, and OpenAI Flex processing (`service_tier: "flex"`) gives the same 50% on ordinary calls without restructuring; if the prefix is shared, caching pays off immediately. Fix: Batch where possible, or set `service_tier: "flex"` on OpenAI calls that can wait; otherwise keep the shared prefix cache-formatted (TokenZero does this automatically). | 25% |
| info | examples/chain-of-density/chain_of_density.py:118 | LLM call inside a loop Per-item calls repeat the full prompt prefix for every element. If items are independent and latency-tolerant, the provider Batch API is 50% cheaper, and OpenAI Flex processing (`service_tier: "flex"`) gives the same 50% on ordinary calls without restructuring; if the prefix is shared, caching pays off immediately. Fix: Batch where possible, or set `service_tier: "flex"` on OpenAI calls that can wait; otherwise keep the shared prefix cache-formatted (TokenZero does this automatically). | 25% |
| info | examples/classification/classifiy_with_validation.py:52 | Repeat-prone task with no response-cache opt-in Classification and extraction calls often see identical inputs. Without a determinism signal no proxy can safely serve repeats from cache. Fix: If identical inputs should get identical answers, send `X-TokenZero-Cache: always` on these calls (or `temperature: 0` / a `seed` on models that still accept them). Repeats are then served in <10 ms at $0. | 10% |
| info | examples/validators/llm_validator.py:20 | Unrecognised model id "gpt-3.5-turbo" Not in the current pricing table: it may be retired or renamed. Retired models return errors or are silently remapped by some providers. Fix: Check the provider model list; TokenZero flags unpriced models in `X-TokenZero-Warning`. | 0% |
| info | examples/anthropic/run.py:20 | Unrecognised model id "claude-3-haiku-20240307" Not in the current pricing table: it may be retired or renamed. Retired models return errors or are silently remapped by some providers. Fix: Check the provider model list; TokenZero flags unpriced models in `X-TokenZero-Warning`. | 0% |
| info | examples/vision/image_to_ad_copy.py:118 | Unrecognised model id "gpt-4-vision-preview" Not in the current pricing table: it may be retired or renamed. Retired models return errors or are silently remapped by some providers. Fix: Check the provider model list; TokenZero flags unpriced models in `X-TokenZero-Warning`. | 0% |
| info | examples/vision/image_to_ad_copy.py:148 | Unrecognised model id "gpt-4-1106-preview" Not in the current pricing table: it may be retired or renamed. Retired models return errors or are silently remapped by some providers. Fix: Check the provider model list; TokenZero flags unpriced models in `X-TokenZero-Warning`. | 0% |
| info | examples/query_planner_execution/query_planner_execution.py:134 | Unrecognised model id "gpt-4" Not in the current pricing table: it may be retired or renamed. Retired models return errors or are silently remapped by some providers. Fix: Check the provider model list; TokenZero flags unpriced models in `X-TokenZero-Warning`. | 0% |
| info | examples/task_planner/task_planner_topological_sort.py:159 | Unrecognised model id "gpt-4-0613" Not in the current pricing table: it may be retired or renamed. Retired models return errors or are silently remapped by some providers. Fix: Check the provider model list; TokenZero flags unpriced models in `X-TokenZero-Warning`. | 0% |
Fix all of it with one line
OPENAI_BASE_URL=https://tokenzero.sh/v1 ANTHROPIC_BASE_URL=https://tokenzero.sh
Keys pass through untouched. Cache formatting and exact caching are on by default; cascade routing stays opt-in. How it works