Key takeaways
- Top models cluster closely and benchmarks saturate in months, so public rankings only shortlist. Your own task eval is the asset that survives a model change.
- Stated notice before retirement runs from about 2 weeks (previews, one Foundry host) to 6 months, and safety or compliance cases can be shorter. Platform dates can differ from the maker's.
- Pick the cheapest tier and effort setting that passes your eval, and re-test quarterly: Epoch measured price falls of 9x to 900x a year for fixed capability, though the fastest may not persist.
- Treat routing as a hypothesis. It must beat the single model's whole cost-quality curve, including that model at lower effort, and needs a reliable quality signal.
- Pull levers cheapest first: prompt, effort, retrieval, tier, cascade, then fine-tuning, which also ties you to one base model's lifecycle.
- Open weights are not an automatic exit: hosted open-weight versions retire too, licences carry conditions, and residency controls vary by model generation.
Every model you choose this year has a retirement date, and it is usually closer than your next budget cycle. One vendor notified developers on 30 September 2026 that a model retires on 30 November; another model on the same page was notified on 5 June and retired on 5 August [1]. The evidence used to justify a choice is just as unstable. Stanford's 2026 AI Index reports four companies clustered within 25 Elo points on the Arena leaderboard as of March 2026 [2].
Our view at NKK is that model selection should stop being a decision and become a procedure. A version name is a moving target and a public ranking is a weak proxy, so the assets worth owning are three things that outlive any model: an eval built from your own tasks, a requirements envelope written per workload, and a cheap path to swap. This guide builds all three. Tiers, prices and dates below come from vendor documentation read on 13 October 2026 and will change; nothing here ranks models.
Why version names and public rankings are a weak foundation
At the top of the market, models are close on average, and the average is the wrong thing to buy on. The gap between the top two models fell from 4.9% in 2023 to 0.7% in 2024 [6]. Stanford describes the consequence as "shifting competitive pressure toward cost, reliability, and domain-specific performance" [2].
The instruments are fragile. SWE-bench Verified performance went "from 60% to near 100% in a single year" [7], and Stanford notes that evaluations "intended to be challenging for years are saturated in months" [2]. An estimated 6.49% of MMLU questions contain errors [8]. Benchmark data leaking into training sets makes results unreliable [9]. The Arena measures crowd preference in pairwise votes [10], and its critics report undisclosed private testing that benefits a handful of providers [11].
Even a clean score may not transfer. Meaning-preserving changes to prompt format shifted accuracy by up to 76 points on one open model, and format performance "only weakly correlates between models" [12]. A ranking measured under someone else's prompt is not a ranking under yours. Nor does a name guarantee behaviour: in 2023, researchers comparing the March and June versions of GPT-3.5 and GPT-4 found that their performance and behaviour "can vary greatly over time" [13]. Vendors now pin weights to model IDs, yet document that the serving infrastructure around them can change [14].
Even the most quoted structural claim moves. Stanford's two editions give four readings of the closed-over-open lead that do not form a trend [6][2]. Do not base a licensing strategy on one reading.
- Jan 20248.04%
- Aug 20240.5%
- Feb 20251.7%
- Mar 20263.3%
| Criterion | What it tells you | What it does not tell you |
|---|---|---|
| Aggregate benchmark score | A rough capability band; good for shortlisting and for ruling options out | Whether the benchmark is saturated, contaminated or mislabelled, or how the model handles your prompt format |
| Arena-style preference rank | What crowd voters preferred in pairwise comparisons | Fitness for your task, or whether private testing shaped the position |
| Vendor tier, price and latency label | Relative cost, context window and speed class on a given date | Your p95 latency, effective context, or next quarter's price |
| Your task eval | Pass rate, cost and consistency on your actual work | Anything outside the cases included; refresh it as traffic changes |
- What it tells you
- A rough capability band; good for shortlisting and for ruling options out
- What it does not tell you
- Whether the benchmark is saturated, contaminated or mislabelled, or how the model handles your prompt format
- What it tells you
- What crowd voters preferred in pairwise comparisons
- What it does not tell you
- Fitness for your task, or whether private testing shaped the position
- What it tells you
- Relative cost, context window and speed class on a given date
- What it does not tell you
- Your p95 latency, effective context, or next quarter's price
- What it tells you
- Pass rate, cost and consistency on your actual work
- What it does not tell you
- Anything outside the cases included; refresh it as traffic changes
Think in tiers and roles, not names
Microsoft's vocabulary separates a model family (a generation), a version (a dated release) and a variant (a size or capability tier within a family) [17]. Decide on variant first and version last. One vendor's table shows a ladder of flagship, high-capability, balanced and fastest-cheapest tiers [15]. On the date we read it, list prices ran from $0.10 (a starting price for the cheapest tier) to $10 per million input tokens and from $0.50 to $50 per million output tokens, roughly a hundredfold spread [15]. Fees already differed by two orders of magnitude in 2023 [18]. With a spread like that, the cheapest tier that passes your eval is the default, and the burden of proof sits with anything more expensive.
A workload is rarely one call. NKK's view is that it helps to name each call's role: planner, worker, router or grader. A router must be fast and cheap, a grader consistent, and a planner can often afford latency.
The NKK Model Selection Card puts this on one page per workload. It lists the role, the eval and pass mark, the envelope (latency, unit cost, context, residency and retention, licence needs), the lowest tier that passes, two qualified candidates from different vendors or platforms, the date the evidence was gathered, and the triggers that force a re-decision. It builds on vendor selection guidance [19], Microsoft's lifecycle policy [17], multi-metric evaluation [20] and OWASP's supply-chain advice [21].
- WhyRole
The job this call performs: planner, worker, router or grader.
- EvidenceEval and pass mark
The task suite, grader type and threshold; pass^k where users see every run. Record when it last ran, the pinned ID and the platform's retirement date.
- ConstraintsEnvelope
p95 latency, unit cost ceiling, context actually needed, residency and retention, licence and weight-access needs.
- DecisionTier and effort
The lowest capability tier and effort setting that passes the eval.
- OptionsCandidates
Two qualified options from different vendors or platforms, each with its own tuned prompt.
- ReviewExit triggers
Retirement notice, price or residency change, or regression on the eval.
Write the card before you shortlist, so the envelope screens candidates. Where it breaks: a card goes stale if nobody owns the quarterly re-test, and thresholds set before real traffic exists can be wrong.
Build the task eval before you compare anything
Anthropic's guidance is blunt: "having a good evaluation set is the most important step in the process" [19]. Its engineers suggest "20-50 simple tasks drawn from real failures is a great start" [16]. OpenAI describes evals as "an essential component to building reliable applications" and says creating one "begins by describing a task to be done by a model" [22]. Microsoft tells customers to compare quality, latency and cost on their own application and data rather than on public benchmarks alone [17]. OWASP goes further: apply red teaming and evaluations when selecting a third-party model, because models can be fine-tuned to bypass published benchmarks [21].
In NKK's view, start where grading is cheapest: structured tasks with checkable outputs.
Build two suites. Capability evals start with a low pass rate and measure what a better model could unlock; regression evals should hold a pass rate near 100% [16]. Use code, model or human graders, and calibrate any LLM judge against human experts [16]. Where users see every run, report pass^k, the probability that all k trials succeed, rather than pass@k [16]. Carry more than accuracy: HELM's multi-metric approach, measuring seven metrics on 16 core scenarios within a 42-scenario evaluation, exists because single numbers hide trade-offs [20].
Tune the prompt for each candidate before scoring. OpenAI notes that different model types "might need to be prompted differently" [23], and given the format sensitivity above [12], an untuned challenger will lose for reasons that have nothing to do with capability. Keep the suite in your own repository: OpenAI's hosted Evals dashboard and API are scheduled to shut down on 30 November 2026 [4].
- Step 1: Harvest real cases
Pull 20 to 50 tasks from real failures and typical requests.
- Step 2: Define pass criteria
Choose code, model or human graders per task; calibrate model graders against humans.
- Step 3: Tune per candidate
Give each candidate its own best prompt, or you measure prompt fit, not capability.
- Step 4: Score the envelope
Record accuracy, p95 latency, unit cost and consistency for every candidate.
- Step 5: Freeze regression
Keep passing cases as the near-100% suite; keep hard cases as the capability suite.
- Step 6: Re-run on change
Model swap, prompt edit, retirement notice or quarterly date: same suite, same report.
Spend the cost, latency and context budget on your own workload
Anthropic's selection guide says "Tuning effort is often a better lever than switching models" [19], and offers two starting points: efficiency-first, upgrading "only if necessary for specific capability gaps", or capability-first, starting strongest and optimising down [19].
Prices for a fixed capability fall quickly. Epoch AI measured declines of 9x to 900x per year depending on the milestone, with GPT-4-level performance on PhD-level science questions falling about 40x per year, and cautioned that the fastest drops were recent and may not persist [5]. That is the price of reaching a given level, not of the best model; it argues for re-testing cheaper tiers every quarter, not for expecting the flagship to cost less.
| Lever | What it changes | What to measure | Evidence |
|---|---|---|---|
| Effort setting | Intelligence versus latency and cost inside one model | Eval pass rate at each setting | Vendor guidance [19] |
| Tier | Price, speed and ceiling | Cheapest tier that passes | List-price spread [15] |
| Batch processing | 50% discount for non-interactive work | Whether the delay is acceptable | Vendor pricing note [15] |
| Context length | Cost, and accuracy on long inputs | Your longest realistic document | Academic tests [24][25] |
Context needs scepticism. RULER found that of 17 models claiming 32K tokens or more, "only half of them can maintain satisfactory performance at the length of 32K" [25], and accuracy "significantly degrades when models must access relevant information in the middle of long contexts" [24]. Both studies tested 2023 and 2024 models; we found no independent study of current million-token models, so test your own longest input with the answer buried mid-document. Our only latency evidence is vendor labels [15], so measure p95 yourself, in your region, on your prompts.
Open weights, closed APIs, residency and vendor risk
"Open" is overloaded. The Open Source Initiative's definition requires sufficiently detailed information about training data, the code, and the parameters, plus freedoms to use, study, modify and share [26]. Weights alone do not meet it. Meta's Llama 4 licence, dated April 2025, is a typical open-weight alternative: licensees above 700 million monthly users must ask Meta, which "may grant to you in its sole discretion", and derivative models must carry the Llama name [27]. Stanford's pages do not define "open" or "closed" for this comparison, so treat the label with caution [2][6]. On the closed side, Stanford reports that "the most capable models are now the least transparent" [28], so documentation is something you must request.
Microsoft's schedule shows hosted open-weight models retiring too: several versions of open-weight families are listed as retired with 2026 dates, and one host's per-token inference carries a 15-day notice period [3]. Self-hosted weights never retire, but capacity, patching and monitoring become yours (NKK reasoning, not evidence).
Residency is a selection criterion, not a deployment afterthought. Anthropic documents two independent settings, where inference runs and where data is stored, with only "us" and "global" inference geos currently available [29]. US-only inference is priced at 1.1 times the standard rate on newer models, and earlier model generations return a 400 error when you set the control [29]. OpenAI states that "Support for regional storage does not imply support for regional processing" and charges a 10% uplift on data residency endpoints for models released on or after 5 March 2026 [30]. Where a region lacks regional processing, content may be processed outside it [30].
Vendor risk has a supply-chain face. OWASP's 2025 edition lists outdated or deprecated models, weak provenance and licensing risk among its LLM supply-chain concerns; we did not check whether a later edition changes this entry [21]. And preservation is not availability: Anthropic commits to preserving the weights of publicly released models for at least the company's lifetime [31], yet retired models fail on request [1].
In the EU, obligations on general-purpose AI model providers have applied since 2 August 2025, Commission enforcement powers began on 2 August 2026, and models placed on the market before 2 August 2025 must comply by 2 August 2027 [32]. The Commission's AI Act page says the Digital Omnibus on AI, in force from 27 July 2026, moved high-risk dates and does not say it changed these general-purpose dates [34]; all dates here come from Commission pages. Article 53(1)(b) requires providers to make documentation available to those who integrate the model [33]. Add "documentation obtained" to your envelope and take legal advice.
| Criterion | Closed API | Open-weight, self-hosted | Open-weight via hosted API |
|---|---|---|---|
| Who sets the retirement date | The provider, and the platform you buy through | You; weights you hold do not retire | The host; one host in our sources gives 15 days' notice on per-token inference |
| Weight access and audit | None; documentation depends on what the provider discloses | Full, subject to the licence | Weights exist elsewhere, but your deployment depends on the host |
| Licence conditions | Commercial terms and usage policy | Vary by licence; one example has a 700 million user threshold and naming rules | Host terms plus the model licence |
| Residency | Differs by provider and model generation | Wherever you run it | The host's regions; check processing as well as storage |
| Operating burden | Provider operates | Yours: capacity, security patching, monitoring | Host operates; you still own evals and migration |
- Closed API
- The provider, and the platform you buy through
- Open-weight, self-hosted
- You; weights you hold do not retire
- Open-weight via hosted API
- The host; one host in our sources gives 15 days' notice on per-token inference
- Closed API
- None; documentation depends on what the provider discloses
- Open-weight, self-hosted
- Full, subject to the licence
- Open-weight via hosted API
- Weights exist elsewhere, but your deployment depends on the host
- Closed API
- Commercial terms and usage policy
- Open-weight, self-hosted
- Vary by licence; one example has a 700 million user threshold and naming rules
- Open-weight via hosted API
- Host terms plus the model licence
- Closed API
- Differs by provider and model generation
- Open-weight, self-hosted
- Wherever you run it
- Open-weight via hosted API
- The host's regions; check processing as well as storage
- Closed API
- Provider operates
- Open-weight, self-hosted
- Yours: capacity, security patching, monitoring
- Open-weight via hosted API
- Host operates; you still own evals and migration
Routing, cascades and the single-model baseline
The research case for routing is real. RouteLLM's routers, which choose between a stronger and a weaker model at inference, cut costs by more than 2x in some cases without compromising quality, and kept working when the model pair was swapped at test time [35]. FrugalGPT reported matching the best individual model with up to 98% lower cost in the authors' experiments on 2023 models [18]. Cascade routing outperformed routing or cascading alone, with good quality estimators named "the critical factor" [36]. A 2026 survey concludes that practical systems are compositional [37].
Vendor measurements are more sober. Anthropic found a multi-model configuration that looked cheaper than the default single model but cost more than that same model at lower effort, and states the test plainly: "A multi-model configuration must beat the single model's whole curve" [38]. In an internal coding benchmark, an advisor configuration scored 90.1% at $2.92 per attempt, while the executor alone at higher effort scored 91.1% at $4.11, so "the advisor buys about what more effort does" [38].
Two further results qualify the picture. Executors given an advisor tool without a prompt asking them to consult it did not call it on any of 198 GPQA Diamond questions, so escalation can fail silently unless you measure the escalation rate [38]. On a 21.6-million-token corpus too large for any context window, an orchestrated setup cost 47% to 55% less than the lead model alone but scored 10 to 12 points lower; the same page reports savings for routine work with a long cost tail, and advises against an orchestrator for one dependent chain of work or work that fits in one context without such a tail [38]. Anthropic labels its numbers internal and directional, not guarantees [38].
In NKK's view, routing saves money when traffic is heterogeneous and you have a reliable quality signal. Otherwise it adds latency and a failure mode.
The NKK Escalation Ladder turns this into a discipline: when quality falls short on your eval, pull the cheapest sufficient lever first, and re-run the eval after every step. It draws on vendor model-selection guidance [19], OpenAI's evals, prompt-with-context, then fine-tune workflow [39] and the routing literature [35]; making retrieval its own step is NKK's view. Where it breaks: the order is a heuristic, and strict output formats at very high volume can justify fine-tuning early.
- Step 1: Sharpen the prompt
Clearer instructions, examples, output schema. Re-run the eval; log cost and latency.
- Step 2: Raise effort
Increase reasoning effort on the same model before switching anything.
- Step 3: Add retrieval
Ground answers in current or proprietary knowledge that changes faster than a model.
- Step 4: Change tier
Move up when a capability gap remains, or down once the eval passes with margin.
- Step 5: Add a cascade or router
Only after a single-model baseline exists, and only if it beats that model's whole cost curve.
- Step 6: Fine-tune
For stable behaviour or format at volume, once cheaper levers are exhausted.
Prompt, retrieve or fine-tune
OpenAI's optimisation loop starts with evals, then prompting with context, and notes that prompt engineering "may be all you need" [39]. For knowledge, evidence favours retrieval. In one 2023 comparison on older open models, retrieval-augmented generation "consistently outperforms" unsupervised fine-tuning for both existing and entirely new knowledge, and models "struggle to learn new factual information" through unsupervised fine-tuning alone [40]. The original RAG paper framed provenance and updating knowledge as open problems that non-parametric memory addresses [41]. Fine-tuning has a different job: shaping behaviour. Low-rank adaptation (LoRA) learns less than full fine-tuning on code and maths, but "better maintains the base model's performance on tasks outside the target domain" [42].
Fine-tuning also carries lifecycle cost. OpenAI states it is "winding down the fine-tuning platform" [39]; active existing customers can no longer create new jobs on 6 January 2027, and inference on existing fine-tuned models continues only until the base model is deprecated [4]. Bedrock's Legacy state blocks new fine-tuning jobs [43], and Microsoft retires fine-tuned models in two phases, training and then deployment [17]. Our inference: every fine-tune is a bet on one base model's lifespan, and the retraining bill returns at each retirement.
- Fine-tune for behaviour or format
- Stable task, high volume: tuned behaviour or format can repay its cost, with a re-tuning plan.
- Retrieve, then route
- Volatile knowledge at volume: retrieval for freshness, a router or cascade for cost.
- Prompt and effort
- Low volume, stable task: prompting and effort settings are usually enough.
- Retrieve
- Low volume, volatile knowledge: retrieval with a capable tier; tuning rarely pays.
Treat deprecation as a lifecycle, not an incident
There is no common standard for notice, and the same model can have different dates on different platforms.
| Criterion | Stated notice or lifecycle | Scope and caveat |
|---|---|---|
| Anthropic API | At least 60 days' notice for publicly released models | Partner platforms set their own schedules |
| OpenAI API | GA models at least 6 months; specialised variants at least 3 months; previews as little as about 2 weeks | Dates are per model snapshot, so check each one |
| Microsoft Foundry | GA lifecycle 18 months, 12 months for Anthropic, DeepSeek, Fireworks and Mistral AI models; at least 60 days' notice for GA, 30 for preview | Retirement dates are not extendable; one host's per-token inference has 15 days |
| Amazon Bedrock | Legacy period of 6 months or 45 days, then end of life | Applies to models launched from 7 September 2026; access can lapse after 15 days of inactivity in Legacy |
| Google Vertex AI | Some models available at least 12 months; at least 45 days to migrate once a date is posted | Timelines may be extended but not moved earlier |
- Stated notice or lifecycle
- At least 60 days' notice for publicly released models
- Scope and caveat
- Partner platforms set their own schedules
- Stated notice or lifecycle
- GA models at least 6 months; specialised variants at least 3 months; previews as little as about 2 weeks
- Scope and caveat
- Dates are per model snapshot, so check each one
- Stated notice or lifecycle
- GA lifecycle 18 months, 12 months for Anthropic, DeepSeek, Fireworks and Mistral AI models; at least 60 days' notice for GA, 30 for preview
- Scope and caveat
- Retirement dates are not extendable; one host's per-token inference has 15 days
- Stated notice or lifecycle
- Legacy period of 6 months or 45 days, then end of life
- Scope and caveat
- Applies to models launched from 7 September 2026; access can lapse after 15 days of inactivity in Legacy
- Stated notice or lifecycle
- Some models available at least 12 months; at least 45 days to migrate once a date is posted
- Scope and caveat
- Timelines may be extended but not moved earlier
Platform pages diverge. Microsoft's schedule lists one Claude model retiring on 10 February 2027, while Anthropic's table says not sooner than 17 February 2027, and the two name different replacements for an older model [3][1]. Anthropic's overview says Foundry follows the Claude API schedule [15]. On Bedrock, "only the dates on the model card apply" [43]. Confirm dates on the platform you deploy on.
Pinning is necessary but not sufficient. Since the 4.6 generation, Anthropic's dateless IDs are pinned snapshots, not evergreen pointers [14], and OpenAI advises pinning production apps to snapshots [23]. Pinned snapshots still carry shutdowns; one is scheduled for 11 December 2026 [4]. Academic work warns that LLM APIs "are often updated silently and scheduled to be deprecated" [45], so keep regression evals running on pinned IDs too.
Do not wait for the vendor's replacement. Microsoft names a replacement only 90 to 120 days before retirement because "declaring a replacement too early risks directing customers to a model that is no longer the best available option", and tells customers to evaluate candidates on their own application and data [17]. Provisioned deployments are not auto-upgraded, and dates cannot be extended [17]. Anthropic warns that deprecated models are likely to be less reliable than active ones [1], and OWASP counts deprecated models as a security issue [21].
Migrations are wider than an ID swap: temperature, top_p and top_k return a 400 error when set to non-default values on Claude 4.7 and later models, so audit actual usage first [1].
The NKK Deprecation Clock is our view of how to run this as a loop. It builds on vendor lifecycle guidance [1][17][43] and on regression testing for evolving APIs [45]. The platform team owns Inventory (quarterly) and Watch (continuous); the workload owner runs the other steps when a notice lands. Where it breaks: Watch depends on vendor notices, and some windows are as short as 15 days or about 2 weeks [3][4], so a second model must already be warm in Shadow before the notice arrives.
- Inventory
Every pinned ID, parameter and platform in production, by workload.
- Watch
Vendor notices and lifecycle fields on the platform you deploy on.
- Re-evaluate
Run the eval on the named replacement and on one alternative.
- Shadow
Send parallel traffic and compare cost, latency and pass rate.
- Cut over
Switch on an explicit upgrade policy; never rely on silent auto-upgrade for critical flows.
- Archive
Store eval results, prompts and the decision date on the Selection Card.
Swap-readiness: how cheaply can you change your mind?
The goal of everything above is cheap reversibility. The NKK Swap-Readiness Ladder describes five levels, built on pinned IDs [14][23], regression testing for evolving APIs [45] and routing research [36]. Test signals, not self-image: can you list every pinned ID in production within a day, and can you run your suite against a second model before lunch? Where it breaks: levels are not strictly sequential, and a gateway can itself become a single point of failure. The scorecard below grades a workload's Selection Card.
- Level 1: Hard-coded
Model names live in application code and nobody owns the list.
- A retirement notice triggers a code search
- Level 2: Pinned and inventoried
Pinned IDs sit in configuration with an inventory of every call.
- You can name every pinned ID in production within a day
- Level 3: Gateway plus regression
A provider-neutral layer exists and the regression eval runs on every change.
- A model change produces a pass-rate report, not an opinion
- Level 4: Two qualified models
A second model is qualified with shadow traffic and its own tuned prompt.
- You can cut over within days of a notice
- Level 5: Measured routing
Routing or cascade with a measured quality signal and automatic fallback.
- Escalation rate and cost per outcome are on a dashboard
Answer all 5 dimensions to see your score and where to start.
What the evidence does not show
- Independent latency data. Our accepted sources contain no independent cross-provider latency benchmark; latency claims rest on vendor documentation [15][19].
- Routing savings today. The 98% and 2x figures come from older models [18][35]; Anthropic's are internal [38].
- Retrieval versus fine-tuning on current models. The comparisons are from 2023 and 2024 [40][42].
- A stable open-versus-closed gap. The Arena readings differ by edition and the leaderboard is contested [2][6][11].
- Stable vendor facts. Vendor pages are mostly undated, Anthropic-heavy and revised often.
- Primary EU legal text. We relied on Commission pages for AI Act dates because the Official Journal text could not be opened when we wrote this; verify before acting.
- Our own frameworks. The Selection Card, Escalation Ladder, Deprecation Clock and Swap-Readiness Ladder are NKK opinion. We have no validation data or case studies for them.
What to do next
- Inventory every model call. List pinned IDs, parameters, platform and owner per workload; reach the "within a day" signal.
- Write 20 to 50 task evals from real failures per priority workload, with graders and a regression suite you hold in your own repository.
- Fill in a Selection Card per workload, including the envelope and the documentation you need from the vendor.
- Qualify a second model on a different vendor or platform, with its own tuned prompt, and run it in shadow.
- Test effort and the cheapest tier against your eval before considering any router; add routing only if it beats the single-model curve.
- Set a quarterly re-test and a deprecation watch on the platform where you deploy, and choose an explicit upgrade policy.
- Record residency and licence checks (storage versus processing, licence thresholds) in the card, and revisit them when the model changes.
Conclusion
The model name is the least durable part of your AI stack. The eval, the written envelope and the practised swap path are the parts that last. Choose the cheapest tier that passes your own tests, keep a second option warm, and treat every retirement notice as a scheduled re-test rather than a surprise.
References
Every figure and regulatory statement in this piece traces to a numbered source below. Links open the original publisher in a new tab.
- Reference 1: Model deprecations(opens in a new tab)Anthropic (Claude Platform docs)Published n.d.Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 2: The 2026 AI Index Report, Chapter 2: Technical Performance(opens in a new tab)Stanford HAIPublished 2026Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 3: Model retirement schedule - Microsoft Foundry(opens in a new tab)Microsoft LearnPublished 21 Sept 2026Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 4: Deprecations (OpenAI API)(opens in a new tab)OpenAI (developers.openai.com)Published n.d.Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 5: LLM inference prices have fallen rapidly but unequally across tasks(opens in a new tab)Epoch AI (Data Insight)Published 12 Mar 2025Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 6: The 2025 AI Index Report, Technical Performance(opens in a new tab)Stanford HAIPublished 2025Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 7: The 2026 AI Index Report(opens in a new tab)Stanford HAIPublished 2026Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 8: Are We Done with MMLU?(opens in a new tab)arXiv (Gema et al.)Published 6 Jun 2024Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 9: Benchmark Data Contamination of Large Language Models: A Survey(opens in a new tab)arXiv (Xu, Guan, Greene, Kechadi)Published 6 Jun 2024Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 10: Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference(opens in a new tab)arXiv (Chiang et al.)Published 7 Mar 2024Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 11: The Leaderboard Illusion(opens in a new tab)arXiv (Singh et al.)Published 29 Apr 2025Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 12: Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design(opens in a new tab)arXiv (Sclar, Choi, Tsvetkov, Suhr)Published 17 Oct 2023Accessed 13 Oct 2026Tier 1: Primary source
- Reference 13: How is ChatGPT's behavior changing over time?(opens in a new tab)arXiv (Chen, Zaharia, Zou)Published 18 Jul 2023Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 14: Model IDs and versioning(opens in a new tab)Anthropic (Claude Platform docs)Published n.d.Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 15: Models overview(opens in a new tab)Anthropic (Claude Platform docs)Published n.d.Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 16: Demystifying evals for AI agents(opens in a new tab)Anthropic EngineeringPublished 9 Jan 2026Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 17: Foundry Models lifecycle and support policy(opens in a new tab)Microsoft LearnPublished 24 Jul 2026Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 18: FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance(opens in a new tab)arXiv (Chen, Zaharia, Zou)Published 9 May 2023Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 19: Choosing the right model(opens in a new tab)Anthropic (Claude Platform docs)Published n.d.Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 20: Holistic Evaluation of Language Models(opens in a new tab)arXiv (Liang et al., Stanford CRFM)Published 16 Nov 2022Accessed 13 Oct 2026Tier 1: Primary source
- Reference 21: Top 10 for LLM Applications 2025: LLM03 Supply Chain(opens in a new tab)OWASP GenAI Security ProjectPublished 2025Accessed 13 Oct 2026Tier 1: Primary source
- Reference 22: Working with evals(opens in a new tab)OpenAI (developers.openai.com)Published n.d.Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 23: Prompt engineering (OpenAI API guide)(opens in a new tab)OpenAI (developers.openai.com)Published n.d.Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 24: Lost in the Middle: How Language Models Use Long Contexts(opens in a new tab)arXiv (Liu et al.)Published 6 Jul 2023Accessed 13 Oct 2026Tier 1: Primary source
- Reference 25: RULER: What's the Real Context Size of Your Long-Context Language Models?(opens in a new tab)arXiv (Hsieh et al.)Published 9 Apr 2024Accessed 13 Oct 2026Tier 1: Primary source
- Reference 26: The Open Source AI Definition 1.0(opens in a new tab)Open Source InitiativePublished n.d.Accessed 13 Oct 2026Tier 1: Primary source
- Reference 27: Llama 4 Community License Agreement(opens in a new tab)Meta (dev.meta.ai)Published 5 Apr 2025Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 28: The 2026 AI Index Report, Chapter 1: Research and Development(opens in a new tab)Stanford HAIPublished 2026Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 29: Data residency (inference geo and workspace geo)(opens in a new tab)Anthropic (Claude Platform docs)Published n.d.Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 30: Data controls in the OpenAI platform(opens in a new tab)OpenAI (developers.openai.com)Published n.d.Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 31: Commitments on model deprecation and preservation(opens in a new tab)AnthropicPublished 4 Nov 2025Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 32: Guidelines for providers of general-purpose AI models(opens in a new tab)European Commission, Shaping Europe's digital futurePublished 28 Apr 2026Accessed 13 Oct 2026Tier 1: Primary source
- Reference 33: Article 53: Obligations for providers of general-purpose AI models(opens in a new tab)European Commission, AI Act Service Desk (AI Office)Published n.d.Accessed 13 Oct 2026Tier 1: Primary source
- Reference 34: AI Act (regulatory framework for AI)(opens in a new tab)European Commission, Shaping Europe's digital futurePublished 3 Aug 2026Accessed 13 Oct 2026Tier 1: Primary source
- Reference 35: RouteLLM: Learning to Route LLMs with Preference Data(opens in a new tab)arXiv (Ong et al.)Published 26 Jun 2024Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 36: A Unified Approach to Routing and Cascading for LLMs(opens in a new tab)arXiv (Dekoninck, Baader, Vechev)Published 14 Oct 2024Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 37: Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey(opens in a new tab)arXiv (Moslem, Kelleher)Published 23 Feb 2026Accessed 13 Oct 2026Tier 1: Primary source
- Reference 38: Optimizing for cost and intelligence (advisor and orchestrator strategies)(opens in a new tab)Anthropic (Claude Platform docs)Published n.d.Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 39: Model optimization(opens in a new tab)OpenAI (developers.openai.com)Published n.d.Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 40: Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs(opens in a new tab)arXiv (Ovadia, Brief, Mishaeli, Elisha)Published 10 Dec 2023Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 41: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks(opens in a new tab)arXiv (Lewis et al.)Published 22 May 2020Accessed 13 Oct 2026Tier 1: Primary source
- Reference 42: LoRA Learns Less and Forgets Less(opens in a new tab)arXiv (Biderman et al.)Published 15 May 2024Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 43: Model lifecycle (Amazon Bedrock User Guide)(opens in a new tab)Amazon Web ServicesPublished n.d.Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 44: Model versions and lifecycle (Vertex AI generative AI)(opens in a new tab)Google Cloud documentationPublished n.d.Accessed 13 Oct 2026Tier 2: Research / official docs
- Reference 45: (Why) Is My Prompt Getting Worse? Rethinking Regression Testing for Evolving LLM APIs(opens in a new tab)arXiv (Ma, Yang, Kastner)Published 18 Nov 2023Accessed 13 Oct 2026Tier 2: Research / official docs