How much does it cost to train an AI model? GPU, data center, power, tokens, and AI compute cost explained

In 2023 someone wrote GPT-4's compute as "sixty-plus million dollars." In 2025 the news spread that Llama 3 405B used tens of thousands of H100s. In 2026 the flagships — GPT-5.6, Claude Fable 5.1, Gemini 3.8 — have a training budget that is "a data-center project," not "a batch of cards." The numbers differ by two orders of magnitude not because reporters can't math, but because the scope differs: some people only report GPU-hours for the successful pretrain; others fold in failed runs, data cleaning, inference clusters, and three years of power.

What the bill actually counts

Tighten the words first. Below, "training cost" means the compute and hall bill for one dense pretrain run to completion — not product inference, not eval clusters sitting idle. Full-project TCO often multiplies by 1.3–2.0, because the same recipe rarely lands on the first try.

From parameters and tokens to FLOPs

For dense Transformer pretraining, the industry uses the Kaplan / Chinchilla approximation:

Training FLOPs ≈ 6 × N × D. N is trainable parameters; D is token count. Forward is about 2ND; backward about 4ND. For mixture-of-experts (MoE), estimate from active parameters — don't slam total parameters into the formula.

Convert to GPU-hours:

GPU-hours ≈ training FLOPs ÷ (peak FLOP/s per card × utilization × 3600)

H100 SXM BF16 peak is often written as about 990 TFLOPS. Communication, checkpoints, and the data pipeline push utilization down to 35%–50%. At 40%, one card is effectively about 4×10¹⁴ FLOP/s. Use the datasheet peak as the denominator and the bill looks too optimistic.

GPUs: rent, buy, utilization

Price bands you can publicly match as of 2026-09 (spot is cheaper, reserved is steadier, regions vary a lot):

Owning the cluster is not "card price ÷ hours" that clean. You still need switches, optics, spares, idle time, and repairs. Cloud is expensive on the markup and cheap because you don't pour a building first. Fine-tune on 8 cards: cloud is almost always right. Two thousand cards for three months straight: finance starts modeling ownership.

Utilization hurts more than list price. A cluster at 40% versus 20% doubles the money on the same 70B recipe. A bad learning rate, a rotten checkpoint, a hiccup in collective comms — all go into GPU-hours. None of it lands in the paper appendix.

Data center: network, cooling, the hall

Training is not "a lot of cards in a row." Bandwidth between cards decides whether you can get near 40% utilization. NVLink / NVSwitch handle intra-node; InfiniBand or equivalent RoCE handles inter-node. The network and optics for 10,000 H100s are often a serious fraction of the card bill — not an accessory.

Three hall-side facts turn "we have cards" into "we can train":

1. Power density. Older data centers were designed for 8–15 kW/rack. AI racks often hit 40–120 kW. No power, no liquid cooling: the cards clock down or sit idle.
2. PUE. New AI campuses commonly sit at 1.15–1.35. PUE 1.4 versus 1.2 is about one-sixth more on the power bill — real money over a year.
3. Build time. Substations, cooling towers, and land are slower than buying cards. What constrains flagship training in 2026 is often megawatts and schedule, not "do we have an algorithm."

So "how much does training cost" sometimes has to be restated: first ask "do you have 20–80 MW of capacity," then ask card price.

Power: how watts become dollars

Board TDP on a single card is not enough. An H100 board is about 700 W; add CPU, NICs, storage, and switches, and 1.0–1.2 kW/GPU is closer to system draw. Then multiply by PUE.

Rough:

Annual power ≈ GPU count × 1.1 kW × PUE × 8760 × tariff

10,000 H100s, PUE 1.25, $0.07/kWh: about 1.1×1.25×8760×0.07×10000 ≈ $8.4 million/year. At $0.12 you hit $14 million. That still excludes inference clusters outside training.

Power is rarely the "largest line item," but it decides whether the cluster can run 24×7 at full pull. Curtailment, demand response, summer capacity — all knock utilization down. The GPU-hour sticker does not change; the recipe gets more expensive.

Four scales: 7B / 70B / 405B / flagship

Below: H100, 40% utilization, cloud $2.5 / GPU-hour, estimating compute for the one successful pretrain. Multiply TCO by 1.4–2.0. These are 2026-09 orders of magnitude, not a quote sheet.

The 7B row: many teams can finish it on the cloud. Data and people cost more than the cards. 70B is already "you need dedicated cluster slots." 405B and flagship are capex: substations, multi-year procurement, a failed-run budget in the board deck. How the product lines split: see Pick a model by task — most products should buy the API, not reprint this row.

Tokens: train once, inference keeps burning

Tokens show up on both sides. The unit price is not the same thing.

• Training tokens are recipe input. 70B × 15T is about $11M in compute — roughly $0.7–$3 per million training tokens (GPU-hours only). Add data and people and it goes higher.
• Inference tokens are product output. Flagship APIs in 2026 commonly run from a few dimes to the teens per million tokens, depending on the model and cache. Self-hosted inference can go lower if utilization is high — after you pay for the cluster.

100 million tokens/day at a blended $3 / million is about $90k/month; three months approaches one "successful 7B pretrain." A billion tokens/day and a few months can pass one 70B pretrain. So externally they say "training costs a hundred million"; internally finance watches whether the inference curve has crossed the training peak.

Retries are a hidden multiplier. The prompt asks for JSON, the model wraps it in markdown, you call again: bill × 2. In an agent loop, tool parameters drift and the next hop is dead. That is why the contract belongs in the decoder, not the system prompt — see Structured Output and Why AI likes JSON.

For developers: don't train a flagship, watch retries

Training your own 70B+ pretrain is the wrong problem for 99% of teams. In 2026 the books that matter are three smaller ones:

1. Tokens per call. System prompt, retrieval chunks, tool returns — all hit the input price. If you can validate it locally, don't throw it back at the model.
2. Failure retry rate. Parse failures, missing fields, enums that fly off — the most expensive kind of "looks like it's working." Schema passes, then write to the database.
3. Swap the model, not the fields. Route to a cheaper Flash / mini; the contract is still the same JSON. Pick a model: 2026 AI model war.

Fine-tune, distill, continued pretrain are another bracket: commonly hundreds to tens of thousands of GPU-hours, cloud bills in the four-to-six-figure range. Still write the recipe as an object first: base_model, gpu_hours, usd, eval_pass_rate, so the next round can diff.

A cost breakdown object

Below is a sample object for "dense 70B, 15T tokens, H100, cloud $2.5." Not a quote — a field contract. Money as number, enums as strings, whether failed runs are included as boolean.

{
  "run_id": "pretrain-70b-20260908-01",
  "model_class": "dense-70b",
  "tokens_trained": 15000000000000,
  "gpu": "H100-SXM",
  "utilization": 0.4,
  "gpu_hours": 4420000,
  "gpu_usd_per_hour": 2.5,
  "compute_usd": 11050000,
  "power_usd": 1800000,
  "datacenter_usd": 4200000,
  "data_usd": 2500000,
  "staff_usd": 6000000,
  "includes_failed_runs": true,
  "retries_factor": 1.4,
  "total_usd": 35770000,
  "cost_per_million_train_tokens_usd": 2.38,
  "currency": "USD"
}

With this site's inference rules, that sample yields:

Whether two runs can be compared depends on whether the scope fields are there. Missing includes_failed_runs, Diff only tells you the total drifted — not whether you ran more failed experiments or the power tariff changed. JSON Diff looks at objects, not two paragraphs of prose.

How to accept it in JSON Toolbox

1. Paste the sample above into JSON Format and confirm it parses.
2. Generate a first draft with Structured Output or JSON → Schema.
3. Take a second set of real run numbers to Schema validation. The first sample is always a bit too clean.
4. Two training runs / two cloud bills: use JSON Diff to see whether GPU-hours drifted or a scope field dropped.
5. For a table people can read, use JSON Table; for frontend checks, convert to Zod.

Everything stays in the browser and is never uploaded.

Common questions

How much does it actually cost to train a large model?

Depends on scale. 7B is commonly a few hundred thousand to one or two million dollars; 70B is $20M–$40M; 405B sits around $100M; a full 2026 flagship round often lands at $200M–$1B. A single number is meaningless.

Is GPU the largest line on the bill?

Compute-hours are usually the biggest head. Full TCO still adds power, cooling, network, the hall, data, and people. Power alone is often 10%–20%, but it decides whether the cluster can run full-out.

Is training more expensive, or inference?

Training is a one-time peak. When a product hits hundreds of millions of tokens a day, a few months of inference can pass one 70B pretrain. Developers manage tokens and retries first.

How do you estimate how many GPU-hours training needs?

For a dense Transformer, 6 × parameters × tokens estimates FLOPs; divide by peak per card and utilization. Use 35%–50% utilization, not the datasheet peak.

How do developers claw compute cost back off the bill?

Retry less. Structured Output and Schema stop "call the model again." Put the contract in JSON; don't stop at the prompt.

Summary

Training cost is not a slogan number. It is GPU-hours from 6ND, times card price, plus the hall, power, data, and failed runs. 7B can finish on the cloud; 70B is a dedicated cluster; 405B and 2026 flagships are capex. For most developers, the more expensive bits are inference tokens and retries after a parse failure. Fold every train, every fine-tune, every cloud bill into the same object: fields, types, required, a scope switch. Money as number, enums closed, includes_failed_runs must be boolean. When the shape is right, you can compare two runs — not two sentences that both say "expensive."

Related tools