Pain Point

High Cloud Inference Cost

The expense of running LLM/ML inference via cloud APIs (per-token or per-request pricing) against S3 data at scale.

22 connections3 resources7 posts

Summary

What it is

The expense of running LLM/ML inference via cloud APIs (per-token or per-request pricing) against S3 data at scale.

Where it fits

This is the economic constraint that limits LLM adoption over S3 data. Embedding generation, metadata extraction, and classification are useful but only viable if inference costs do not exceed the value of the results.

Misconceptions / Traps
  • Cost is not just API pricing. Egress charges for moving S3 data to inference endpoints, and storage costs for embeddings, add to the total.
  • "Run it locally" is not free either. Local inference has GPU hardware, power, and maintenance costs. The break-even volume depends on model size and throughput.
Key Connections
  • Local Inference Stack solves High Cloud Inference Cost — runs models on local hardware
  • Offline Embedding Pipeline constrained_by High Cloud Inference Cost — batch processing amortizes cost
  • Embedding Generation, Metadata Extraction, Data Classification constrained_by High Cloud Inference Cost
  • Hybrid S3 + Vector Index constrained_by High Cloud Inference Cost — embedding generation is expensive
  • scoped_to LLM-Assisted Data Systems, S3

Definition

What it is

The expense of running LLM/ML inference via cloud APIs (per-token or per-request pricing) against S3-stored data at scale.

Recent developments

Latest signals
  • H100 SXM5 spot at $1.03/hr — 40-85% cheaper than hyperscalers. Spheron spot pricing leads the market at $1.03/hr H100 SXM5 (May 2026); neo-cloud providers now offer H100 capacity 40-85% below AWS/GCP/Azure published rates. Per Spheron — GPU Cloud Pricing 2026.
  • AWS H100 pricing dropped 44% in June 2025. Hyperscaler response to neo-cloud price pressure — AWS cut H100 pricing by 44% in June 2025, with similar adjustments across GCP and Azure following. Per Spheron — GPU Cloud Pricing 2026.
  • B200 on-demand is now the best $/token leader. B200 on-demand at $0.42/M tokens edges out H100 PCIe ($0.47/M) despite higher hourly rate; on spot, B200 at $2.12/hr yields ~$0.15/M tokens — the cost leader for checkpoint-friendly workloads. Per GMI Cloud — GPU pricing A100 vs H100 vs H200.
  • Serverless GPU compute fills the spot/dedicated middle. Modal (~$3.95/hr H100), Together AI ($3.49/hr → $2.25/hr reserved), and Crusoe Cloud all serve the per-second-billing inference-API alternative to hosted OpenAI/Anthropic APIs. Per Crusoe Cloud Pricing.
  • H200's 141 GB VRAM beats H100 once concurrency scales. H200 at $2.60/hr often beats H100 at $2.00/hr economically past a certain concurrency threshold — the larger HBM holds more KV cache + more concurrent requests per GPU. Per GMI Cloud — Best Cloud GPUs for AI Inference.
  • Nebius reserved-month pricing as the AWS alternative. Nebius's $2.00/hr H100 and $2.30/hr H200 with multi-month agreements is the standard "rent capacity, not API tokens" path for teams escaping hosted-API per-token cost. Per Spheron — GPU Cloud Pricing.
  • The flat-rate subscription is no longer a stable unit of inference capacity. 2025–2026 delivered a documented wave of effective-price increases executed without headline price changes: Cursor's June 2025 silent shift from flat requests to API-rate credits (CEO apology + refunds), GitHub Copilot's premium-request multipliers and June 1, 2026 move to usage-based AI Credits (vendor-published), Anthropic's March 2026 peak-hour quota burn (user-discovered, staff-confirmed) and its announced net ~17% weekly-limit reduction effective September 14, 2026, and xAI's May 15, 2026 retirement of Grok 4.1 Fast with automatic redirect billing at grok-4.3's ~6.25× input rate (vendor-published migration doc). The full case ledger lives on the new AI Subscription Quota Opacity pain point. Per GitHub — Copilot usage-based billing, BleepingComputer — Claude Code weekly limits, and xAI — May 15 retirement.
  • COGS analyses say the $20 frontier tier is sold below cost for heavy users. Analyst estimates place provider losses on power users at 10–40× subscription revenue, with extreme agentic accounts reportedly consuming up to ~$14K/month in API-equivalent compute on a $20 plan (aggregator-repeated attribution to SemiAnalysis; treat the headline number as directional). Under ~$400B/year of AI-infrastructure CapEx requiring amortization, the squeeze on flat-rate tiers is structural, not tactical (independent-analyst estimate). Per LessWrong — Power Overwhelming: dissecting the $1.5T AI revenue shortfall.
  • The floor keeps dropping while the subscriptions tighten. DeepSeek's V4-Pro 75% cut went permanent May 2026 ($0.435/M input, $0.87/M output — vendor-published); Zhipu's GLM-4-Flash class remains free. The economic exit paths this index maps — Local Inference Stack, open-weight floor models, gateway-level caching — are priced against exactly this squeeze. Per DeepSeek API pricing docs.

Connections22

Outbound2
scoped_to2
LLM-Assisted Data SystemsS3
Inbound20
solves11
Technology8
MinIO MemKVS3 Express One ZoneAmazon S3 VectorsAWS LambdaWasabi AiRWEKANVIDIA GPUDirect RDMA for S3LiteLLM
Architecture2
Local Inference StackHybrid Retrieval
Model Class1
DeepSeek V4
constrained_by7
Hybrid S3 + Vector IndexOffline Embedding PipelineOnline Embedding Refresh PipelineClaude Fable 5Embedding GenerationMetadata ExtractionData Classification
related_to2
Cloud AI Storage Price InversionNAND Flash Supply Crisis

Resources3

DocsHigh
docs.aws.amazon.com/sagemaker/latest/dg/inference-cost-optim...

Official AWS SageMaker documentation on inference cost optimization best practices, covering multi-model endpoints, autoscaling, instance selection, and S3 model streaming to reduce costs.

BlogHigh
developer.nvidia.com/blog/reducing-cold-start-latency-for-ll...

NVIDIA technical blog showing how streaming models from S3 reduces cold-start latency and infrastructure costs compared to pre-loading models to local storage.

BlogMedium
introl.com/blog/inference-unit-economics-true-cost-per-milli...

Detailed breakdown of inference unit economics including GPU costs, model storage on S3, and hidden expenses that make up 60-80% of total spend.

Featured in

2026-09-03When the Quota Became a VariableWe measured our own $20 inference plan dropping from ~2M tokens per session to ~700K — same plan, same price, no changelog. Then we pulled the industry-wide evidence: a documented 2025–2026 wave of quota re-denominations, peak-hour throttles admitted only after user telemetry caught them, and deprecations that silently rebill at 6× the rate. Also the counter-evidence, because the thesis only survives if it's tested: OpenAI uncapped text, Anthropic doubled limits in May, and DeepSeek's price floor keeps falling. The pattern isn't uniform gouging — it's bifurcation plus unit games. And it's the strongest argument yet for the stack this index maps.2026-07-05The Frontier Moved Again — and the Floor CompoundedIn June we argued the AI stack had split into a closed, expensive frontier and an open, cheap floor. Six weeks later the frontier proved the point the hard way: Claude Fable 5 was pulled offline for 19 days by an export-control order, then restored with a retention mandate that quietly bars the most sensitive data from ever touching it. Meanwhile the floor didn't just hold — it compounded. Object storage got RDMA-fast, the query engines converged on one substrate, and the data plane grew a control plane built for agents instead of humans. This is the field report.2026-06-16The Frontier and the Floor: The AI Stack Just Split in TwoIn a single quarter, Anthropic's Claude Fable 5 reset the closed frontier while DeepSeek V4 put frontier-of-last-year capability into open weights at roughly one-fiftieth the price. The two events look like a race. They're the opposite: the AI stack is bifurcating into a closed, expensive frontier for the hardest autonomous work and an open, cheap floor for the high-volume inference that actually runs your data infrastructure. The question stopped being 'which model' and became 'which tier.'2026-06-05When Inference Became Cheaper Than Storage: The May 2026 Cost InversionIn May 2026 DeepSeek made its 75% price cut permanent. V4-Flash output is now 107x cheaper than GPT-5.5. For the first time the cost of intelligence dropped below the cost of moving data — and every storage decision made for the 2024 cost curve is now suspect. This is the continuation of the I/O-stack rewrite.2026-05-16When AI Memory Became an Architecture: KV-Cache Persistence, MCP, and the Night S3 Got Its Memory TierStateful agents needed stateful storage. Tonight 33 new nodes joined the index: Mem0, Zep, LMCache, SGLang, Mooncake, MCP, cuObject, BlueField-4, Animesis CMA. The architectural thesis: AI memory infrastructure crystallized around object storage in 2026, and the persistence layer the industry settled on is S3.2026-05-10When the AI Stack Became an I/O Stack: S3 Vectors GA, Real-Time Lakehouses, and the May 2026 Storage RewriteAmazon S3 Vectors hit GA at 20 trillion vectors per bucket. Apache Paimon is doing 40 million rows per second at ByteDance. Aliyun OSS embedded similarity search directly in the storage control layer. Three independent signals, one architectural truth: 2026's AI bottleneck is no longer compute — it's I/O, and the storage layer is being rewritten to absorb it.2026-05-07When DeepSeek Made Storage Strategic: The China Direction Reshaping the S3 EcosystemDeepSeek's $5.6M training run didn't just reset model architecture. It made object storage performance-critical, made silicon export controls a storage decision, and pushed Aliyun OSS, Tencent COS, and Huawei OBS from regional curiosities into the global S3 vocabulary. This index added 13 nodes to track what changed.