Pain Point

High Cloud Inference Cost

The expense of running LLM/ML inference via cloud APIs (per-token or per-request pricing) against S3 data at scale.

22 connections 3 resources 6 posts

Summary

What it is

The expense of running LLM/ML inference via cloud APIs (per-token or per-request pricing) against S3 data at scale.

Where it fits

This is the economic constraint that limits LLM adoption over S3 data. Embedding generation, metadata extraction, and classification are useful but only viable if inference costs do not exceed the value of the results.

Misconceptions / Traps
  • Cost is not just API pricing. Egress charges for moving S3 data to inference endpoints, and storage costs for embeddings, add to the total.
  • "Run it locally" is not free either. Local inference has GPU hardware, power, and maintenance costs. The break-even volume depends on model size and throughput.
Key Connections
  • Local Inference Stack solves High Cloud Inference Cost — runs models on local hardware
  • Offline Embedding Pipeline constrained_by High Cloud Inference Cost — batch processing amortizes cost
  • Embedding Generation, Metadata Extraction, Data Classification constrained_by High Cloud Inference Cost
  • Hybrid S3 + Vector Index constrained_by High Cloud Inference Cost — embedding generation is expensive
  • scoped_to LLM-Assisted Data Systems, S3

Definition

What it is

The expense of running LLM/ML inference via cloud APIs (per-token or per-request pricing) against S3-stored data at scale.

Recent developments

Latest signals
  • H100 SXM5 spot at $1.03/hr — 40-85% cheaper than hyperscalers. Spheron spot pricing leads the market at $1.03/hr H100 SXM5 (May 2026); neo-cloud providers now offer H100 capacity 40-85% below AWS/GCP/Azure published rates. Per Spheron — GPU Cloud Pricing 2026.
  • AWS H100 pricing dropped 44% in June 2025. Hyperscaler response to neo-cloud price pressure — AWS cut H100 pricing by 44% in June 2025, with similar adjustments across GCP and Azure following. Per Spheron — GPU Cloud Pricing 2026.
  • B200 on-demand is now the best $/token leader. B200 on-demand at $0.42/M tokens edges out H100 PCIe ($0.47/M) despite higher hourly rate; on spot, B200 at $2.12/hr yields ~$0.15/M tokens — the cost leader for checkpoint-friendly workloads. Per GMI Cloud — GPU pricing A100 vs H100 vs H200.
  • Serverless GPU compute fills the spot/dedicated middle. Modal (~$3.95/hr H100), Together AI ($3.49/hr → $2.25/hr reserved), and Crusoe Cloud all serve the per-second-billing inference-API alternative to hosted OpenAI/Anthropic APIs. Per Crusoe Cloud Pricing.
  • H200's 141 GB VRAM beats H100 once concurrency scales. H200 at $2.60/hr often beats H100 at $2.00/hr economically past a certain concurrency threshold — the larger HBM holds more KV cache + more concurrent requests per GPU. Per GMI Cloud — Best Cloud GPUs for AI Inference.
  • Nebius reserved-month pricing as the AWS alternative. Nebius's $2.00/hr H100 and $2.30/hr H200 with multi-month agreements is the standard "rent capacity, not API tokens" path for teams escaping hosted-API per-token cost. Per Spheron — GPU Cloud Pricing.

Connections 22

Outbound 2
scoped_to2
LLM-Assisted Data SystemsS3
Inbound 20
solves11
Technology8
MinIO MemKVS3 Express One ZoneAmazon S3 VectorsAWS LambdaWasabi AiRWEKANVIDIA GPUDirect RDMA for S3LiteLLM
Architecture2
Local Inference StackHybrid Retrieval
Model Class1
DeepSeek V4
constrained_by7
Hybrid S3 + Vector IndexOffline Embedding PipelineOnline Embedding Refresh PipelineClaude Fable 5Embedding GenerationMetadata ExtractionData Classification
related_to2
The 2026 NAND/Flash Supply ShortageCloud AI Storage Price Inversion

Resources 3

DocsHigh
docs.aws.amazon.com/sagemaker/latest/dg/inference-cost-optim...

Official AWS SageMaker documentation on inference cost optimization best practices, covering multi-model endpoints, autoscaling, instance selection, and S3 model streaming to reduce costs.

BlogHigh
developer.nvidia.com/blog/reducing-cold-start-latency-for-ll...

NVIDIA technical blog showing how streaming models from S3 reduces cold-start latency and infrastructure costs compared to pre-loading models to local storage.

BlogMedium
introl.com/blog/inference-unit-economics-true-cost-per-milli...

Detailed breakdown of inference unit economics including GPU costs, model storage on S3, and hidden expenses that make up 60-80% of total spend.

Featured in

2026-07-05 The Frontier Moved Again — and the Floor Compounded In June we argued the AI stack had split into a closed, expensive frontier and an open, cheap floor. Six weeks later the frontier proved the point the hard way: Claude Fable 5 was pulled offline for 19 days by an export-control order, then restored with a retention mandate that quietly bars the most sensitive data from ever touching it. Meanwhile the floor didn't just hold — it compounded. Object storage got RDMA-fast, the query engines converged on one substrate, and the data plane grew a control plane built for agents instead of humans. This is the field report. 2026-06-16 The Frontier and the Floor: The AI Stack Just Split in Two In a single quarter, Anthropic's Claude Fable 5 reset the closed frontier while DeepSeek V4 put frontier-of-last-year capability into open weights at roughly one-fiftieth the price. The two events look like a race. They're the opposite: the AI stack is bifurcating into a closed, expensive frontier for the hardest autonomous work and an open, cheap floor for the high-volume inference that actually runs your data infrastructure. The question stopped being 'which model' and became 'which tier.' 2026-06-05 When Inference Became Cheaper Than Storage: The May 2026 Cost Inversion In May 2026 DeepSeek made its 75% price cut permanent. V4-Flash output is now 107x cheaper than GPT-5.5. For the first time the cost of intelligence dropped below the cost of moving data — and every storage decision made for the 2024 cost curve is now suspect. This is the continuation of the I/O-stack rewrite. 2026-05-16 When AI Memory Became an Architecture: KV-Cache Persistence, MCP, and the Night S3 Got Its Memory Tier Stateful agents needed stateful storage. Tonight 33 new nodes joined the index: Mem0, Zep, LMCache, SGLang, Mooncake, MCP, cuObject, BlueField-4, Animesis CMA. The architectural thesis: AI memory infrastructure crystallized around object storage in 2026, and the persistence layer the industry settled on is S3. 2026-05-10 When the AI Stack Became an I/O Stack: S3 Vectors GA, Real-Time Lakehouses, and the May 2026 Storage Rewrite Amazon S3 Vectors hit GA at 20 trillion vectors per bucket. Apache Paimon is doing 40 million rows per second at ByteDance. Aliyun OSS embedded similarity search directly in the storage control layer. Three independent signals, one architectural truth: 2026's AI bottleneck is no longer compute — it's I/O, and the storage layer is being rewritten to absorb it. 2026-05-07 When DeepSeek Made Storage Strategic: The China Direction Reshaping the S3 Ecosystem DeepSeek's $5.6M training run didn't just reset model architecture. It made object storage performance-critical, made silicon export controls a storage decision, and pushed Aliyun OSS, Tencent COS, and Huawei OBS from regional curiosities into the global S3 vocabulary. This index added 13 nodes to track what changed.