Technology
Cachey
Read-through cache for S3-compatible storage (Rust, hybrid memory+disk)
2 connections
Definition
What it is
Read-through cache for S3-compatible storage (Rust, hybrid memory+disk)
Recent developments
Latest signals
- Utilizing KV Cache for Sampling and Reasoning. HuggingFace / Vals AI Benchmark / Datasets, 2025. KV cache within original layer-wise CoE design yields poor performance. Per arxiv.org.
- LLM.int8():8-bit Matrix Multiplication for Transformers at Scale. End-to-end benchmarks for BLOOM 176B in Hugging Face using optimized implementation with cached attention values. Int8 inference close to millisecond latency per token vs 16-bit. Per arxiv.org.
- arxiv.org. Table 5: Inference latency(ms) comparison across models, frameworks, and sequence lengths. Tests conducted on HuggingFace single-GPU deployment with MetaX C550 GPUs and on vLLM with single- and multi-GPU settings. Per arxiv.org.
- GitHub. HF Torch Cache. License: MIT Python 3.10+. Efficient caching layer for Hugging Face models using PyTorch serialisation. Accelerate model initialisation while reducing disk redundancy by converting native Hugging Face checkpoints to optimised PyTorch format. Per GitHub (lmmx/hftorchcache) (2025-01-29).
Connections 2
Outbound 2
scoped_to2