You Might Not Need a Vector Database for the First Pass

If you are building on your own object storage, the question that decides your architecture is not "which vector database" or "which query engine." It is how many of them you have to run at once. Every separate system in the retrieval path is another thing to deploy, keep in sync with the bytes in your bucket, and pay for while it sits idle. The releases worth your time this month are the ones that remove a box from that diagram rather than add one.

Two shipped in October, from opposite ends of the stack, moving toward the same place. ClickHouse 26.10 pushes filtering and ranking down into the Parquet and Iceberg readers themselves — the analytical engine reaching into the file layer to do work a vector database used to own. Qdrant 1.20.0 moves the other way, taking on quantization and full-text scoring to defend the standalone niche as it narrows. Read them together; the gap between them is closing.

The reader stopped reading the whole file

The headline in ClickHouse 26.10 (shipped around 25 October) is that Top-K dynamic filtering now pushes GROUP BY … ORDER BY … LIMIT boundaries directly into the Parquet and Iceberg readers.1 As a feature that sounds like planner trivia. As numbers it is not: reader scan time drops from 0.37 s to 0.077 s, and total bytes scanned on the Top-K query falls from 9.45 GB to 144 MB.1 That second number is the one to sit with. The engine reads roughly 1.5% of what it used to, because the reader now knows the query only wants the top rows and prunes row groups and data files before touching them, instead of streaming everything up to the engine and sorting at the top.

The same release does the string-filter version: LIKE, startsWith, endsWith and position filters are applied during deserialization, cutting peak query memory on those scans from 275 MB to 5 MB.1 Different mechanism, same principle — decide what you don't need before you pay to materialize it. And it ships inline binary quantization: arrayPackBitsToFixedString packs a 768-dimension float32 embedding, normally 3,072 bytes, into a 96-byte FixedString, with bitHammingDistance to compare two of them in place.12 One bit per dimension, a 32× shrink, the distance run inside the engine over your object-store tables.

This is the pattern this index files under decoupled vector search: the format stops being a dumb byte container the engine scans linearly, and becomes something the reader interrogates. You get the benefit without changing your storage — the bytes are still Parquet or Iceberg in a bucket you control. A query that used to justify a cluster because it scanned 9 GB now scans 144 MB and fits on one box.

A vector prefilter you don't need a vector database for

Here is what the quantization buys you, concretely: a coarse nearest-neighbour prefilter can now run in SQL over Parquet or Iceberg without standing up a vector store for the first pass. Binary-quantize your embeddings into a column, Hamming-rank to a few thousand candidates with the new Top-K pushdown doing the heavy pruning, and only then — if you even need to — hand that shortlist to a precise re-rank. For a lot of retrieval workloads that first pass is the workload. It is the shape pgvector made familiar — keep the vectors where the rest of your data already lives — arriving in the analytical engine, over object storage, at a scan footprint that was not previously on the table. Observed from the release notes; whether it displaces a dedicated store for your latency budget is yours to measure, not mine to assert.

Two operational notes before you upgrade for any of this. ClickHouse 26.10 makes default compression size-aware: parts under 100 MB stay on LZ4, parts at or above 100 MB default to ZSTD(3).1 Smaller objects on your S3 tier, more CPU at merge time — budget it rather than discover it. And the reason to take this release regardless of the features: CVE-2026-116366, an unauthenticated remote-code-execution hole in network-accessible ClickHouse Server, was disclosed on 30 October and is fixed in 26.10.3 If your server is reachable from anything you don't trust, that is the upgrade; the Top-K pushdown is a bonus.

The standalone vector DB adapts rather than retreats

From the other side, Qdrant 1.20.0 (5 October) is what a dedicated vector database does when the generalists start absorbing its job. It adds 8-bit and 16-bit TurboQuant vector datatypes and BM25 full-text scoring directly in the engine, plus per-element nested payload indexing and a payload-storage configuration that supersedes the old on_disk_payload flag.4

Put those beside ClickHouse's. Quantized vectors and built-in keyword scoring are exactly the primitives the analytical engines are now reaching for — Qdrant is building toward the same quantize-and-hybrid-rank shape from the vector-native direction. That is a vendor specializing, not a vendor in trouble. The standalone RAM-first vector database isn't disappearing; it's narrowing to the latency-critical serving tier, where a warm in-memory index with tail-latency guarantees genuinely beats a scan over object storage. Qdrant is sharpening that edge — faster payload writes, tighter quantization, feedback-aware queries — rather than pretending the lake-resident pattern isn't coming for the batch and mid-latency tiers.

What to actually do with this

For someone running local-first S3 infrastructure, the decision this month is cleaner than it was last quarter:

  • Push the first filter-and-rank pass into the engine, over your existing tables. If you already query Parquet or Iceberg with an analytical engine, Top-K pushdown and binary-quantized Hamming prefiltering let a lot of retrieval happen in place. You don't need a second database to find candidates; you need one to serve them fast, and only once latency forces it.
  • Keep the dedicated vector store for the hot path, not the whole path. Qdrant 1.20's niche is real: sub-millisecond, high-QPS, feedback-ranked serving. That's a smaller surface than "all vector search" — run it for what it's uniquely good at, not reflexively for everything.
  • Treat the compression and CVE changes as the reason to move now. The size-aware ZSTD default and CVE-2026-116366 are upgrade-forcing on their own; the pushdown and quantization come along for free once you're on 26.10.

None of this needs the parts that aren't here yet. There is no shipped native VECTOR type in Iceberg, no Parquet physical-type for quantized vectors, no S3-side vector primitive in this wave — so none of the above rests on them. It rests on two releases that already shipped: an engine that reads less of your bucket to answer more, and a vector engine that got denser and learned to score text.

Bottom line

Two shipped releases, pushing the same direction from opposite ends — the engine reading less, the vector store getting denser. If you track these calls, the scoring and its revision conditions live on the ledger; this post is about what these releases let you build.

Works cited

Footnotes

  1. Changelog 2026 — ClickHouse Documentation — ClickHouse 26.10 (~25 October 2026): Top-K dynamic filtering pushdown into Parquet and Iceberg readers (0.37 s → 0.077 s scan; 9.45 GB → 144 MB scanned), deserialization-time string filtering (275 MB → 5 MB peak memory), inline bit-packing functions, and the size-aware LZ4→ZSTD(3) default compression switch (PR #108786). ↩ ↩2 ↩3 ↩4 ↩5

  2. ClickHouse 26.10 Release Call: 104 New Features, 2× Less Hash — the official release presentation demonstrating arrayPackBitsToFixedString packing a 768-float32 embedding into a 96-byte FixedString and bitHammingDistance for in-engine similarity (vendor-published). ↩

  3. CVE-2026-116366 — ClickHouse, Inc. — unauthenticated remote code execution against network-accessible ClickHouse Server, disclosed 30 October 2026, fixed in 26.10. ↩

  4. Releases · qdrant/qdrant — GitHub and Qdrant 1.20 — Full-Text Search Improvements — Qdrant 1.20.0 (5 October 2026): 8/16-bit TurboQuant vector datatypes, BM25 full-text scoring, per-element nested payload indexing, and a payload-storage configuration superseding on_disk_payload. ↩