Colpali Log

2025-2-8

Implemented the Lancedb indexing.

Big issue in modal container and lancedb:

• If we do the indexing on modal with lancedb, we will get indeies but encounter `

RuntimeError: lance error: LanceError(IO): Execution error: ExecNode(Take): thread panicked: task 25 panicked with message "called Result::unwrap() on an Err value: JoinError::Panic(Id(150),\"called Option::unwrap() on a None value\", ...)"

We got no choice, so we file a issue: bug(python): tbl.create_index(metric="cosine") causes Rust panic in Modal container, but works locally · Issue #2105 · lancedb/lancedb

2025-2-9

Reorganized the code. Now the code is clean.

Tested RuntimeError again by copying the local Indies to modal container. Still get same Error.

Next, try quantization.

Insight: Indexing is crucial here. In the Colpali case, indexing does not reduce accuracy. Additionally, even a single image can be indexed effectively since it generates 1,030 vectors, providing sufficient data for PQ (Product Quantization) to learn features. The more images available, the better the indexing performance, approaching 99.999% of native MaxSim computation accuracy.

2025-2-10

Test indexing performance on RTX 4090

Results for query: 'What is Sushan Wild party?'

This looks good, but each time indexing is different. Most of time the indexing will messing result. We need a solution for it.

2025-2-11

Went through the theoretical pipeline of how to build best ColPali.

Make a plan:

Stage 1

• Native ColPali Implementation
  at 2025-2-12

Stage 2

• Hybrid (HNSW + Rerank)
  at 2025-2-13

Stage 3

• Hybrid + Binary Quantization

2025-2-12

Native ColPali finished. test result:
dataset: test-pdfs

Query 1: What is Sushan Wild party?
Query 2: Which party got more women in 112th?
Query 3: Who are transformers paper's authors?

2025-2-13

• Hybrid (HNSW + Rerank)

Problem 1: upsert() of Qdrant has uploading limit which is 17. So we only process with for loop to upsert embeddings.

Today I implemented: HNSW, mean_pooling_columns and mean_pooling_rows and get prefetch.

Some questions has been answered:

Q1: How prefetch Works in Qdrant?

search_queries.append(
    QueryRequest(
        query=q_embedding,
        prefetch=[
            Prefetch(query=q_embedding, limit=200, using="mean_pooling_columns"),
            Prefetch(query=q_embedding, limit=200, using="mean_pooling_rows")
        ],
        limit=top_k,
        with_payload=True,
        with_vector=False,
        using="original"
    )
)

This means:
After Qdrant finds the top top_k matches from "original", it also fetches up to 200 entries from "mean_pooling_columns" and "mean_pooling_rows" that are related to those results.

1. Primary Search (using="original")
   Your search is performed only on "original", meaning that Qdrant finds the most similar vectors in that space.
2. Prefetching (prefetch=[...])
   After Qdrant retrieves the best matching points (data entries) from "original", it also fetches their related embeddings from "mean_pooling_columns" and "mean_pooling_rows". Prefetched vectors are not used for ranking but can be useful for additional processing.

Q2: Why Use prefetch with mean_pooling_columns and mean_pooling_rows?

How prefetch work?

2025-2-14

81 Imges

Search with prefetch

pipeline.search_with_text_queries.remote(queries, prefetch_size=20, top_k=3)

Query: What is Sushan Wild party?

• data_test_1.png (score: 10.005)
• Unifying_Multimodal_Retrieval_via_DSE_page_9.png (score: 9.797)
• Exploring_the_Limits_of_Language_Modeling_page_11.png (score: 9.752)
  Query: Which party got more women in 112th?
• data_test_2.png (score: 17.327)
• data_test_1.png (score: 14.920)
• Generating Sequences_With_Recurrent_Neural_Networks_page_29.png (score: 9.291)
  Query: Who are transformers paper's authors?
• attention_is_all_you_need_page_7.png (score: 12.354)
• attention_is_all_you_need_page_9.png (score: 11.396)
• attention_is_all_you_need_page_4.png (score: 11.261)

Search time: 0.12687s

Search without prefetch

pipeline.search_without_prefetch.remote(queries, top_k=3)

Query: What is Sushan Wild party?

• data_test_1.png (score: 10.005)
• Unifying_Multimodal_Retrieval_via_DSE_page_9.png (score: 9.797)
• Exploring_the_Limits_of_Language_Modeling_page_11.png (score: 9.752)
  Query: Which party got more women in 112th?
• data_test_2.png (score: 17.327)
• data_test_1.png (score: 14.920)
• in_context_scheming_reasoning_paper_page_13.png (score: 9.379)
  Query: Who are transformers paper's authors?
• attention_is_all_you_need_page_7.png (score: 12.354)
• attention_is_all_you_need_page_9.png (score: 11.396)
• attention_is_all_you_need_page_4.png (score: 11.261)

HNSW

hnsw_config=HnswConfigDiff(m=0) # HNSW switched off

Number of neighbours to consider during the index building. Larger the value - more accurate the search, more time required to build index.

Binary Quantization

Query: What is Sushan Wild party?

• Exploring_the_Limits_of_Language_Modeling_page_10.png (score: 1272.000)
• Exploring_the_Limits_of_Language_Modeling_page_11.png (score: 1272.000)

Query: Which party got more women in 112th?

• data_test_2.png (score: 1840.000)
• data_test_1.png (score: 1522.000)

Query: Who are transformers paper's authors?

• attention_is_all_you_need_page_7.png (score: 1396.000)
• attention_is_all_you_need_page_9.png (score: 1396.000)

Search time: 0.19166s

The result is not that perfect. BQ will change the accuracy of MaxSim searching.

2025-2-25

“Full Model vs. State Dict” in PyTorch

Finished classifier training.

2025-3-2

curl -X POST "https://tu-zhenzhao--stylemi-app-v2-api-service.modal.run/search?amount=5"
-H "Content-Type: multipart/form-data"
-F "file=@nail/183491c3-519c-446a-92d2-0e0955ff1eba.jpg"

2025-4-6

compared two classify_images_async

• Embedding one: 30mins
• pre-embeded one: 10s

For 1600 images

2025-4-23

Page not found · GitHub · GitHub

This code can runs large image dataset. Try to upload 20k images.

For 1000 batch: Memory 6.97GB, Low 3.41GB

Each hour with $2.03, ~4k images

$5 for 8k images

2025-5-1

Iteration 1

Using original method for reranking with prefilter on for final_df (.where(f"id IN {tuple_ids}", prefilter=True))

Iteration 2

Now indexed pooling_rows and pooling_cols , but no scaler and no original indexing.

Iteration 3

Now indexed pooling_rows and pooling_cols and scaler, but no original indexing.

Iteration 4

Now indexed pooling_rows and pooling_cols and scaler and original indexing.

original indexing time

2025-5-6

Key facts about our table

• 25 000 rows – each is a ColPali page.
• multivector columns (pooled_rows, pooled_cols, original) hold ≈1 030 token-vectors per row.
  • Only IVF-PQ + cosine is available for multivectors in LanceDB Cloud.
• id is high-cardinality (mostly unique). BTREE is the recommended scalar index.
• All calls are remote (RemoteTable) → every search is a separate HTTPS round-trip.

Baseline:

no indices

Plan shows full_scan: true, prefilter: false.

Add IVF-PQ on the two pooled columns

• What changed – the first two scans become ANN look-ups; each touches only a few hundred centroids + PQ codes.
• What did not change –
  • The IN (…) filter still has to walk the whole table (no scalar index).
  • Distance for the refine step still computed on all 25 k rows because the pre-filter has no rapid way to eliminate them.

The 9–10 s gain we observed is exactly the two full scans you removed.

Add BTREE scalar index on

id

• The WHERE id IN (…) PREFILTER now executes via the BTREE in O(|candidates| log N) instead of a table walk.
• Pre-filter runs before distance-computation, so only the 50 + 50 ≈ 200 candidate rows survive to the refine step.
  • Distance ops: 200 rows × 1 030 ≈ 206 k dot-products → a few ms.
• Network overhead still 3 RTTs, but each request body is only a few kB.

Query-to-query variation (1.6 vs 2.7 s) is mostly:

• size of the IN (…) list (Batman query produced ≈180 IDs, LLM query ≈310),
• job queuing on the shared Cloud worker,
• network round-trip jitter.

Execution plan now shows

VectorSearch
  index: IVF_PQ(cosine)          -- for pooled vectors
  prefilter: true
  row_count: 197                 -- refine touches only ~200 rows
full_scan: false                 -- ✅ no table-wide scan

Why we didn’t index the “original” column

• At 25 k pages it was already < 1 s after rerank; indexing “original” would shave only a few hundred ms.
• Keeping it flat lets us A/B the quality difference between full token-set vs. pooled ANN embeddings.

If we need even lower latency later (or many more pages), we can build an HNSW on a single-vector surrogate (e.g., CLS-token embedding) or switch prefetch_limit down to 50.

Take-away table

Size of Multi-vector

Eg:

Reranker. rerank number: 200