Vector search in Postgres is an area that has actually seen really active advancement in the last couple of months. While Pgvector is understood to the majority of people, a couple of weeks ago we encountered Lantern, which likewise develops a Postgres-based vector database. We believed about benchmarking both to compare the 2 techniques. In this post, we’ll cover:
- Quick background on doing vector search in PostgreSQL.
- Comparing Pgvector and Lantern in regards to syntax and ease of usage.
- Standards comparing index production time, size, latency, throughput, and recall.
- Summary of the outcomes
Introduction to Vector Search in PostgreSQL
Among the factors for the appeal of Vector Search nowadays relates to the development of effective embedding designs and their usage in AI. You have actually most likely experienced the number of individuals, start-ups, and huge business are checking out how to benefit from vectors and integrate them into their items
Such interest has actually likewise drawn in the Postgres neighborhood. Pgvector emerged with the capability to produce IVVFlat indexes on existing tables with an easy DDL declaration. Therefore, individuals were offered the capability to quickly carry out resemblance inquiries.
Pgvector is not the only one in this area. The neighborhood has actually come up with other options. One such example was pg_embeddingIn July, Neon released a post where they demonstrated how its extension was 20x faster than Pgvector by utilizing HNSW indexes. Pgvector rapidly captured up and presented the HNSW index. Around that time, on September 29th, Neon stopped supporting pg_embedding and recommended moving to Pgvector.
Another Postgres extension in the vector search domain is Lanternestablished by a business of the exact same nameIn October, a post on their blog site declared that their extension might outshine Pgvector by 90x in index production time. That was an impressive accomplishment!
I invested some time playing with Lantern, and I ‘d like to share some of my findings in this blog site post. I divided it into 2 parts: the very first one is a fast qualitative contrast, and the 2nd one is a quantitative contrast utilizing a popular standard.
Let’s dive in.
PGVector vs Lantern: Similarities and distinctions
The following table sums up some elements of both extensions:
| Pgvector | Lantern | |
|---|---|---|
| Present Version (date) | v0.5.1 (Oct 10, 2023) | v0.0.11 (Dec 16, 2023) |
| Repo appeal (# of stars) | 7.6 K | 435 |
| Index Type | IVVFlat, HNSW | HNSW |
| Range Metrics | L2, Inner, Cosine | L2, Cosine, Hamming |
| Sample Index Creation | CREATE INDEX ON items USING hnsw (embedding vector_l2_ops) WITH (m = 16, ef_construction = 64); |
CREATE INDEX ON small_world USING hnsw (vector dist_l2sq_ops) WITH (M=2, ef_construction=10, ef=4, dim=3); |
| Test Query | SELECT * FROM items ORDER BY embedding <-> '[3,1,2]' LIMIT 5; |
SET enable_seqscan = false; SELECT * FROM small_world ORDER BY vector <-> ARRAY[0,0,0] LIMIT 1; |
My takeaway is that if you recognize with among the extensions, you can quickly comprehend how to utilize the other. No significant issues are anticipated on that front.
Leveraging ANN-Benchmarks
For a quantitative contrast, I took the ANN-Benchmarks and extended it to support the Lantern extension. It was simply a matter of developing a copy of the Pgvector directory site, calling it Lantern and making a couple of modifications to utilize the matching API.
The standard begins a container that has actually Postgres 15 set up and after that it allows the matching extension. It inserts a dataset into a table and constructs a vector index (in both cases an HNSW index). After that, it performs a lot of inquiries and examines the recall.
Along the method, the benchmark gathers numerous metrics, such as:
- Develop Time
- Index Size
- Remember
- Throughput
- Latencies (p50, p95, p99 and p999)
We can compare the 2 extensions along these axes.
When it comes to the HNSW specifications, I utilized the following:
| ef_construction | [128]200 |
| m | |
| ef_search | 10, 20, 40, 80, 120, [128,] 200, 400 |
Comparing Index Creation, Recall, Latency and Throughput
I attempted various datasets, for brevity, let us just talk about the outcomes of the SIFT 128 dataset, which was likewise utilized in one of Lantern’s posts
These are the numbers I managed following the directions in the post, utilizing comparable building and search specifications (e.g. m={8, 16} ef_construction=128 and ef_search=128. Here I utilized Lantern’s external indexer. Let us see the construct time:
Pgvector takes in between 1.71 x and 1.73 x to construct the very same index with the exact same specifications. The resulting index is in between 13% and 15% larger in Pgvector:
That’s cool … And what about the resulting throughput, latency and recall? Here are the outcomes:
Ok, the recall is comparable, nevertheless Pgvector exceeds Lantern in QPS and latencies. Particularly, Pgvector can process in between 44% and 53% more questions per 2nd, and it’s latencies are in between 29% and 35% smaller sized.
It appears that Lantern sacrifices throughput in exchange of index production time and index area.
Hold on … however is this real for other information points? Let’s see.
With the building and construction criteria m = {16, 24} and ef_construction = {200}the production time is still much better with Lantern: in between 1.9 X and 2.3 X. And the resulting index is larger in Pgvector: in between 1.13 X and 1.20 X compared to Latern’s index.
As in the past, the recall is comparable when ef_search differs (although Pgvector’s is a bit much better). This is the chart for building and construction criteria ef_construction=200 and m=16As the ef_search is increased, the recall of both indexes get closer.
Pgvector has 62-84% much better throughput and 38-45% much better latencies:
Increasing the m specification to 24, we get comparable conclusions:
And when again, Pgvector has 42-58% greater throughput and 30-39% much better latencies for all worths of ef_search:
The basic patterns followed my observations when utilizing Gist-960 and Glove-200. You can see more outcomes here
Conclusions
For benefit, the following table sums up the above outcomes utilizing relative numbers (i.e. pgvector/lantern:
| m= ; ef_construction=128; ef_search=128 | m= 16,24; ef_construction=200; ef_search=[10-400] | Notes | |
|---|---|---|---|
| Construct Time | 1.71X-1.73 X | 1.92X-2.35 X | Lantern yields much better outcomes |
| Index Size | 1.13X-1.15 X | 1.13X-1.20 X | Lantern yields much better outcomes |
| Latency | 0.64X-0.70 X | 0.54X-0.69 X | Pgvector yields much better outcomes |
| Throughput | 1.44X-1.53 X | 1.42X-1.84 X | Pgvector yields much better outcomes |
| Remember | 1.00X-1.01 X | 1.00X-1.09 X | Remember is comparable, Pgvector is a little much better |
Pgvector is the most popular Postgres extension for vector search. At the time of this writing, the github repository counts 7.6 K stars and is actively being talked about on the internet. It is likewise supported on a lot of handled Postgres suppliers (consisting of Tembo CloudIs much easier for you to gain access to.
Lantern is a young job that leverages the popular USearch engineSince today, the extension has actually been starred more than 400 times in github and remains in really active advancement.
Both extensions use comparable API and support the HNSW index. According to my experiments (Pgvector 0.5.1 and Lantern 0.0.11), Lantern’s index development is quicker and produces smaller sized indexes. Pgvector supplies much better recall, latency and throughput.
We must watch on both tasks and see how they progress. I am positive that we’ll see a number of enhancements in the following months.
Oh, and if you have an interest in rapidly incorporating vector searches in your app, ensure to have a look at pg_vectorize to see how to do that with just 2 function contacts Postgres.
Appendix
The experiments in this post were performed utilizing a maker with the following qualities:
| VM | E2-standard-8 (8 vCPUs, 4 cores, 32GB memory) |
| Storage | 100GB |
| Running System | Debian 11.8 |
| Postgres | 15 |
| Postgres Config | maintenance_work_mem=5GB; work_mem=2GB; shared_buffers=12GB |
| Lantern | 0.0.11 |
| PGVector | 0.5.1 |
For info about other outcomes, please see here