Elasticsearch 8.11 improves how it manages its indexing buffer, resulting in less segment merging.

How it works in Elasticsearch 8.10 and earlier

Change 1: flush a single segment at a time

Change 2: flush shards in a round-robin fashion

Conclusion

Less merging and faster ingestion in Elasticsearch 8.11

Lucene 9.9 was released recently and brings major speedups to query evaluation

Apache Lucene 9.9, the fastest Lucene release ever

Improve the MAXSCORE algorithm to evaluate disjuntive queries more like a conjunctive query when possible, which helps evaluate even fewer hits

More skipping with block-max MAXSCORE

In this blog, you'll learn how vector search has been integrated into Elasticsearch and the trade-offs that we made.

Vector search is integrated in Elasticsearch through Apache Lucene

Cons

Merges need to recompute HNSW graphs

Searches need to combine results from multiple segments

RAM needs to scale with the size of the data set to retain optimal performance

Pros

Data sets can scale beyond the total RAM size

Lock-free search

Support for incremental changes

Visibility consistency with other data structures

Incremental snapshots

Filtering and hybrid support

Compatibility with other features

Looking ahead: Separation of indexing and search

The main cons of taking advantage of Apache Lucene for vector search come from the fact that Lucene ties vectors to segments. These cons include:  1) Merges need to recompute HNSW graphs. 2) Searches need to combine results from multiple segments. 3) RAM needs to scale with the size of the data set to retain optimal performance. However, tying vectors to segments is also what enables major features such as efficient pre-filtering, efficient hybrid search, and visibility consistency.

What are the cons of using Apache Lucene for vector search?

The pros of taking advantage of Apache Lucene for vector search is that it enables features such as efficient pre-filtering, efficient hybrid search, visibility consistency, among others.

What are the pros of using Apache Lucene for vector search?

Vector search is integrated into Elasticsearch through Apache Lucene.

How is vector search integrated into Elasticsearch?

Vector search in Elasticsearch: Integration & the design rationale

Vector search in Elasticsearch: The rationale behind the design

2023 has been another big year for Apache Lucene, this blog reviews major milestones of 2023

Author

Adrien Grand

Articles

Less merging and faster ingestion in Elasticsearch 8.11

Apache Lucene 9.9, the fastest Lucene release ever

More skipping with block-max MAXSCORE

Vector search in Elasticsearch: The rationale behind the design

What happened in Lucene land in 2023?