How I Built Semantic Search Over 300M+ Academic Papers

I build Rivul AI, a free research workspace (project page). One part of it is paper search: semantic search over metadata for 300M+ academic papers. The index is built from the OpenAlex snapshot, available as open data (CC0) on the AWS Open Data Registry.

The usual answer at this scale is a distributed vector database. I was curious whether a single machine could work for a read-heavy, rarely updated corpus like this one. This post covers what worked, what did not, and the numbers I measured.

A quick note on numbers. OpenAlex indexes about half a billion scholarly works. My index covers 300M+ of them. It leaves out OpenAlex's expansion records, keeps only works with a usable title, and removes older copies of works that appear more than once in the snapshot.

1. Why Single-Stage Retrieval Breaks Down at Scale

The embedding model I use, BGE-small (bge-small-en-v1.5), produces 384-dimensional vectors. Each paper record becomes one dense vector, built from its title and abstract (or the title alone when there is no abstract). The index searches that metadata, not the papers’ full text. Stored as float32, one vector takes 1,536 bytes. Multiply that by 300M+ and the raw vectors alone run to hundreds of gigabytes, before any index or metadata. Comparing every query against all of them is not realistic on one machine.

So I split the work into stages. Each stage is cheap enough to run on the output of the one before it, and each one is more precise than the last. That is the funnel.

2. Architecture: The Multi-Stage Retrieval Funnel

Technically, it is a multi-stage retrieval system: dense vector retrieval for candidates, then re-scoring, exact-title matching and metadata-aware ranking.

1Query embeddingBGE-small, 384 dimensions

2IVF-PQ candidatesAbout 1,000 from 300M+ records

3Calibrated INT8 re-scoringCosine on candidates only

4Exact title and prefix mergePasted titles land first

5Citation and recency blendCitations and recency nudge the order

• Query embedding. The query is embedded with BGE-small, using the instruction prefix the model recommends for retrieval queries: "Represent this sentence for searching relevant passages:".
• IVF-PQ candidates. FAISS returns about 1,000 candidates from an IVF-PQ index. IVF splits the vector space into lists around learned centroids and scans only the lists nearest the query. PQ compresses each vector into a 32-byte code: 32 sub-quantisers of 8 bits each. On a 1M-record test corpus, scanning more lists raised mean Recall@20 from 0.91 to 0.97 at 1,000 candidates, with very little extra search time.
• Calibrated INT8 re-scoring. The candidates are re-scored against a second copy of every vector, stored in INT8 at 384 bytes, one byte per dimension. Each dimension has its own scale, fitted from a calibration sample, so every dimension uses the full range of 127 levels on each side of zero. Only the candidates are dequantised before the cosine similarity is computed.
• Exact title and prefix merge. Vector search is weak at something people do all the time: pasting the exact title of a paper. So titles are also looked up directly. An exact normalised title match gets a large boost. A query that matches the start of a title pulls in up to 50 works, chosen by citations from up to 50,000 stored matches, with a smaller, bounded boost. A paper that is semantically much closer can still win.
• Citation and recency blend. Citations and recency nudge the final order. Citations are log-scaled and saturate at 100,000, so the weight is spread across the range real papers occupy. Recency adds a small boost that grows with the publication year.

3. Adaptive Ranking: Normalizing Boosts by Score Spread

This one surprised me. Some queries have candidates packed within 0.02 cosine of each other. Others spread across 0.2. A fixed citation weight means something very different in each case. In a tight cluster it overwhelms similarity. In a wide one it barely registers.

It matters at this scale because thousands of works cluster around any concept. Without some weight on citations, a preprint with no citations can outrank a landmark paper on a 0.03 similarity edge. With too much weight, famous papers crowd out the right answer.

So the boosts are scaled by the spread of each query's candidate scores. The weights are set at a reference spread of 0.10: 0.15 for citations and 0.03 for recency. A query's actual spread scales them, clamped between 0.25 and 2 times. A tight query and a wide query now get the same relative pull.

4. Filtering Strategy: Choosing Paths From Measured Costs

Filters look simple: a year range, a minimum citation count, open access, a topic or a field. At this scale they are not. If you filter a fixed candidate list after the fact, a selective filter leaves almost nothing. In one test, asking for 8 times the usual candidates with a filter that matched 0.04% of the corpus left 3 usable candidates out of 8,000.

So I measured the two options instead of guessing:

Operation	Measured cost
Ask FAISS for 1,000 candidates	about 11 ms
Ask FAISS for 64,000 candidates	about 20 ms
Ask FAISS for 256,000 candidates	about 56 ms
Score eligible rows one by one	about 100 µs per row

Asking for more candidates is nearly flat in cost. Walking eligible rows by hand grows steadily. That gave three paths:

• A filter that matches at most 10,000 rows is scored exactly, row by row.
• Otherwise the candidate request is scaled by how rare the eligible rows are, from 8 times up to 512 times.
• If the page is still thin because the matching rows sit in other parts of the vector space, the search probes more lists, once.

5. Relevance Floors for Attribute-Based Sorting

Sorting by most cited, newest or oldest showed a different problem. The candidate pool always holds some loosely related works. Sort a title search by citations, and a famous but unrelated paper that merely landed in the pool jumps to the top.

So attribute sorts only order rows whose similarity is within half of the candidate score spread from the best match. Exact title matches always stay in. Weaker matches are left out rather than appended at the end, where they would restart the order part-way down the list.

6. Data Quality: Immutable Index, Read-Time Corrections

The built index never changes. Rebuilding it takes a long time, so I wanted a way to correct individual records without touching it.

Any scholarly corpus of 300M+ records that merges many upstream sources, such as DOIs, publisher deposits and record merges, will have some messy entries. That is the nature of data at this size. The categories I ran into:

• A record whose publication year did not match when it started being cited.
• A citation count that looked implausible for the paper shown.
• A landmark paper whose conference version was missing from the snapshot I built from.

Two small files handle these at read time.

Overrides patch the display fields of a named work. Identity and the normalised title used for exact lookup cannot be changed, so a correction can never redirect one work to another. Corrected values are used for ranking, sorting and filtering too. Citation counts are never replaced, because there is no verified replacement value. A doubtful count is flagged instead: ranking and citation filters treat that work as uncited, while the card still shows the published number.

Supplements add up to 500 hand-verified records for works missing from the snapshot. They are embedded at startup with the same query model and merged at query time. A supplement record joins semantic results only when it scores at least as well as the weakest candidate the index returned, so it appears only where it would have been retrieved anyway.

Every entry records the sources it was checked against and when. Fixes like these can also be reported back to OpenAlex, so they help everyone who uses the data.

7. Performance: Measured Latency

On 2026-09-25 I ran 30 varied queries against the live index, 20 results each, sorted by relevance. Server-side search time had a median of 245 ms and a p95 of 275 ms. That is the time the index reports for the search itself, not the round trip over the internet. Thirty queries is a small sample, so I read it as a snapshot, not a benchmark. All of it runs on one machine with a fast NVMe drive.

8. Lessons Learned and What I'd Change

• Design for updates earlier. I built the index as an immutable snapshot first and treated updates as later work. Adding and removing records without a rebuild is the harder problem, and I would plan for it from the start.
• Re-tune at full scale sooner. I grew the index in stages, from 10K records up to the full corpus, with quality checks at each step. Small samples can flatter compression, and some search settings tuned at 1M records were carried to the full corpus. I would measure them again at full size earlier.
• Keep two kinds of benchmark from day one. Natural-language queries and exact-title lookups fail in different ways. I measure them separately, and one blended average would have hidden both problems.
• Build the correction path before launch. Read-time overrides and supplements turned out to be one of the most useful parts of the system.

9. Resources

• Rivul AI, free for every account
• Rivul AI project page
• How Search papers works
• OpenAlex, the open dataset behind the index
• OpenAlex snapshot on the AWS Open Data Registry