Comparing built-in full-text search, an embedded Tantivy index, and a new database as a sidecar, with a staged rollout for each.
Our catalog API answered structured filters well but "find the thing called roughly this" badly — ILIKE '%term%' scans that missed typos and ranked nothing.
Three options, compared
- Built-in FTS (DuckDB's extension): no new dependency, decent ranking, limited typo tolerance. Ship first, measure.
- Embedded Tantivy index: proper fuzzy matching and relevance, at the cost of building and syncing an index. The pattern is a split — Tantivy finds and ranks IDs, the database fetches and joins rows.
- A search sidecar (Elasticsearch/OpenSearch): most capable, most to operate. Only when the first two provably fail.
The lesson: put search behind an interface so the API layer doesn't know which backend answered. Then you can start with the cheapest option and swap engines behind the same contract as requirements sharpen. Names aren't prose — numerals, subtitles and aliases matter more than which engine you pick.