Reading Time: 2 minutes

Quickwit is an unparalleled cost-efficient log search engine designed to solve scaling issues in actual log search solutions.

Those issues are mainly originated by the exponential grow of all data, including log datasets: in a data-driven world, terabyte datasets are the new standard. And large tech and cybersecurity companies already need to manage petabytes of logs.

This mean that:

  • Infrastructure costs are prohibitive at terabyte and petabyte scale
  • Cluster management is overly complex
  • Indexing is slow and unstructured logs are poorly handled

Quickwit is fast, scalable, and reliable at petabyte-scale:

  • Built in Rust, and powered by Tantivy, a leading search engine library maintained by Quickwit.
  • With decoupled compute and storage (all data on object storage)
  • Stateless
  • Schemaless

Quickwit is at least 10x more cost-efficient than existing solutions and you don’t need a whole team to manage petabytes.

Quickwit architecture is scalable and efficient and it’s mainly divided in two parts:

  • Indexing process: Creates split files from JSON documents and uploads them to the object storage.
  • Search process: Sends precise range bytes queries to splits.

And there is still space for improving, according with their roadmap:

  • April 2024: Distributed ingest
    • High-throughput indexing on tens of thousands of indexes.
  • Q2 2024: OpenSearch Dashboard support
    • Enable OpenSearch users to migrate seamlessly to Quickwit with their existing dashboards.
  • Q3 2024: Pipe-based query language
    • Introduction of a flexible and powerful query language similar to SPL (Splunk Query Language)
  • Q4 2024 – 2025: Metrics support
    • New storage engine optimized for time series data.