Quickwit is an unparalleled cost-efficient log search engine designed to solve scaling issues in actual log search solutions.

Those issues are mainly originated by the exponential grow of all data, including log datasets: in a data-driven world, terabyte datasets are the new standard. And large tech and cybersecurity companies already need to manage petabytes of logs.

This mean that:

• Infrastructure costs are prohibitive at terabyte and petabyte scale
• Cluster management is overly complex
• Indexing is slow and unstructured logs are poorly handled

Quickwit is fast, scalable, and reliable at petabyte-scale:

• Built in Rust, and powered by Tantivy, a leading search engine library maintained by Quickwit.
• With decoupled compute and storage (all data on object storage)
• Stateless
• Schemaless

Quickwit is at least 10x more cost-efficient than existing solutions and you don’t need a whole team to manage petabytes.

Quickwit architecture is scalable and efficient and it’s mainly divided in two parts:

• Indexing process: Creates split files from JSON documents and uploads them to the object storage.
• Search process: Sends precise range bytes queries to splits.

And there is still space for improving, according with their roadmap:

• April 2024: Distributed ingest
  • High-throughput indexing on tens of thousands of indexes.
• Q2 2024: OpenSearch Dashboard support
  • Enable OpenSearch users to migrate seamlessly to Quickwit with their existing dashboards.
• Q3 2024: Pipe-based query language
  • Introduction of a flexible and powerful query language similar to SPL (Splunk Query Language)
• Q4 2024 – 2025: Metrics support
  • New storage engine optimized for time series data.