Quickwit is a highly scalable, distributed and cheap log search engine. Or in simpler words: “Elasticsearch but on an object storage”. You can read the technical explanation of how it works on our blog.

We use it as the storage solution for our Vega Storage product (a direct competitor to Scanner).

Why Quickwit (and not Snowflake / ClickHouse / Apache Pinot)?

  • Simple on-boarding - Ingest existing data on S3 / Azure Blob Storage / GCS.
    • We configure Quickwit to index files by reading from an SQS receiving ObjectCreated events. These events are generated for each new file written to S3. No need for a copy step to ingest data, it is streamed.
  • Built for semi-structured logs (JSONs) - Simpler to use than VARIANT tricks in structured SQL databases.
  • Provides a helm template so we can run it On-Prem k8s.
  • Cheap.

We’ve forked the project, currently only for little missing features and configurations.

Important stuff to know about Quickwit:

  • A Quickwit index is comprised of:
    • Index - quickwit create index - Configure the index name, fields, timestamp field, and tokenizers.
      • Note that static fields are fields you explicitly set in the config, and dynamic fields are fields that you didn’t set. To control what to do about duynamic fields, you set the mode. We always set to dynamic (the default), so dynamic fields get indexed too, mimicking the behavior of other SIEMs (like Splunk).
    • Source - quickwit create source - An index by itself is only a configuration, to configure the indexer to pull data from some remote source (e.g. SQS), or use quickwit index ingest from the CLI. An index can have multiple sources.
      • A source can also contain a script that runs on each log, right before indexing. Simply add the transform.script in the config file, to run VRL code on each log. VRL is an interpreter written in rust for the Vector project, it’s pretty fast, but has some caveats. For example, don’t ever push to add to an array in a loop, it will re-allocate and copy the entire array in each iteration of the loop, making the script run on a 30MB array for more than 30 seconds, leading to quickwit killing the indexing pipeline for that index (until the next restart). Use map_values instead.
  • The trino connector uses the Elasticsearch API as it supports pagination and more types of queries (e.g. regexp).
  • The index configuration should always set a field called _raw, which utilizes the concatenate type. Together with the undocumented include_dynamic_fields: true, this special field will contain an inverted index of all tokens of all the other fields. This field is used when running the free text search command. It’s config should look like so:
doc_mapping:
  field_mappings:
    - name: _raw
      type: concatenate
      include_dynamic_fields: true
      tokenizer: whitespace
      record: position
  • regexp queries, according to the Quickwit devs, can have very different performance characteristics depending on the regex, because the regex runs on the FST storing the tokens, prefix queries (hello.*) are faster than suffix queries (.*world). For more info on the internals, read Search logs faster than Sonic.
  • query_string queries are simply converted to tantivy queries. When possible, use this instead of regex, it will be faster.
  • The tokenizer configuration is super important and a little confusing. A tokenizer splits fields into tokens, where each token is the key in the inverted index (posting list).
    • A little explanation about the important predefined tokenizers:
      • raw - The default tokenizer for dynamic fields. It doesn’t split the field into tokens, at all. Also filters out tokens larger than 255 bytes. The implications of not tokenizing are:
        • regexp queries run on each token, meaning in raw it runs on the entirety of a field. For example, if a field contains: hello there, world, you can search .*there, w.* and it will work.
        • query_string will only work if you provide the entire field, as again, there’s only 1 token per field.
      • default - The default tokenizer for static fields. Splits tokens by rust’s is_alphanumeric. Also filters out tokens larger than 255 bytes. The implications of tokenizing are:
        • regexp queries run on each token, meaning in default it runs on each word of a field. For example, if a field contains: hello there, world, you can search hell.* and it will find work, but .*there, w.* won’t. Unfortunately Quickwit’s regex queries don’t support position.
        • query_string will work as expected, you can provide a specific token like hello.
      • whitespace - Same as default but splits tokens by rust’s is_ascii_whitespace.
    • We currently use whitespace on _raw, because it lets us support queries on IPs (which contain dots). Ideally, we should index using default. To understand why, imagine a log that looks like: hello (world), with whitespace, the user can’t search for world, they must search (world).
    • We currently use raw for dynamic fields, because it lets us do any regex query. Ideally, we should index using default, as then we can run the faster query_string instead of regexp. The problem is that Quickwit currently only supports exact and prefix queries when using query_string.
    • You can define your own tokenizers in the index config, for example:
doc_mapping:
  tokenizers:
    - name: whitespace_lowercase
      type: whitespace
      filters:
        - remove_long
        - lower_caser

    - name: default_long
      type: simple  # simple == default
      filters:
        - remove_long_size: 4096

Dev Environment

When running in Tilt, we use Localstack S3 instead of connecting to a real object storage in the cloud. The S3 and sqs queue are being initialize on start automatically.

For loading fake data to quickwit you can use scripts/load_data_into_quickwit.py - it will load 10,000 events and give them a random date from the last week. It runs with VegaSecurity tenant by default - if you like you can pass the --tenant flag for other tenants.

Setting Quickwit monolith and ingest manually

In case you want to setup quickwit manually to test some feature with your own logs you can follow this easy steps:

  1. under /tmp run mkdir qw1
  2. under qw1 directory run:
    1. mkdir config
    2. cp <you quickwit path>/quickwit/config/quickwit.yaml config/quickwit.yaml
    3. mkdir qwdata
  3. get the test-quickwit-index-config.yaml or other index config and the actual .json of the log you are going to ingest (in this case its stackoverflow.posts.transformed-10000.json).
  4. run quickwit binary ./quickwit (you can run it in background or other terminal and see the prints/logs).
  5. run quickwit binary commands (after you compiled it with cargo build):
    1. ./quickwit index create --index-config '/<path to file>/test-quickwit-index-config.yaml'
    2. ./quickwit index ingest --index vegasecurity_regex_test --input-path <path to file>/stackoverflow.posts.transformed-10000.json --force where the .json file is what you are going to index.

Note: quickwit binary is under quickwit/target/debug directory after you compile itindex in.