On-Call Goals

  • Keep customer impact low
  • Restore service quickly and safely - REVERT FIRST, FIX LATER
  • Leave the system better than you found it

If You Are Stuck

  1. Start with the Alerts section below
  2. Reduce scope: pick one service, one tenant, one queryId
  3. Ask in the incidents channel for help

Getting Started

Adding On-Call Schedule to Your Calendar

Not sure when you’re on call? Follow these steps:

  1. PagerDuty: Go to People → My On-Call Shifts → Export Calendar → WebCal Feed
  2. Google Calendar: Add another calendar → From URL → Paste the address starting with webcal://

Your on-call schedule will now show up in your calendar.

Setting Up PagerDuty Reminders

Get notified before your shift starts or ends:

  1. Go to PagerDuty → My Profile → Notification Rules
  2. Configure the “Before I go on-call or off-call…” section

On-Call Coverage

The on-call shift covers 24 hours a day, 7 days a week.

Nighttime Alerts

We aim to limit night-time alerts to only real customer-affecting issues. The main alerts triggering night-time on-call will be:

Alert Type Description
E2E Continuous Failures Only at a critical threshold
Queries Hang Trino queue issues
Query Health Vega monitoring service (federated query health checks)

We also have a “red button” 🔴 escalation path, available to a few designated people, with strict rules on when it may be used.

Dashboards

Quickwit

Dashboard Link
Searcher Quickwit Searcher Dashboard
Indexer Quickwit Indexer Dashboard

Trino

Dashboard Description
Online View Online user queries
Offline View Offline queries

Vega

Dashboard Link
Vega App Vega Dashboard
Query Healthiness Query Healthiness Dashboard
Historical Hits Jobs Historical Hits Jobs Dashboard

About Query Healthiness Dashboard

This dashboard is based on our own service called “vega-monitoring” that actively queries our system using federated query GraphQL.

  • Once a minute: Lists all connectors in the vegae2e tenant and runs a federated query with take 1 on each data source
  • Every 10 seconds: Runs a federated query on the postgres_federated_query_active_monitoring connectors (per tenant)
  • All queries are routed to online Trino as they follow online user query flows

Filtering Noise (Logs & Traces)

Use these filters to narrow down issues:

  • tenant_name
  • queryid
  • user (if applicable)

Service-Specific Debugging

Trino

See the Trino Debugging Guide for detailed troubleshooting steps.

Trino Query Logger

Trino query events are logged to S3 via Vector and can be queried for debugging purposes.

Option 1: AWS Athena

Query the following tables directly in Athena:

Table Description
trino_query_events.online_events Online user query events
trino_query_events.offline_events Offline/background query events
trino_query_events.unmatched_events Events that didn’t match online/offline patterns

Option 2: Vega App

Query directly from the Vega app using the S3 connector:

  • Connector: trino-queries (in vegae2e tenant)

Quickwit

Check the Quickwit Searcher and Indexer dashboards for performance metrics.

Temporal

Known features that run in Temporal:

  1. Federated Query
  2. Detections

You can search queries in Temporal by the query id as the Workflow ID:

Temporal Workflows

E2E Tests

Understanding E2E Failures

When E2E tests are running (either during deploy on staging or periodically on staging), there’s a Playwright report that you can download to see what tests failed/flaked.

Key details:

  • Tests have 4 retries today
  • After the first unsuccessful try, Playwright records the screen and web trace

Accessing Playwright Reports

  1. Download the Playwright zip from the GitHub Actions artifacts
  2. Open the Playwright HTML report
  3. Download the trace zip
  4. Open it via command line or web UI:
npx playwright show-trace <path-to-trace.zip>

Or use the web interface: trace.playwright.dev

E2E Monitoring Dashboard

View patterns of failures and flakiness over the last hours/days:

Datadog E2E Monitoring Dashboard

There’s also a scheduled message sent at 9am every day in #main-stability that reflects the status from the past day.

Quick Reference

What Where
On-call schedule PagerDuty → My On-Call Shifts
Main dashboards Datadog (links above)
Temporal workflows cloud.temporal.io
E2E test results GitHub Actions artifacts + #e2e-tests-prod
Get help #incidents channel