Checkout the Groundcover network-map of Vega, filter only our production cluster (production-us-east-2) and play with the options to better understand the different components and how they interact with each other.

Not all microservices are documented yet, here are some of the most relevant ones:

Gateway

services/go/gateway

The gateway to all requests coming in. It’s the only service that receives connections from outside the cluster.

It sits behind AWS ALB, doing HTTPS termination and WAF for us.

Responsibilities:

  • Handle all incoming HTTP API requests.
  • Most APIs are handled by the GraphQL endpoint api/v1/query.
    • Schema file located in services/go/gateway/api/v1/services/endpoints/schema.graphql.
  • Authentication & Authorization.
  • Audit logs.

The gateway is the most important service to keep lean from features, this is because it must always be up and running to handle any request that might come in.

Other services can be designed to not have 100% uptime, as they can for example read from persistent storage like a queue. The gateway doesn’t have this luxury.

Something to note is stateful connections to users. What to do if you need to keep an incoming connection from the user open? For example, streaming events to a GraphQL subscription.

There are 3 solutions to this:

  • Direct - Connect to the service responsible to handling the stream via gRPC (which supports streaming), and yield whatever that service streams directly to the open user connection.
  • Indirect - Register an endpoint (gRPC / HTTP) that receives an event and an id that connects it back to the connection, look up the connection by the id, and send the event to that connection. You must remember that the gateway has horizontal scaling. Connections are stateful in the gateway’s pod memory. This means that you need to send the event to the correct gateway pod. This can be solved by making the ingress load balancing method to be based on a hash of the id (nginx supports this via hash load balancing).
  • Indirect Pub/Sub - Same as indirect, but instead of gRPC / HTTP, you subscribe to a Pub/Sub channel, and filter out events to ids that are not owned by you. Because all gateway pods receive each event, you don’t need to concern yourself with load balancing.

If possible, choose the direct method, it’s the simplest.

TrinoManager

services/go/trinomanager

Some functionalities are easier for us to implement in Go code, as we can utilize common. This is where trinomanager helps.

Responsibilities:

  • Expose utility gRPC endpoints to be used by trino. For example: querying the database for all connector instances to create trino catalogs.
  • Run queries over trino. When deploying trino, trino-coordinator gets restarted, dropping all running queries. We use temporal to retry a trino query until it’s finished.

Endpoints

services/go/endpoints

Responsibilities:

  • Safe API calls - Exports a gRPC endpoint to make API requests in a fault tolerant manner (exponential back-off retries) using temporal. In addition, if a specific endpoint also needs authentication, it will make the required API calls to get a session token, and store it in the database to be reused by other API invocations to the same endpoint.
  • CRUD for connectors - This service is the owner of all the models concerning connectors (e.g. connectors, data sources, endpoints).

Vega Monitoring

services/go/vega-monitoring

Active monitoring service that continuously validates the health of our federated query system and connectors. It runs in our production clusters and monitors the vegae2e tenant.

Dashboard: Query Healthiness Dashboard

How It Works

The service runs actual user query flows (not just connection tests) to ensure the full query path is healthy:

Federated Query Healthiness (every 10 seconds):

  • Executes a real federated query via GraphQL on the postgres monitoring connectors
  • Queries: @postgres_federated_query_active_monitoring/health_check | take 1 (and the _miso variant on the miso tenant)
  • Validates end-to-end query execution through online Trino
  • Legacy elastic/splunk connector probes are deprecated: those connectors stay on the e2e tenants for manual testing only

Connectors Healthiness (every 60 seconds):

  • Lists all connectors in the vegae2e tenant
  • Runs a federated query with take 1 on each data source
  • Ensures all configured connectors are queryable

BFF Healthiness:

  • Checks the health endpoint of the frontend service
  • Validates that the BFF can reach the Gateway

Architecture Notes

  • All monitoring queries use the GraphQL API (same path as real users)
  • Queries are routed to online Trino as they simulate online user query flows
  • Metrics are exported to Datadog for alerting and dashboards
  • The service uses a Datadog monitor for alerting

Configuration

Key environment variables (set via Helm values):

Variable Description Default
FederatedQueryHealthinessEnabled Enable federated query monitoring false
FederatedQueryHealthinessInterval Interval in seconds 10 (production)
FederatedQueryHealthinessEngineTable Data source to query @postgres_federated_query_active_monitoring/health_check
FederatedQueryHealthinessHost Target environment URL https://app.vega.io
VegaStorageHealthinessInterval Interval for connector checks 60

Metrics

  • federated_query_healthiness_seconds - Query execution time histogram
  • bff.healthiness.count - BFF health check counter
  • bff.healthiness.duration - BFF health check duration