The Broker facilitates connectivity to client on-perm systems that are not directly accessible from the internet.

Overview

The Broker consists of the following key services:

  1. DNS Server
  2. tun2proxy
  3. FRP Server (frps)
  4. FRP Client (frpc)
  5. Broker Tunneler

Service Architecture

  • The DNS Server and tun2proxy services run as sidecar containers alongside Trino Coordinator and Worker pods.
  • FRP Server and FRP Client are deployed as separate instances for each client application in our cluster.
  • The Broker Tunneler is a single pod running in the Vega cluster, responsible for routing and managing broker clients traffic.

When creating a new connector instance, users can mark it as an on-premises system. This action provides the following installation command to set up the Broker on the client’s on-perm environment:

export BROKER_ID=K2qi6qd0m8ZonnxnStn1KLtIstkO
export BROKER_ACCESS_KEY=K2qi6qd0m8ZonnxnStn1KLtIstkO3axgJ74kEO3oMv4ji7HprCAXoslx8f1xoUgo4dpPkf0
curl -s https://app.vega.io/api/v1/broker_init_script | sudo -E bash

These BROKER_ID and BROKER_ACCESS_KEY values are generated using Descope and identify the Broker with its tenant and connector instance.

Note: Each new broker is linked to a unique subnet and to a unique port.

DNS Server

services/go/dnsserver

The DNS Server operates as a sidecar container within the Trino pod, providing DNS functionality specifically tailored for Trino.

  • Database Synchronization: It synchronizes with the database to manage a list of hostnames for broker-related connector instances, mapping each hostname to its corresponding broker subnet. These hostnames are now designated as internal hostnames.

  • External Hostname Resolution: When Trino makes a query for an external hostname:

    • The DNS Server forwards the request to the cluster’s original DNS server.
    • The original DNS server responds with the external IP, enabling Trino to proceed with the query.

  • Internal Hostname Resolution: For internal hostnames:

    • The DNS Server delegates the resolution to the appropriate tun2proxy instance.
    • The resolved IP is returned to Trino for further processing.

tun2proxy

services/go/tun2proxy

tun2proxy is a tool designed to redirect all traffic from a TUN interface to a SOCKS or HTTP proxy.

Key Features

  • TUN Interface Creation:

    • The tun2proxy service creates a TUN interface for each Broker-specific subnet using information from the database.
    • Each Broker is assigned a unique subnet (e.g., 198.18.x.0/24), and this subnet is tied to its respective TUN interface.

  • IP Route Configuration:

    • tun2proxy sets up IP route rules to ensure:
      • Outgoing traffic destined for fake IPs in the 198.18.x.0/24 subnet is routed to the corresponding TUN interface.

  • DNS Handling:

    • When tun2proxy receives DNS queries from the DNS Server:
      • It returns a “fake” IP from the 198.18.x.0/24 subnet (e.g., 198.18.x.1, where x represents the Broker’s subnet).

  • Traffic Redirection:

    • Traffic to the fake IP is:
      • Routed through the TUN interface.
      • Encapsulated in SOCKS.
      • Forwarded to the appropriate FRP Client by the tun2proxy instance.

Special Notes

  • The 198.18.0.0/15 subnet is a special-purpose address block reserved for benchmarking and testing purposes, ensuring no IP collisions occur with external networks or internal Vega Kubernetes cluster service IPs.
  • tun2proxy uses subprocesses to configure all required networking.

Example Command

Below is an example of deploying the 7th broker of the “Nike” tenant using tun2proxy:

# Create a TUN interface called 'tun7'
ip tuntap add name tun7 mode tun
ip link set tun7 up

# Create a new route rule to route all traffic for the 198.18.7.0/24 subnet through 'tun7'
ip route add 198.18.7.0/24 dev tun7

# Start a tun2proxy instance:
# - Listen on the 'tun7' interface
# - Forward traffic to the FRP client for the "Nike" tenant
# - Act as a virtual DNS server that returns fake IPs from the 198.18.7.0/24 subnet
/usr/bin/tun2proxy --tun tun7 \
  --proxy "socks5://{user}:{pass}@FrpClient-Nike:{port}" \
  --dns virtual \
  --virtual-dns-pool 198.18.7.0/24

FRP Server (frps) and FRP Client (frpc)

frp is a fast reverse proxy designed to expose local services behind NAT or firewalls to the internet.

FRP Server (frps)

  • Deployed within the client’s app in our cluster (infra/helm/apps/client/{tenant_name}) to handle incoming traffic from FRP Clients.
  • Serves as a centralized gateway for all tunneled traffic originating from tenant FRP Clients.

FRP Client (frpc)

services/go/frpclient

  • Deployed within the client’s app in our cluster (infra/helm/apps/client/{tenant_name}):
    • Synchronizes with the Broker database.
    • Listens on relevant ports to receive traffic from tun2proxy instances.
    • Forwards traffic to the frps server for communication with the client’s Broker.

Note: The Broker container (installed in the client’s on-premises environment) includes an frpc (FRP Client) binary. This is the standard frp client binary, not our custom Go service which using it.

Traffic Flow

  1. The FRP Client in the cluster receives traffic from the tun2proxy instance.
  2. It forwards the traffic through the FRP Server to the tenant’s on-premises FRP Client.
  3. The tenant’s FRP Client resolves the target hostname and completes the request.

Broker Container

  • Files:
    • infra/docker/py/broker/Dockerfile
    • services/py/broker

The Broker container includes the FRP client and a Python script responsible for monitoring its health. It authenticates with the gateway using a provided access key, and upon successful authentication, the gateway returns the necessary FRP client configuration keys. The Broker then launches frpc and checks its health every 15 seconds. If the frpc health check fails, the supervisor automatically re-authenticates the Broker against the gateway.

Watchtower Integration

The Broker also supports automated updates via Watchtower, which handles Docker container base image updates. Watchtower is installed alongside the VegaBroker in the installation script and listens for new images published to ECR.

Below is an example installation script (located at https://vega.io/broker/init.sh):

#!/bin/bash

curl -fsSL https://vega.io/broker/get_docker.sh | sh

docker rm -f VegaBroker VegaWatchtower > /dev/null 2>&1
docker pull containrrr/watchtower
docker pull public.ecr.aws/b3d8e7u6/broker

docker run -d --name VegaBroker \
  -e BROKER_ID=$BROKER_ID \
  -e BROKER_ACCESS_KEY=$BROKER_ACCESS_KEY \
  -e HTTP_PROXY=$HTTP_PROXY \
  public.ecr.aws/b3d8e7u6/broker

docker run -d \
  --name VegaWatchtower \
  --restart unless-stopped \
  -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower VegaBroker --interval 3600

docker logs VegaBroker

Whenever a file in the Broker (such as the Dockerfile or the Python code) changes, the deploy.yaml pushes a new image to ECR with the test tag. After successful end-to-end tests — which include installing and querying over the Broker — this image is tagged as latest. Watchtower detects the new latest tag and automatically updates the VegaBroker container.

Currently, there is no mechanism to ensure that ongoing queries to the Broker are completed before the update, so connections may be terminated during the process.

Example Workflow

When Trino queries a Broker catalog (e.g., splunk.mysplunk.com):

  1. DNS Resolution:
  • Trino sends a DNS query to resolve splunk.mysplunk.com.
  • The DNS Server identifies the hostname as internal and queries the corresponding tun2proxy instance.
  • tun2proxy returns a fake IP (e.g., 198.18.1.1).
  • The DNS Server sends the fake IP back to Trino.
  1. Traffic Redirection:
  • Trino establishes a TCP connection to the fake IP.
  • Routing rules direct the traffic to the TUN interface managed by tun2proxy.
  • The TUN interface encapsulates the traffic in SOCKS and forwards it to the FRP Client.
  1. Final Resolution:
  • The FRP Client in the cluster forwards the traffic through the FRP Server to the tenant’s FRP Client.
  • The tenant’s FRP Client resolves the target hostname (splunk.mysplunk.com) and accesses the on-premises system.

Advantages

  • Transparent Integration: Requires no changes to Trino for enabling the feature.
  • Protocol Support: Supports all TCP protocols, including HTTP and others, without requiring changes to Trino connectors. Avoids the complexity of implementing SOCKS proxy support for non-HTTP protocols like MySQL or UDP.
  • Secure Communication: Traffic is encrypted and authenticated throughout the pipeline.
  • Future-Proof Design: The tun2proxy and DNS solution is reusable with new tools or systems, even if Trino is replaced in the future, ensuring no changes are required.

Future Improvements

  • Consolidate FRP Servers: Reduce the number of FRP Server instances to improve efficiency and manageability.
  • Direct Proxy Options: Introduce an option for direct proxying from Trino connectors to the FRP Client for specific use cases, eliminating the need for tun2proxy and the DNS server, thereby reducing network overhead and complexity.