thses tools will be used daily, use them wisely

tsh

Teleport cli

tsh is the command line tool for teleport. we use it to connect to the clusters and databases.

Temporary Admin Permissions

We have a Slack automation to get temporary admin permissions for databases and AWS for 12 hours.

How to Request Temporary Admin Access

  1. Run /temp_admin in any Slack chat
  2. Follow the prompts to request access

After Receiving Admin Permissions

After your request is approved, refresh your local Teleport session (required after roles change):

tsh logout
tsh login --proxy=vega.teleport.sh:443 --auth=okta-integration

Connecting to Database with Admin User

Once your session is refreshed, connect to the database with the admin user:

# Connect to production database as admin
tsh db connect prod-db --db-user=vega_admin --db-name=vega

# Connect to staging database as admin
tsh db connect staging-db --db-user=vega_admin --db-name=vega

# Connect to dev database as admin
tsh db connect dev-db --db-user=vega_admin --db-name=vega

To install tsh:

get the version from the webapi:

teleport_version=$(curl -s https://vega.teleport.sh/webapi/find | jq -r '.server_version')

macOS:

curl -O https://cdn.teleport.dev/tsh-$teleport_version.pkg
sudo installer -pkg tsh-$teleport_version.pkg -target /

Linux:

curl -O https://cdn.teleport.dev/teleport-v$teleport_version-linux-amd64-bin.tar.gz
tar -xzf teleport-v$teleport_version-linux-amd64-bin.tar.gz
cd teleport && sudo ./install   

usage:

tsh login --proxy=vega.teleport.sh:443 --auth=okta-integration  # login to teleport
tsh kube login dev                                              # login to dev cluster
tsh kube login staging                                          # login to staging cluster
tsh kube login prod                                             # login to prod cluster
tsh db connect dev-db --db-user=vega_admin --db-name=vega       # connect to dev-db database
tsh db connect staging-db --db-user=vega_admin --db-name=vega   # connect to staging-db database
tsh db connect prod-db --db-user=vega_admin --db-name=vega      # connect to prod-db database

Database Connection via Teleport

You can access our PostgreSQL databases via Teleport using either the CLI (PSQL) or GUI clients.

Available Database Users

User Description
vega_admin Full admin access
vega_ro Read-only access

Option 1: PSQL Access (CLI)

  1. Open Teleport (Teleport Connect app or via Okta)
  2. In the Resources tab, filter by type and choose Database
  3. Click Connect on the desired database
  4. Run the connection command:
tsh db connect prod-db --db-user=vega_admin --db-name=vega
tsh db connect staging-db --db-user=vega_admin --db-name=vega
tsh db connect dev-db --db-user=vega_admin --db-name=vega

Option 2: GUI Client Access (PgAdmin, DBeaver, etc.)

For GUI clients, you need to create a proxy connection. There are two methods:

The --tunnel flag creates a local tunnel without requiring SSL certificate files:

tsh proxy db prod-db --db-user vega_ro --db-name vega --tunnel

This will output connection details like:

Started authenticated tunnel for the Postgres database "prod-db" in cluster "vega.teleport.sh" on 127.0.0.1:12345.
Use the following credentials to connect to the database:
  Host: 127.0.0.1
  Port: 12345

Connect your GUI client using these connection details (no SSL configuration needed).

Method B: Using SSL certificates
tsh proxy db prod-db --db-user vega_admin --db-name vega

This command will:

  • Create a proxy connection on a random port
  • Create SSL artifacts locally

Then configure your GUI client with the SSL paths:

## kubectl `kubectl` is the command line tool for interacting with Kubernetes clusters. It allows you to deploy applications, inspect and manage cluster resources, and view logs.

To install kubectl:

kubectl Linux
kubectl macOS

kubectx

kubectx is a tool for switching between Kubernetes contexts. It allows you to quickly switch between different clusters.

To install kubectx:

brew install kubectx

usage:

kubectx                  # view all contexts
kubectx <context_name>   # switch to a context(cluster)

kubens

kubens is a tool for switching between Kubernetes namespaces. It allows you to quickly switch between different namespaces.

To install kubens:

brew install kubectx    # kubens is part of kubectx, so no need to re-install if u ran the above

usage:

kubens                  # view all namespaces
kubens <namespace_name> # switch to a namespace

prompt

in many cases we need to know on which cluster were on and which namespace. some commands cannot be reveresed and we don’t want to accidentally run them on the wrong cluster or namespace.

to solve this problem we use a prompt that shows the current cluster and namespace.

to config your prompt in zsh add this to your .zshrc:

PROMPT=$PROMPT'$(kube_ps1)'   # see https://github.com/jonmosco/kube-ps1 for install and details

if youre using p10k you need to remove the line:

  typeset -g POWERLEVEL9K_KUBECONTEXT_SHOW_ON_COMMAND='kubectl|helm|kubens|kubectx|oc|istioctl|kogito|k9s|helmfile'

K9s

K9s is a terminal based UI to interact with your Kubernetes clusters.

install

To install K9s:

 brew install derailed/k9s/k9s

AWS CLI

we use aws cli to interact with aws services. mainly connecting to our ECR (Elastic Container Registry) to pull our services docker images.

install

AWS SSO Login Script (rendered from scripts/aws-sso-login)
#!/bin/bash

function help_function() {
  echo "Usage: aws-sso-login [option...]" >&2
  echo
  echo "   -c, -config, --config                 config aws-sso-login for first time use"
  echo "   -h, -help, --help                     get help about the script"
  echo "   -p, -profiles, --profiles             list all available profiles"
  echo
  exit 1
}

function sso_init() {
  login_profile="
[profile dummy]
sso_start_url = https://d-9067f17bb4.awsapps.com/start
sso_region = us-east-1
sso_account_id = 000000000000
sso_role_name = dummy
region = us-east-2"
  filename=~/.aws/credentials.sso
  rm -f "$filename"
  echo "$login_profile" > "$filename"
  aws sso login --region us-east-1 --profile dummy
}

# Slugify the account name (lowercase, non-alnum → '-', squeeze and trim) and
# rewrite "vega" → "prod" so AWS_PROFILE=prod in tilt and CI keeps working.
function normalize_profile_name() {
  local name
  name=$(echo "$1" \
    | tr '[:upper:]' '[:lower:]' \
    | tr -c 'a-z0-9-' '-' \
    | sed -E 's/-+/-/g; s/^-//; s/-$//')
  if [ "$name" = "vega" ]; then
    name="prod"
  fi
  echo "$name"
}

# sso_config <profile_name> <account_id> <role_name>
function sso_config() {
  local profile_name="$1"
  local account_id="$2"
  local role_name="$3"

  if grep -q "^\[profile ${profile_name}\]\$" ~/.aws/credentials.sso; then
    return 1
  fi
  cat >> ~/.aws/credentials.sso <<EOF

[profile $profile_name]
sso_start_url = https://d-9067f17bb4.awsapps.com/start
sso_region = us-east-1
sso_account_id = $account_id
sso_role_name = $role_name
region = us-east-2
EOF
}

# Pre-flight: confirm the dummy profile exists in the file AWS CLI reads
# (~/.aws/config, which we symlink to credentials.sso). Doing this *before*
# calling `aws sso login` matters because that command streams a verification
# URL to the user — capturing its output to inspect for an error string would
# hide that URL until the command finishes.
function has_dummy_profile() {
  [ -r ~/.aws/config ] && grep -q '^\[profile dummy\]$' ~/.aws/config
}

function sso_login() {
  if ! has_dummy_profile; then
    echo "No SSO config found (missing dummy profile in ~/.aws/config)."
    if (: < /dev/tty) 2>/dev/null; then
      local yn=""
      read -r -p "Run aws-sso-login -c now? [Y/n]: " yn </dev/tty
      case "$yn" in
        ""|y|Y|yes|YES)
          config_script
          return $?
          ;;
      esac
      return 1
    fi
    echo "Run: aws-sso-login -c" >&2
    return 1
  fi
  aws sso login --region us-east-1 --profile dummy
}

function config_script() {
  mkdir -p ~/.aws
  if [ -L ~/.aws/credentials ]; then
    rm ~/.aws/credentials
  fi
  if [ -f ~/.aws/credentials ]; then
    mv ~/.aws/credentials ~/.aws/credentials.login
  fi
  ln -sf credentials.sso ~/.aws/config
  loggedInUser=$(stat -f %Su "/dev/console")
  shellcheck=$(finger "$loggedInUser" | grep 'Shell:*' | cut -f3 -d ":")
  if [[ $shellcheck = *"bash"* ]];
    then
      sed -i '' '/AWS_PROFILE/d' ~/.bash_profile
  elif [[ $shellcheck = *"zsh"* ]];
    then
      sed -i '' '/AWS_PROFILE/d' ~/.zshrc
  fi
  for file in ~/.aws/sso/cache/*; do
  if [ -f "$file" ]; then
    if [[ "$file" != *"botocore"* ]]; then
      rm "$file"
    fi
  fi
  done
  sso_init
  aws_access_token=$(grep -v "botocore" $(ls -1d ~/.aws/sso/cache/*) | grep -o '"accessToken": "[^"]*' | grep -o '[^"]*$')

  # Capture the accounts list up front so a CLI failure surfaces as an
  # explicit error rather than an empty loop body that prints "Setup complete".
  local accounts_output
  if ! accounts_output=$(aws sso list-accounts \
    --profile dummy \
    --access-token "$aws_access_token" \
    --region us-east-1 \
    --query 'accountList[].[accountName,accountId]' \
    --output text 2>&1); then
    echo "Failed to list SSO accounts: $accounts_output" >&2
    return 1
  fi

  while IFS=$'\t' read -r account_name account_id; do
    [ -z "$account_id" ] && continue

    local account_profile
    account_profile=$(normalize_profile_name "$account_name")

    # Disambiguate slug collisions ("Foo Bar" and "Foo-Bar" both → foo-bar).
    # Without this the second account is silently dropped by sso_config.
    if grep -q "^\[profile ${account_profile}\]\$" ~/.aws/credentials.sso; then
      local original_slug="$account_profile"
      account_profile="${account_profile}-${account_id: -4}"
      echo "  warning: profile '$original_slug' already taken, using '$account_profile'" >&2
    fi

    local roles_output
    if ! roles_output=$(aws sso list-account-roles \
      --profile dummy \
      --account-id "$account_id" \
      --access-token "$aws_access_token" \
      --region us-east-1 \
      --query 'roleList[].[roleName]' \
      --output text 2>&1); then
      echo "Failed to list roles for $account_name ($account_id): $roles_output" >&2
      continue
    fi

    # Sorted so the role-selection prompt below shows roles in a stable order.
    local -a roles=()
    while IFS= read -r role; do
      [ -n "$role" ] && roles+=("$role")
    done <<< "$(printf '%s\n' "$roles_output" | sort)"

    if [ ${#roles[@]} -eq 0 ]; then
      echo "No roles available for $account_name ($account_id), skipping"
      continue
    fi

    # Each account maps to exactly one profile. Multi-role accounts prompt
    # the developer to pick which role backs that profile.
    local primary_role="${roles[0]}"
    if [ ${#roles[@]} -gt 1 ]; then
      echo
      echo "Account $account_name ($account_id) has multiple roles."
      echo "Pick which role should back the \"$account_profile\" profile:"
      local i=1
      for role in "${roles[@]}"; do
        echo "  $i) $role"
        i=$((i+1))
      done
      # The outer loop's stdin is bound to a here-string, so prompt against
      # /dev/tty. If there's no controlling terminal (CI, piped run),
      # silently default to the first role.
      if (: < /dev/tty) 2>/dev/null; then
        local selection=""
        while true; do
          read -r -p "Selection [1-${#roles[@]}] (default 1): " selection </dev/tty
          selection="${selection:-1}"
          if [[ "$selection" =~ ^[0-9]+$ ]] && [ "$selection" -ge 1 ] && [ "$selection" -le ${#roles[@]} ]; then
            primary_role="${roles[$((selection-1))]}"
            break
          fi
          echo "  invalid selection: $selection"
        done
      else
        echo "(no tty — defaulting to first role: ${roles[0]})"
        primary_role="${roles[0]}"
      fi
    fi

    if sso_config "$account_profile" "$account_id" "$primary_role"; then
      echo "Profile $account_profile has been added (role: $primary_role)"
    else
      echo "  warning: profile $account_profile already configured, skipping" >&2
    fi
  done <<< "$accounts_output"

  echo
  echo "Setup complete. Configured profiles:"
  grep '^\[profile ' ~/.aws/credentials.sso \
    | sed -E 's/^\[profile (.*)\]$/  - \1/' \
    | grep -v '  - dummy$' \
    | sort
  echo
  echo "Use any of the above with: AWS_PROFILE=<name> aws ..."
}

function enable_sso() {
  if [ ! -f ~/.aws/credentials.login ]; then
    mv ~/.aws/credentials ~/.aws/credentials.login
  fi
  if [ -f ~/.aws/credentials ]; then
    rm ~/.aws/credentials
  fi
  if [ -f ~/.aws/config ]; then
    echo "Error - AWS SSO is already enabled"
    return 1
  fi
  ln -s credentials.sso ~/.aws/config
  echo "AWS SSO has been enabled"
}

function disable_sso() {
  if [ -L ~/.aws/credentials ]; then
    echo "Error - AWS SSO is already disabled"
    return 1
  fi
  if [ ! -f ~/.aws/credentials.login ]; then
    mv ~/.aws/credentials ~/.aws/credentials.login
  fi
  if [ -f ~/.aws/credentials ]; then
    rm ~/.aws/credentials
  fi
  rm ~/.aws/config
  ln -s credentials.login ~/.aws/credentials
  echo "AWS SSO has been disabled"
}

function list_profiles() {
  grep "sso_role_name" ~/.aws/config | grep -v "dummy" | cut -d= -f2
}

function old_profile()
{
  /usr/bin/sed -i .old "s/profile .*-/profile /g" ~/.aws/credentials.sso
}

if ! command -v aws &> /dev/null; then
    echo "aws-cli is not installed. Please install before continuing."
    exit 1
fi

if [ $# -eq 0 ] || [  "$1" == "" ]; then
    sso_login
    exit $?
fi

while [ "$1" != "" ]; do
  case $1 in
  -c | -config | --config)
    config_script
  ;;
  -d | -disable | --disable)
    disable_sso
  ;;
  -e | -enable | --enable)
    enable_sso
  ;;
  -h | -help | --help)
    help_function
  ;;
  -p | -profiles | --profiles)
    list_profiles
  ;;
  -o | -old_profile | --old_profile)
    old_profile
  ;;
  *)
  echo "Error: Unknown option: $1" >&2
    help_function
  exit 1
  ;;
  esac
  shift
done
export AWS_PROFILE=prod # set the profile to prod
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 123456789012.dkr.ecr.us-east-1.amazonaws.com # login to prod's ECR

# example command to list the s3 buckets in dev
export AWS_PROFILE=dev # set the profile to dev
s3 ls                          # list the s3 buckets

Mirrord

Mirrord is a tool that allows you to mirror your local traffic to your remote Kubernetes cluster, Providing a quick feedback loop. mirrord runs in two places - in the memory of your local process (mirrord-layer), and as a pod in your cloud environment (mirrord-agent).

install

you should install the plugin for your IDE as well as the CLI. We prepared the IDE configuration for VScode and Intellij in vega repository.

Look for the .mirrord , .vscode and .run folders for the default configuration.

stealing vs mirroring
  • mirroring: mirorr the incoming traffic to the local process
  • stealing: steal the incoming traffic and send it to the local process, and send the outgoing traffic to the remote cluster

In most cases we would like to use stealing, in order to debug the incoming and the outgoing traffic, that is the default mode.

running in the cluster with mirrord
  • you can open a shell in the cluster with mirrord
mirrord exec /bin/bash

ArgoCD

ArgoCD is a declarative, GitOps continuous delivery tool for Kubernetes.

we use ArgoCD to deploy our applications to the cluster and manage them with helm.

ArgoCD