This page defines key terms and concepts used throughout the Vega platform to ensure consistent understanding across the development team.

Data Architecture & Multi-Tenancy

Tenant

A tenant represents a customer or organization that uses the system. Tenants allow for data isolation and customization.

Examples from the codebase:

  • VegaSecurity - Internal Vega Security tenant (used for tenant_name for data that is non-tenant specific)
  • customer1, customer2, customer3 - Example customer tenants

Connector

A connector is a type of data storage or SIEM system that can receive and store security logs. It’s the destination system where data sources are ingested.

Examples:

  • Splunk - Enterprise SIEM platform
  • Elasticsearch - Search and analytics engine
  • Google SecOps - Google’s security operations platform
  • Sentinel - Microsoft’s SIEM
  • Quickwit - Log analytics platform
  • PostgreSQL - Database system

Connector Instance

A connector instance is a configured instance of a connector that belongs to a specific tenant. It represents an actual connection to a customer’s Splunk, Elasticsearch, or other SIEM system.

Example:

  • Connector: Splunk (the type)
  • Connector Instance: “Customer ABC’s Splunk” (configured with their specific URL, credentials, etc.)

Data Source

A data source is a logical type of security data that represents a specific vendor’s product and log type. It’s defined by three components: vendor, product, and type.

Examples:

  • aws_cloudtrail_logs (Vendor: AWS, Product: CloudTrail, Type: logs)
  • microsoft_azure_activity_log (Vendor: Microsoft, Product: Azure, Type: Activity Log)
  • okta_system_log (Vendor: Okta, Product: System, Type: log)
  • crowdstrike_falcon_events (Vendor: CrowdStrike, Product: Falcon, Type: events)

DataSource Instance

A datasource instance is a specific instance of a data source that’s stored in a particular connector instance. It represents where and how a data source is actually stored (which table, schema, etc.).

Example:

  • Data Source: aws_cloudtrail_logs (the logical type)
  • DataSource Instance: AWS CloudTrail logs stored in Sentinel at "sentinel_catalog"."security"."aws_cloudtrail" table

Key properties:

  • scheme: The schema name (e.g., “security”)
  • table: The table name (e.g., “aws_cloudtrail”)
  • filter: Optional filter conditions
  • connectorInstance: Which connector instance stores this data

Real-world scenario:

  1. Data Source: aws_cloudtrail_logs (logical type)
  2. Connector: Sentinel (platform type)
  3. Connector Instance: “Acme Corp Sentinel” (Acme’s actual Sentinel instance)
  4. DataSource Instance: AWS CloudTrail logs stored in Acme’s Sentinel at table security.aws_cloudtrail
  5. Tenant: acme_corp (the customer)

This architecture allows multiple customers to use the same data source types and connector types, but with their own specific instances and configurations.

Data Source Discovery

Data source discovery is an automated process that scans a connector instance and automatically creates datasource instances based on what it finds.

How it works:

  1. Discovery Mappings: The system maintains mappings that connect identifiers (like Splunk sourcetypes or table names) to data sources

    • Example: "aws:cloudtrail" (identifier) → aws_cloudtrail_logs (data source)
    • Stored in database as DataSourceDiscoveryMapping

  2. Discovery Process: When triggered (manually via UI or automatically):

    • System queries the connector using connector-specific logic
    • Splunk: Queries for indexes and sourcetypes using tstats
    • Sentinel/Elasticsearch: Queries for available tables in schemas
    • Google SecOps: Queries for UDM search tables
    • Vega Storage: Discovers available indexes

  3. Matching: For each discovered identifier:

    • Looks up in discovery mappings to find matching data source
    • If match found → creates datasource instance with that data source
    • If no match → creates datasource instance with “unknown” data source

  4. Result: Datasource instances are automatically created and linked to the connector instance

Example (Splunk):

  • System finds: Index "security" with sourcetype "aws:cloudtrail"
  • Checks mappings: "aws:cloudtrail" → matches aws_cloudtrail_logs
  • Creates: DataSource Instance with table="security", filter="sourcetype='aws:cloudtrail'"

This automation eliminates manual configuration for known data sources while still capturing unknown data for later classification.

Data Processing & Normalization Terms

OCSF (Open Cybersecurity Schema Framework)

  • Definition: A standardized schema framework for cybersecurity data that provides a common language for security events
  • Usage: Vega uses a format built based on OCSF principles to normalize security logs into a consistent structure
  • Examples: OCSF classes like authentication, file_activity, process_activity, account_change

Views (Trino Views)

  • Definition: SQL views that represent different stages of data processing in Trino
  • Types:
    • Base Views: datasource.sql - basic data source representation
    • Parser Views: Connector-specific data transformation
    • Normalization Views: OCSF-compliant structured data
    • Vega Metadata Views: Platform-specific metadata

Parser

  • Definition: SQL transformations that convert raw log data from specific connectors into a standardized format
  • Types:
    • Connector-specific parsers: aws_cloudtrail_logs_splunk_parser.sql
  • Purpose: Extract and structure fields from raw logs before normalization

Normalization

  • Definition: The process of converting parsed data into Vega’s OCSF-inspired JSON structures
  • Files: aws_cloudtrail_logs.sql (creates standardized JSON with OCSF-based fields)
  • Output: Structured JSON with fields like actor.user.name, src_endpoint.ip, etc.

Understanding these terms helps you navigate the Vega platform architecture:

Data Flow: Raw Logs → Parser → Normalization → OCSF Structure → Detection Engine

This terminology page should be updated as new concepts are introduced to the platform or existing terms evolve.§