How DataSurface Works

1. Control Your Domain

Each GovernanceZone is managed by a specific Git repository. Zone owners control their teams, datastores, workspaces—independent from other zones. Central team declares your zone, then you manage everything within it.

# Central team creates declaration (in ecosystem repo)
ecosystem.add(
    GovernanceZoneDeclaration("Finance",
        GitHubRepository("company/finance-data", "main"))
)

# Finance team defines their zone (in their own repo)
zone = ecosystem.getZoneOrThrow("Finance")
zone.add(
    TeamDeclaration("Treasury",
        GitHubRepository("company/treasury-data", "main"))
)
team = zone.getTeamOrThrow("Treasury")

2. Set Policies for Your Data

Define who can access your data and for what purpose. Policies are enforced automatically during PR validation and at runtime.

zone.add(
    InfraStructureLocationPolicy(
        "Finance Data Residency",
        PlainTextDocumentation("Finance data must stay in US"),
        {LocationKey("AWS:USA/us-east-1"),
         LocationKey("AWS:USA/us-west-2")}
    )
)

3. Submit Pull Requests to Central

Your changes are submitted as PRs to the central ecosystem repository. Automated validation ensures you only modified your authorized zone. After merge, your data and policies are live across all environments.

Federated Governance at Scale

Each zone operates independently with its own repo and team. M&A integration? Just add a new GovernanceZone. Acquired company retains control of their data while sharing according to your policies.

1. Declare Your Data

Define your database schema, specify how to connect (credentials, hostname), and set ingestion frequency. Commit to your team's Git repository.

team.add(
    Datastore("SalesDB",
        PlainTextDocumentation("Sales database"),
        datasets=[customers, orders, products]
    )
)

2. Submit Pull Request

Your changes are validated: schemas checked, authorization verified, policies enforced. Central team reviews and merges.

3. Done

DataSurface starts ingesting your data automatically. You control who can access it through policies. No pipeline code to write or maintain.

1. Request the Data You Need

Create a Workspace, specify which datasets you need, and define your requirements (live-only vs. full history, latency, retention period).

team.add(
    Workspace("CustomerAnalytics",
        DataPlatformManagedDataContainer("analytics"),
        DatasetGroup("LiveData",
            sinks=[DatasetSink("SalesDB", "customers")],
            platform_chooser=WorkspacePlatformConfig(
                hist=ConsumerRetentionRequirements(
                    r=DataMilestoningStrategy.LIVE_ONLY
                )
            )
        ),
        DatasetGroup("HistoricalOrders",
            sinks=[DatasetSink("SalesDB", "orders")],
            platform_chooser=WorkspacePlatformConfig(
                hist=ConsumerRetentionRequirements(
                    r=DataMilestoningStrategy.FORENSIC
                )
            )
        )
    )
)

2. Get Infrastructure Assignment

Central ops reviews and assigns your Workspace to a DataPlatform (production, dev, or specific consumer database for high-volume use).

3. Query Your Views

DataSurface creates workspace-specific SQL views. Connect and query. Views enforce permissions—you see only your authorized data.

-- Connection provided by ops team
SELECT * FROM customer_analytics.customers_v_l
WHERE signup_date > '2025-01-01';

1. Define Runtime Environments

Configure DataPlatforms (dev/qa/prod), specify infrastructure (Kubernetes namespace, databases, Airflow), set version selectors (which git tags to deploy).

2. Map Workspaces to Platforms

Assign consumer Workspaces to appropriate DataPlatforms. High-priority or high-volume consumers can get dedicated Consumer Replica Groups (CQRS).

3. Monitor & Scale

Watch batch metrics, latency, and load. Add more consumer databases as needed. DataSurface handles replication automatically.