CQRS: Decoupling Ingestion from Consumption

Command Query Responsibility Segregation (CQRS) is a pattern that separates write operations (ingestion) from read operations (consumption). In DataSurface, this means we ingest data once and replicate it to multiple, optimized consumer destinations.

Why CQRS?

Traditional data platforms force everyone to query the same massive data lake or warehouse where data is ingested. This creates contention, locks you into a single vendor, and forces a "one size fits all" performance model.

With DataSurface CQRS:

  • Ingest Once: Expensive extraction from source systems happens only one time.
  • Consume Everywhere: Data is replicated to multiple Consumer Replica Groups (CRGs) tailored to specific needs.
  • Physical Isolation: A heavy analytical query on Snowflake won't slow down an operational dashboard on Postgres.

Hybrid Cloud Example

Consider a scenario where you have legacy on-premise data but want to leverage cloud analytics.

1. The Primary Ingestion (Write Side)

Location: On-Premise Data Center
Technology: Local Kubernetes & Postgres

DataSurface ingests raw data from legacy mainframes or databases into a local Primary Ingestion Platform. This keeps ingress costs zero and ensures low-latency capture.

CRG 1: Cloud Analytics Platform

Purpose: Enterprise Analytics & BI

Data is replicated to a cloud analytics platform. Marketing and Finance teams run massive queries here without affecting operational systems.

CRG 2: AWS Aurora

Purpose: High-Performance App Backend

A subset of the data is replicated to Aurora to power a customer-facing web application requiring low-latency lookups.

CRG 3: Onsite Postgres

Purpose: Dev/Test & Non-Prod

Developers work against a local Postgres replica. This avoids cloud egress costs and prevents accidental load on production cloud resources.

Flexible Workspace Placement & Cost Optimization

A Consumer Workspace is defined logically in the model. You can choose to place that workspace on any available CRG.

You might start a project on the Onsite Postgres CRG to keep costs low during development. When ready for production scale, you simply change the configuration to deploy that same Workspace to a Cloud Analytics CRG. The consumer code doesn't change—DataSurface handles the platform switch.

Conversely, if a project on an expensive cloud platform becomes too costly but doesn't require that specific scale, you can seamlessly move that Workspace to a lower-cost CRG when it makes commercial sense.