Consumer Aligned Data Flows

In DataSurface, data movement is driven entirely by consumer demand. Producers merely signal availability; consumers define the terms of delivery. This inversion of control decouples teams and future-proofs your architecture.

1. Producer Awareness vs. Consumer Action

A data producer's role is simple: indicate in the model that data is available. They define the schema and the source location. Crucially, nothing happens just because a producer defines a dataset.

No pipelines are built, no storage is provisioned, and no compute is consumed until a consumer explicitly declares a need for that data in their Workspace. This "pull-based" model prevents data swamps and wasted resources.

2. Consumer Defined Requirements

The consumer defines the "what" and the "how" of their data needs. They specify:

  • Delivery Mode: Do they need full history (SCD2) for analysis, or just the live current state (SCD1) for operational dashboards?
  • Technology: What type of database do they want to query? A consumer can request the data in any supported technology, regardless of where the producer stored it.
  • Latency: Can they tolerate high latency for batch reporting, or do they need low-latency updates?
  • Retention: How long should the records be kept? The consumer sets the retention policy for their workspace.

DataSurface handles the complexity of meeting these diverse requirements from a single source.

3. The "Zero-Build" Promise: Data Logistics

DataSurface's ultimate goal is to solve data logistics. We believe that:

  • Producers should focus solely on maintaining the quality and availability of their data.
  • Consumers should focus entirely on leveraging value from the data they consume.

Nobody should be building data pipelines. The complex logistics of moving, transforming, and securing data is handled entirely by DataSurface.

Just as Amazon frees buyers and sellers from worrying about shipping logistics, DataSurface frees data teams from pipeline engineering.

Just as internet users and website owners don't worry about how packets move across the network, DataSurface users don't worry about the underlying transport. It just happens.

DataSurface strives to make data collaboration as seamless as Amazon makes commerce or the internet makes connectivity.

4. Long-Term Engineering Freedom

If consumers or producers were responsible for building these pipelines manually, every new requirement would be a customized engineering project. This leads to "pipeline debt" that is impossible to migrate.

With DataSurface, moving from technology A to technology B (e.g., migrating from on-prem Hadoop to cloud Snowflake, or from Redshift to Databricks) is a configuration change, not a rewrite. As new consumers emerge with different needs, or as the firm's technology strategy evolves, the model adapts without requiring a complete re-engineering of the data estate.