Schema Evolution & The Deprecation Protocol

Data changes are inevitable. DataSurface ensures that when schemas evolve, they do so safely. Backwards compatibility is verified through automated linting, and breaking changes are managed through a strict Deprecation Protocol that forces producer-consumer alignment before code is merged.

Automated Backward Compatibility Checks

Every time a developer opens a Pull Request (PR) to modify a `Datastore` schema, the DataSurface linter runs a compatibility check against all existing consumers.

Safe Changes (Allowed)

Adding a nullable column, relaxing a constraint.

Breaking Changes (Blocked)

Renaming a column used by a consumer, changing a type from String to Integer.

If a change is unsafe, the PR check fails immediately, preventing the breakage from ever reaching production.

The Deprecation Protocol

What if a Producer needs to make a breaking change or retire a dataset? They cannot simply break their consumers. DataSurface enforces a multi-step "handshake" protocol:

Step 1: Producer Attempts Deprecation (Blocked)

The Producer opens a PR marking a Datastore as deprecated=True. The linter scans the ecosystem and finds active consumers (Workspaces) that depend on this data.

Result: PR FAILED. The error message lists exactly which Consumer Workspaces have not yet agreed to handle deprecation.

Step 2: The Conversation

This failure forces a conversation. The Producer must contact the Consumer teams (listed in the PR failure) and negotiate a migration plan.

Step 3: Consumer Opt-In (PR Approval)

Each Consumer team must update their own `Workspace` definition to explicitly allow consumption of the deprecated dataset (e.g., deprecationsAllowed=DeprecationsAllowed.ALLOWED). They open PRs for their own workspaces.

Result: PR SUCCEEDED. Consumers signal they are ready/aware.

Step 4: Producer Retries (Success)

Once all consumers have merged their "allow" flags, the Producer re-runs their original PR.

Result: PR SUCCEEDED. The dataset is now formally deprecated, and the ecosystem is in a known, safe state.

Why This Matters

This protocol eliminates "surprise breakages." It turns an organizational coordination problem into a code-enforced gate. No one can break the build for others without explicit, recorded consent.