Executive Summary

A self-serve data platform for data mesh succeeds when it behaves like an internal developer platform: it offers a small number of paved paths that domains can adopt quickly (templates + golden paths), while the platform team centrally manages cross-cutting concerns (identity, policy, observability, cost, metadata/lineage). [4]

Control Plane Capabilities

• A Data Product Registry API (resource-oriented) plus contract-first specs using OpenAPI for request/response APIs and AsyncAPI for event-driven interfaces. [5]
• A Data Contract standard (ODCS) stored alongside code and validated in CI/CD, enabling automated governance checks. [6]
• A Provisioning service that turns product manifests into infrastructure via IaC (Terraform/Pulumi) and GitOps workflows (e.g., Argo CD). [7]

Data Plane Capabilities

• Orchestration: Airflow and/or Dagster. Airflow for broad scheduling; Dagster for asset-centric, testable pipelines. [8]
• Streaming: Kafka or Pulsar. Pulsar has explicit multi-tenancy; Kafka relies on ACLs. [9]
• Lakehouse storage: Iceberg or Delta Lake. Both provide schema evolution and time travel/rollback primitives. [10]
• Compute: Spark (batch + micro-batch) and/or Flink (stateful streaming). [11]

Metadata, Lineage, & Observability

A metadata platform (DataHub or Amundsen) plus OpenLineage as the interoperability layer for run-level lineage events. [12]

Reference Architecture Overview

A data mesh-aligned platform should map naturally to the four widely cited principles: domain-oriented ownership, data as a product, self-serve platform, and federated computational governance. [14] The key move is to implement “self-serve” primarily as APIs + paved workflows, not a ticket queue.

This separation of planes is what makes self-service safe: the control plane exposes standardized interfaces, while the data plane scales independently with workload. [3]

Platform APIs for Data Products

This section focuses on how the platform exposes “data product as a first-class resource” using stable APIs, consistent auth, and machine-checkable contracts.

Recommended Patterns

• Contract-first specs: Use OpenAPI for synchronous APIs. Use AsyncAPI for event-driven interfaces. [18][19]
• Versioning: Adopt Semantic Versioning for data product interfaces. [20]
• AuthN/AuthZ: Use centralized identity via OpenID Connect (OIDC). Apply least privilege using RBAC at runtime layers. [2][22]

Example API Contracts

Data Product Registry (OpenAPI)

openapi: 3.1.0
info:
  title: Data Product Registry API
  version: 1.0.0
paths:
  /data-products:
    post:
      summary: Register a new data product
      security: [{ oidc: [ "dpr.write" ] }]
      requestBody:
        required: true
        content:
          application/json:
            schema: { $ref: "#/components/schemas/DataProductCreate" }
      responses:
        "201":
          description: Created
          headers:
            Location:
              schema: { type: string }
  /data-products/{productId}/versions:
    post:
      summary: Publish a new version of a data product
      security: [{ oidc: [ "dpr.write" ] }]
      parameters:
        - name: productId
          in: path
          required: true
          schema: { type: string }
        - name: Idempotency-Key
          in: header
          required: true
          schema: { type: string }
      responses:
        "202":
          description: Accepted for validation + provisioning
components:
  securitySchemes:
    oidc:
      type: openIdConnect
      openIdConnectUrl: https://idp.example.com/.well-known/openid-configuration
  schemas:
    DataProductCreate:
      type: object
      required: [name, domain, owner, description]
      properties:
        name: { type: string }
        domain: { type: string }
        owner: { type: string, description: "Group or team id" }
        description: { type: string }

Streaming Interface (AsyncAPI)

asyncapi: 3.0.0
info:
  title: Orders Data Product Events
  version: 1.2.0
servers:
  prod:
    host: broker.prod.example:9092
    protocol: kafka
channels:
  orders.created:
    address: orders.created
    messages:
      OrdersCreated:
        payload:
          type: object
          required: [order_id, created_at, customer_id]
          properties:
            order_id: { type: string }
            created_at: { type: string, format: date-time }
            customer_id: { type: string }
operations:
  publishOrdersCreated:
    action: send
    channel:
      $ref: "#/channels/orders.created"

Data Contracts as "Product Constitution" (ODCS)

apiVersion: v3.1.0
kind: DataContract
id: orders
name: Orders
version: 1.2.0
status: active
schema:
  - name: orders
    columns:
      - name: order_id
        type: string
        required: true
      - name: created_at
        type: timestamp
        required: true
quality:
  - type: freshness
    maxLagMinutes: 15
terms:
  - type: pii
    classification: restricted

Data Product Provisioning

To keep domains autonomous while preventing “snowflake infrastructure,” provisioning should be template-driven, policy-guarded, and declarative.

Example Backstage Scaffolder Template

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: data-product-iceberg-batch
spec:
  owner: platform-team
  type: data-product
  parameters:
    - title: Data Product Info
      required: [name, domain, owner]
      properties:
        name: { type: string }
        domain: { type: string }
        owner: { type: string }
  steps:
    - id: fetch-base
      action: fetch:template
      input:
        url: ./skeleton
        values:
          name: ${{ parameters.name }}
          domain: ${{ parameters.domain }}
          owner: ${{ parameters.owner }}
    - id: publish
      action: publish:github
      input:
        repoUrl: github.com?repo=${{ parameters.name }}&owner=${{ parameters.owner }}

Infrastructure-as-Code for Pipelines

Terraform Module Interface (Pseudo-code)

module "data_product_orders" {
  source = "git::ssh://git.example.com/platform/iac-modules.git//data-product"
  
  product_id   = "orders"
  domain       = "commerce"
  environment  = "prod"
  
  # Streaming interface
  kafka_topics = [
    { name = "orders.created.v1", partitions = 24, retention_hours = 168 }
  ]
  
  # Lakehouse interface
  lake_tables = [
    { name = "commerce.orders", format = "iceberg", partitioning = ["days(created_at)"] }
  ]
  
  # Access policy hooks
  owner_group = "grp-commerce-data"
  pii_level   = "restricted"
}

Pipeline Definitions

with DAG("orders_daily", schedule="@daily") as dag:
    extract = ExtractFromKafka(topic="orders.created.v1")
    transform = SparkSubmit(job="orders_transform.py")
    load = WriteIcebergTable(table="commerce.orders")
    
    extract >> transform >> load

@assetdef orders_raw(kafka_resource):
    return kafka_resource.read("orders.created.v1")

@asset(deps=[orders_raw])
def orders_iceberg(orders_raw, iceberg_io):
    iceberg_io.write("commerce.orders", orders_raw)

Internal Developer Platform Capabilities

A self-serve data platform becomes sustainable when it is delivered as an internal developer platform (IDP): portal UX + self-serve actions + guardrails + observability and cost controls.

• Developer portal: Backstage’s software templates scaffold components; catalog processes entities. [59]
• RBAC enforcement: Close to execution surfaces (Kubernetes RBAC, Kafka ACLs). [35][37]
• Observability: OpenTelemetry for traces, metrics, and logs. [60]
• Cost controls: Showback/chargeback via namespace/label attribution (e.g., Kubecost). [61]

Integration & Reference Stack Options