Project Guide | Lakehouse Contract Lab

Live Demo

• Open the public GitHub Pages demo
• Scope: credential-free, synthetic-data demo for data-contract review and evaluators.

A runnable Spark + Delta Lake medallion pipeline demo that enforces data contracts at layer boundaries, applies declarative quality gates, and frames governed KPI export paths for Snowflake and Databricks Unity Catalog review. Built as a working reference implementation for contract-first data engineering with synthetic fixture data.

---

System Overview

A contract-first data lab that turns data quality from a slide into a repeatable pipeline and review artifact.

Evaluation Path

• Start here: Inspect the contract checks, quality reports, and medallion layer artifacts before deployment notes.
• Local smoke check: Run make smoke-no-build to exercise the API transiently on port 8097. For interactive docs, run make serve and open http://127.0.0.1:8096/docs.
• Checks: Run make verify; CI uses prebuilt artifact validation when a Spark/Java runtime is unavailable.

Architecture Notes

• Architecture guide summarizes the system scope, first files to inspect, runtime commands, and known boundaries.
• Quality notes lists the local checks, CI surface, and release expectations for this repository.
• Enterprise readiness notes outlines security, data, operations, integration, and handoff expectations.

Architecture

flowchart LR
    subgraph Ingestion
        CSV["source_orders.csv<br/><i>12 rows, 4 intentional violations</i>"]
    end

    subgraph Bronze["Bronze Layer"]
        B_TABLE[("bronze_orders<br/>Delta Table")]
        B_CONTRACT["Contract:<br/>- Raw envelope preserved<br/>- ingested_at attached<br/>- source_rank for dedup"]
    end

    subgraph Silver["Silver Layer"]
        GATES{"Quality Gates<br/>4 rules"}
        S_TABLE[("silver_orders<br/>Delta Table")]
        REJECTED[/"Rejected Rows<br/>Review Queue"/]
        S_CONTRACT["Contract:<br/>- customer_id NOT NULL<br/>- region NOT NULL<br/>- amount > 0<br/>- Dedup by order_id"]
    end

    subgraph Gold["Gold Layer"]
        G_TABLE[("gold_region_kpis<br/>Delta Table")]
        G_CONTRACT["Contract:<br/>- Region-level KPIs only<br/>- Only accepted rows<br/>- Fixed output schema"]
    end

    subgraph Export["Export Layer"]
        SF["Snowflake<br/>MERGE Upsert"]
        DB["Databricks<br/>Unity Catalog"]
        S3["AWS S3<br/>Artifacts"]
    end

    subgraph API["FastAPI Service"]
        HEALTH["/health"]
        QR["/quality-report"]
        PREVIEW["/table-preview"]
    end

    CSV --> B_TABLE
    B_TABLE --> GATES
    GATES -->|"Passed"| S_TABLE
    GATES -->|"Failed"| REJECTED
    S_TABLE --> G_TABLE
    G_TABLE --> SF
    G_TABLE --> DB
    G_TABLE --> S3
    G_TABLE --> API
    REJECTED --> QR

---

Technical Proof Boundary

---

Tech Stack

---

Local (no Docker)

## Clone and set up
git clone https://github.com/KIM3310/lakehouse-contract-lab.git
cd lakehouse-contract-lab
make install


## If your default python3 is older than 3.11:
make BOOTSTRAP_PYTHON=/path/to/python3.11 install


## Run the full pipeline: lint, test, build artifacts
make pipeline


## Start the API server
make serve
open http://127.0.0.1:8096/docs

Docker (recommended for full Spark + Delta rebuild)

cp .env.example .env
docker compose up --build

## API available at http://localhost:8096/docs

No Java Runtime?

If Java is not installed, the pipeline validates checked-in prebuilt artifacts instead of rebuilding Spark/Delta outputs. Tests and the API layer work without a JVM.

make test       # runs 81+ tests against prebuilt artifacts
make serve      # starts the API server

Makefile Reference

---

Quality Gates (Bronze to Silver)

Rules are defined declaratively in data/quality_rules.json and enforced as chained PySpark WHEN expressions. Failed rows land in a rejected DataFrame with a rejection_reason label, accessible at /api/runtime/quality-report. Rejected rows are never discarded -- they form a review queue for data engineers to investigate upstream quality issues.

Gold aggregates accepted silver rows by region into KPI columns: gross_revenue_usd, accepted_orders, completed_orders, pipeline_orders, distinct_customers.

---

Consolidated Operating Pattern

Data platform operating patterns folds demo-pack and rollout-playbook material into the canonical contract-first path used by this repository.

---

Core API

---

Deployment

All cloud integrations are env-var gated -- the project runs fully locally without any cloud credentials.

Snowflake -- set SNOWFLAKE_ACCOUNT, SNOWFLAKE_USER, SNOWFLAKE_PASSWORD. Gold KPIs are written via MERGE-based upserts to LAKEHOUSE_LAB.GOLD.REGION_KPIS.

Databricks Unity Catalog -- set DATABRICKS_HOST + auth (CLI profile, service-principal OAuth, or token). Gold KPIs land as Delta tables; catalog/schema auto-created.

AWS S3 -- set AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, S3_ARTIFACT_BUCKET to enable artifact upload.

GCP Cloud Run -- Terraform config in infra/terraform/.

---

Project Structure

lakehouse-contract-lab/
|-- app/
|   |-- main.py                  # FastAPI app serving pipeline artifacts
|   |-- snowflake_adapter.py     # Snowflake MERGE-based export adapter
|   |-- databricks_adapter.py    # Databricks Unity Catalog export adapter
|   |-- resource_pack.py         # Source data and config loaders
|-- scripts/
|   |-- build_lakehouse_artifacts.py  # Full medallion pipeline (Bronze/Silver/Gold)
|-- data/
|   |-- source_orders.csv        # 12-row synthetic dataset with quality violations
|   |-- quality_rules.json       # Declarative quality gate definitions
|   |-- export_targets.json      # Export target configurations
|   |-- validation_cases.json    # Test validation cases
|-- artifacts/                   # Generated pipeline outputs (JSON + Delta)
|-- tests/                       # 81+ pytest tests (adapters, API, pipeline, resource pack)
|-- docs/
|   |-- adr/                     # Architecture Decision Records
|   |-- data-platform-operating-patterns.md # Consolidated rollout and demo patterns
|   |-- data-contracts.md        # Contract-first approach documentation
|   |-- medallion-architecture.md # Layer-by-layer architecture guide
|-- infra/terraform/             # GCP Cloud Run deployment
|-- .github/workflows/ci.yml     # CI: lint, test, build, smoke, Docker

---

Operating Commands

• make verify runs the Python bootstrap, lint, pipeline tests, artifact checks, and smoke checks.
• python -m pytest tests -q exercises the contract compiler, export adapters, API surface, and resource pack.
• python scripts/build_prebuilt_artifacts.py refreshes the local JSON/Delta artifacts used by demos and API smoke checks.

---

Related Projects

---

Cloud + AI Architecture

• Cloud + AI architecture blueprint
• Machine-readable architecture manifest
• Validation command: python3 scripts/validate_architecture_blueprint.py

Enterprise Productization

• Product operating model defines the product scope, trust boundary, operating checks, and service path for this repository.

System Architecture

• System architecture maps the runtime boundary, data/control flow, cloud or local deployment surface, and operating assumptions for this repository.

Service Architecture

• Service architecture defines the cloud resources, account information, cost controls, and production guardrails needed to turn this repo into a scoped service without publishing public financial assumptions.

Search And Service Surface

• Public entry: free synthetic pipeline and contract examples
• Paid boundary: quoted private AI readiness sprint for lakehouse contract migration and quality reporting scope
• Canonical URL: https://lakehouse-contract-lab.pages.dev/
• Lead capture: https://kim3310-doeon-kim-portfolio.pages.dev/?offer=lakehouse-contract-lab&inquiry=private-ai-readiness-sprint#private-inquiry
• Resource route: https://kim3310-doeon-kim-portfolio.pages.dev/resources/lakehouse-contract-lab/
• Commercial route: https://kim3310-doeon-kim-portfolio.pages.dev/?offer=lakehouse-contract-lab#service-offers
• Machine-readable offer: docs/service-offer.json
• Search growth implementation: docs/search-growth-implementation.md
• Revenue architecture: docs/revenue-architecture.md