Salesforce Commerce
Cloud Data Pipeline
Production-grade SFCC data pipeline built using Azure Databricks, ADLS Gen2, and a Bronze–Silver–Gold Lakehouse architecture for analytics-ready datasets at scale.
3 API Domains
3 Lakehouse Tiers
∞ Incremental Loads
Azure Cloud Native
01 — Objective
Design and implement a scalable, production-ready data engineering
pipeline that ingests Salesforce Commerce Cloud REST API data,
processes nested JSON structures, and delivers analytics-ready
datasets to a data warehouse using Azure Databricks and Lakehouse
architecture.
02 — Architecture
System Overview
03 — Lakehouse Architecture
Bronze · Silver · Gold
Bronze
Raw JSON ingestion from SFCC REST APIs, stored as-is in ADLS Gen2
Archive
Silver
Nested JSON flattening, schema normalization, incremental
watermark loads
Gold
Dimensional models (fact & dim tables) optimized for BI &
reporting
04 — Pipeline Highlights
What's Under the Hood
Incremental data extraction from SFCC REST APIs — Customers,
Orders, Products
Raw JSON ingestion (Bronze) with durable storage in ADLS Gen2
Archive
Nested JSON flattening and schema normalization using Databricks
notebooks
Incremental loads using watermark strategy to ensure no duplicate
processing
Dimensional modeling — fact & dimension tables for analytics
consumption
Gold-layer analytics datasets optimized for BI tools & executive
reporting
05 — Software Toolkits
Built With
