DLT Append Flow(Union) & Autoloader | Pass parameter in DLT pipeline |Generate tables dynamically

Published: None

Source: https://www.linkedin.com/pulse/dlt-append-flowunion-autoloader-pass-parameter-tables-mohapatra-qjkyc?trackingId=ux5WBiGsSi2CFJi4kUVf3A%3D%3D

---

DLT Append Flow(Union) & Autoloader | Pass parameter in DLT pipeline |Generate tables dynamically

Arabinda Mohapatra

Running Kafka streams after dark, diving into genetic code by daylight, and wrestling with Databricks and Tableflow in every spare moment—sleep is optional

January 7, 2025

• Create a folder and then create two workflows

DLT PIPELINE FOLDER

WORKFLOWS

Streaming Tables:

• orders_autoloaders_bronze ( 10K records)
• orders_bronze ( 7.5M records)
• customer_bronze (750K records)
• orders_bronze_union ( 7.5M records

Materialized Views:

• joined_vw ( 7.5M records)
• joined_silver (7.5M records)
• order_agg_gold (6 records)

Delta Workflows

1. 00__seup_for_introduction

Added some records to bronze.orders_raw

Volume landing & landing/files and landling/autoloader/schema directory created