topic Re: How to design Airship Integration with Azure Databricks in Data Engineering

How to design Airship Integration with Azure Databricks

Datalight — Tue, 09 Sep 2025 15:04:44 GMT

Hello,

I have to push data from Airship and persists it to Delta tables. I think We can used SFTP , May someone please help me how to design the inbound part , it using SFTP on Airship end to push file on ADLS Gen2.

networking and security considerations of how this could work.

Re: How to design Airship Integration with Azure Databricks

ManojkMohan — Tue, 09 Sep 2025 16:07:09 GMT

Inbound Flow Design
Enable SFTP on the ADLS Gen2 (or Azure Blob Storage) account;

Generate and register an SSH public/private key pair with Airship, enter your SFTP endpoint credentials (username, host, port, key) in Airship’s settings to authenticate uploads.

Configure Airship to push files (CSV or other supported formats) to the specific SFTP directory in your ADLS Gen2 account, designated for inbound data.

Trigger Azure Data Factory (ADF) or Databricks jobs using storage events (e.g., SFTP commit events, which ensure files are fully uploaded before processing) to ingest these files, transforming them into Delta tables

Re: How to design Airship Integration with Azure Databricks

Datalight — Wed, 10 Sep 2025 12:06:52 GMT

@ManojkMohan : what will be the better approach. to have seperate Landing zone and than Bronze layer or only 1 Bronze layer can be treated as landing zone too. ?

Re: How to design Airship Integration with Azure Databricks

szymon_dybczak — Wed, 10 Sep 2025 12:38:39 GMT

Hi @Datalight ,

If I were you I would add separate landing zone. In our poject landing zone has been extremely valuable. Among other things it lets you separate concerns in terms of extracting data vs loading/processing it. It also allows you to easily reprocess, for all the reasons that can be needed.

Re: How to design Airship Integration with Azure Databricks

ManojkMohan — Wed, 10 Sep 2025 15:03:47 GMT

Suggested Design

SFTP uploads from Airship go to a Landing Zone folder in ADLS Gen2.
Storage event triggers an orchestration pipeline (ADF/Databricks) to read and process raw files.
Processed data lands in the Bronze Layer Delta tables with schema enforcement.
Silver and Gold layers follow based on business requirements.

Re: How to design Airship Integration with Azure Databricks

Datalight — Wed, 10 Sep 2025 15:20:30 GMT

@szymon_dybczak : Thanks a lot.

What do you think, how much effort would be in devops side from bronze to landing first than bronze, if we already deployed the 4 data pipeline use cases.