The Upstream is sending 2 files of different schema. The Storage Account has Private Endpoints. there is no public access.no public IP (NPIP) = yes.How to design using only Databricks :-1. Databricks API to read data file from Adobe and Push it to AD...
I have 2 Databricks Instance. Databricks A and Databricks B.The Application Hightouch is consuming Data from Databricks B through Unity Catalog.I have to create a Data pipeline to push data from Databricks A to B Without using Delta sharing.Diagram: ...
Hello,I am not allow to use Delta share.What could be best approach to send Data from Databricks A to Databricks B. AS Shown in diagram.what mechanism we can used to transfer the data. for example:- Do I need to open port or any other mechanism like ...
I am creating a Data Pipeline as shown below.1. Files from multiple input source is coming to respective folder in bronze layer.2. Using Databricks to perform Transformation and load transformed data to Azure SQL. also to ADLS Gen2 Silver (not shown ...
Hello,I am using Databricks first time. May someone please help me how to do orchestration for the pipeline shown below.Kindly share the steps how to implement Orchestration , what all steps we have to consider.Thanks a lot
Actually . have to design logging in Databricks CI Satellite EDLAP.Can I do it in ADLS Gen2 Silver layer or do I need to have any other component.Can Someone please help me how we can have folder structure and can we have delta table for logging. wh...
@NandiniN : Thanks a lot. I am pretti new to data world. Can you just provide what additional benefit Databricks can provide, and how realtime streaming support by Databricks. That would be very helpful.
