topic Scala Connectivity to Databricks Bronze Layer Raw Data from a Non-Databricks Spark environment in Data Engineering

Scala Connectivity to Databricks Bronze Layer Raw Data from a Non-Databricks Spark environment

BkP — Fri, 28 Oct 2022 07:55:01 GMT

Hi All,

We are developing a new Scala/Java program which needs to read & process the raw data stored in source ADLS (which is a Databricks Environment) in parallel as the volume of the source data is very high (in GBs & TBs). What kind of connection is required for reading high volume data in parallel manner in such cases ? JDBC don’t seem to be the right choice , as it cannot run multiple threads. Also, delta sharing has been tried but its not working. Can you please provide some pointer to some Scala/Java codebase, design and connectivity options for this requirement ?

Note that :- This is not an ETL process. After Data Read the raw data will be undergoing through curation and enrichment by the program and sent to downstream applications for consumption. We only have Gemfire on our spark cluster.

Any pointers would be a great help. Thanks in advance

Thank You,

Badal Panda

Re: Scala Connectivity to Databricks Bronze Layer Raw Data from a Non-Databricks Spark environment

BkP — Fri, 28 Oct 2022 08:06:25 GMT

More Info :

The source data in ADLS is from ERPs like SAP and JDE. The data format is parquet & both Full Load data and Delta Load data is available.

Re: Scala Connectivity to Databricks Bronze Layer Raw Data from a Non-Databricks Spark environment

BkP — Mon, 31 Oct 2022 19:31:28 GMT

hello experts. any advise on this question ?? tagging some folks from whom I have received answers before. Please help on this requirement or tag someone who can help on this

@Kaniz Fatma , @Vartika Nain , @Bilal Aslam