Scala Connectivity to Databricks Bronze Layer Raw Data from a Non-Databricks Spark environment

[BkP]

Hi All,

 

We are developing a new Scala/Java program which needs to read & process the raw data stored in source ADLS (which is a Databricks Environment) in parallel as the volume of the source data is very high (in GBs & TBs). What kind of connection is required for reading high volume data in parallel manner in such cases ? JDBC don’t seem to be the right choice , as it cannot run multiple threads. Also, delta sharing has been tried but its not working. Can you please provide some pointer to some Scala/Java codebase, design and connectivity options for this requirement ? 

Note that :- This is not an ETL process. After Data Read the raw data will be undergoing through curation and enrichment by the program and sent to downstream applications for consumption. We only have Gemfire on our spark cluster.

Any pointers would be a great help. Thanks in advance

Thank You,

Badal Panda

[BkP]

More Info :

The source data in ADLS is from ERPs like SAP and JDE. The data format is parquet & both Full Load data and Delta Load data is available.

[Kaniz]

Hi @Badal Panda​, We can read our file via the Spark DataFrameReader API and write the result to Delta Lake.

Notice how we use the %scala language magic command, which lets us run the cell using Scala.  

%scala
 
#define paths
val rawpath = "path"
val bronzepath = "path"

#read the file
val file = spark.read.parquet(rawpath)

[BkP]

hello experts. any advise on this question ?? tagging some folks from whom I have received answers before. Please help on this requirement or tag someone who can help on this

@Kaniz Fatma​ , @Vartika Nain​ , @Bilal Aslam​