topic Read and saving Blob data from oracle to databricks S3 is slow in Data Engineering

Read and saving Blob data from oracle to databricks S3 is slow

RKNutalapati — Sat, 16 Oct 2021 09:16:34 GMT

I am trying to import a table from oracle which has around 1.3 mill rows and one of the column is a Blob, the total size of data on oracle is around 250+ GB. read and save to S3 as delta table is taking around 60 min. I tried with parallel(200 threads) read using JDBC. Still its taking more time.

Appreciate your valuable suggestions to speed up the process

Re: Read and saving Blob data from oracle to databricks S3 is slow

Anonymous — Sun, 17 Oct 2021 18:56:00 GMT

Hello, @Rama Krishna N - My name is Piper and I'm one of the community moderators. Thanks for your question. Let's give it a bit longer to see what the community says. Thank you for your patience.

Re: Read and saving Blob data from oracle to databricks S3 is slow

User16829050420 — Wed, 20 Oct 2021 08:04:26 GMT

Can you check the parallel threads and confirm if the read or write operation is slower? Read operation slowness can be caused because of network issues or concurrency issues on the database.

Re: Read and saving Blob data from oracle to databricks S3 is slow

RKNutalapati — Wed, 20 Oct 2021 08:28:22 GMT

Thanks @Ashwinkumar Jayakumar for reply. I had tried with dataFrame.count, it didn't take much time. Please suggest if there is any other best approach to check read operation slowness.

Re: Read and saving Blob data from oracle to databricks S3 is slow

User16829050420 — Wed, 20 Oct 2021 08:36:09 GMT

Hello @Rama Krishna N - We will need to check the task on the Spark UI to validate if the operation is a read from oracle database or write into S3.

The task should show the specific operation on the UI.

Also, the active threads on the Spark UI will show if the specific operation is a database operation.

Re: Read and saving Blob data from oracle to databricks S3 is slow

vinita_mehta — Wed, 16 Oct 2024 14:21:05 GMT

Any update on this topic what should be the best option to read from oracle and write in ADLS.