Databricks Community

Nagarathna · 05-14-2025

Hi team,I have a dataframe with 1269408570800 rows . I need to write this data to unity catalog table.How can I upload huge quantity of data ?I'm using databricks i runtime 15.4 LTS with 4 workers and each worker type is i3.4xlarge and driver of type...

Nagarathna · 04-15-2024

I am trying to reading json from aws s3 using with open in databricks notebook using shared cluster.Error message:No such file or directory:'/dbfs/mnt/datalake/input_json_schema.json'In single instance cluster the above error is not found.

Nagarathna · 05-14-2025

Thanks for your reply.Below is my answer to all the queries.1. What error is the job returning after the 2 hours execution?-> Below is the screenshot of error 2. What is the data source we are talking about? Are you trying to load data from a databas...

Nagarathna · 04-16-2024

Hey,Thanks for suggesting this approach.But I want to know why the json file cannot be read from AWS S3 bucket using "with open" in python with shared instance mode cluster. The code works perfectly fine if I'm using a single instance mode cluster.co...

Databricks Community

User Stats

User Activity

How to write trillions of rows to unity catalog table.

File not found error when trying to read json file from aws s3 using with open.

Re: How to write trillions of rows to unity catalog table.

Re: File not found error when trying to read json file from aws s3 using with open.