Databricks Community

aswinvishnu · 05-17-2025

Hi all,I have use case where I need to push the table data to GCS bucket,query = "${QUERY}" df = spark.sql(query) gcs_path = "${GCS_PATH}" df.write.option("maxRecordsPerFile", int("${MAX_RECORDS_PER_FILE}")).mode("${MODE}").json(gcs_path)This can ...

aswinvishnu · 04-29-2025

Hi all,Usecase: I want to send the result of a query to GCS bucket location in json format.Approach: From my java based application I create a job and that job will be running a notebook`. Notebook will have something like this```query = "SELECT * FR...

aswinvishnu · 05-20-2025

Thanks a lot @cgrant . This removed '_started_...' , '_committed_..', but still generated _SUCCESS file.spark.conf.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "false")this removed _SUCCESS files also.

aswinvishnu · 05-01-2025

Hi @lingareddy_Alva,Thanks for the reply. I tried the 'ACCESS_TOKEN' auth type too, but it didn't made any difference.

Databricks Community

User Stats

User Activity

Avoiding metadata information when sending data to GCS

Exporting table to GCS bucket using job

Re: Avoiding metadata information when sending data to GCS

Re: Exporting table to GCS bucket using job