Databricks

venkat09 · 01-31-2023

* Reading Avro files from s3 and then writing to the delta table * Ingested sample data of 10 files, which contain four columns, and it infers the schema automatically as expected * Introducing a new file which contains a new column [foo] along wi...

venkat09 · 05-18-2022

Followed the documentation and facing issue while running dbx execute on all-purpose/interactive cluster, which is up and running already. Ran this command dbx execute --cluster-id=XXXXXX --job=dbx-demo-job --no-rebuild --debug. If anyone faced it ...

venkat09 · 01-31-2023

I am attaching the sample code notebook that helps to reproduce the issue.

venkat09 · 01-21-2023

Typo error in my second point of the previous post. Click the execution plan of your task[this is available under SQL/Dataframe tab in Spark UI]. It explains what operations run in the photon engine and what didn't execute by photon.

venkat09 · 01-19-2023

Tasks covered by yellow shade on DAG differentiate the operators are executed within photon engineAlso, if you clean the plan of your task, it explains well in detail what the operations run in the photon engine and what cannot be executed within it.

venkat09 · 05-18-2022

before running package it as wheel before running `dbx execute` fix the issue

Databricks

User Stats

User Activity

Schema Evolution - Auto Loader for Avro format is not working as expected

Running into an issue while setting up dbx locally?

Re: Schema Evolution - Auto Loader for Avro format is not working as expected

Re: How do I know how much of a query/job used Photon?

Re: How do I know how much of a query/job used Photon?

Re: Running into an issue while setting up dbx locally?