Databricks Community is an open-source platform for data enthusiasts and professionals to discuss, share insights, and collaborate on everything related to Databricks. Members can ask questions, share knowledge, and support each other in an environment that ensures respectful interactions.
Hi there! I am calling the job list API (via the Python SDK): GET /api/2.2/jobs/list docs.databricks.com/api/workspace/jobs/list Does anyone know what ordering is applied / calculated for th...
...here is a header = true parameter, but I don't see a footer parameter in documentation. Has anyone found a good way to deal with the presence of a FOOTER in CSV files being detected by AutoLoader?
Hi, I want to read data from the Azure Event Hub using SQL. Can someone please give me an executable example where you can also use the connection string from the event hub using the SQL function s...
We have been using spark.read with inferSchema = True to validate AutoLoader schema inferencing. But I have a suspicion that they do these differently from each other and may not always yield the i...
I'm having difficulty adding a mask function to columns while creating streaming tables using the DLT Python method create_streaming_table() like this but it does not work, the streaming table is cre...
Hey. I am testing a continuous workflow job which executes the same notebook, so rather simple and it works well. It seems like it re-creates the job cluster for every iteration, instead of just re-...
...aximum size(in terms of MB or GB) per row databricks can accept or it can store. Saw some references in google and AI, they are saying upto 2.5GB. Is it true? If anyone knows the exact number, p...
Hi Community. Since I wasn't able to find a way to set a job tag dynamically at runtime based on a parameter that is passed to the job, I was wondering if it is possible or if there is an equivalent...
How can I use Terraform to assign an external location to multiple workspaces? When I create an external location with Terraform, I do not see any option to directly link workspaces. it also only li...