Databricks Community

Queries with streaming sources must be executed with writeStream.start();

When I try to perform some transformations on a streaming data , I get Queries with streaming sources must be executed with writeStream.start(); error My aim is to do a lookup for every column in each rows in the streaming data . steaming_table=spark...

Databricks with CloudWatch metrics without Instanceid dimension

I have jobs running on job clusters. And I want to send metrics to the CloudWatch. I set CW agent followed this guide.But issue is that I can't create useful metrics dashboard and alarms because I always have InstanceId dimension, and InstanceId is d...

Register for Databricks Office HoursSeptember 28: 11:00 AM - 12:00 PM PT | 6:00 - 7:00 PM GMT Databricks Office Hours connects you directly with exper...

Register for Databricks Office HoursSeptember 28: 11:00 AM - 12:00 PM PT | 6:00 - 7:00 PM GMTDatabricks Office Hours connects you directly with experts to answer your Databricks questions.Join us to:• Troubleshoot your technical questions• Learn the ...

Can we run multiple jobs in parallel on one cluster?

Structured streaming job sees throughput being capped after running normally for a few days

The job (written in PySpark) uses azure eventhub as source and use Databricks delta table as sink. The job is hosted in Azure Databricks.Transformation part is simple, the message body is converted from bytes to json string, the json string is then a...

How to resolve the error INVALID_PARAMETER_VALUE error in the Delta Live Table pipeline?

I am trying to execute a DLT pipeline, but I am getting an error which says - "INVALID_PARAMETER_VALUE: The field 'node_type_id' cannot be supplied when an instance pool ID is provided."I am using my company's Azure Databricks platform with premium b...

I have created a piece of python code, which lead to some python error.The job have failed with Internal Error, see below.The message after clicking o...

I have created a piece of python code, which lead to some python error.The job have failed with Internal Error, see below.The message after clicking on it states somewhat miseleading info:Meanwhile the real issue is fortunatelly described in Logs I d...

External table issue

Hello Team,I am using df.write command and the table is getting created. If you refer the below screenshot the table got created in Tables folder in dedicated sql pool. But i required in the External Tables folder. RegardsRK

What is the way to do DEEP CLONE and copy the checkpoints folder also?

When I use Deep Clone, I dont see the checkpoint folder to be copied?Is there a possibilities to copy the checkpoint folder also, as I have to resume the streaming job to updated location.

Delta table Concurrent Updates for Non-partitioned tables

When we implemented the concurrent updates on a table which do not have a partition column we ran into ConcurrentAppendException [ensured where the condition is different for each concurrent update statement]So do we need to go by partition approach ...

Run a simple spark-scala jar (hello-world) on existing running cluster

I have created a simple hello-world jar that I would like to run as a job. I also have an existing cluster. Now when I create a job to run on existing cluster, it fails for some unknown reason (don't see much in the errors), while if I run the same j...

How do I make a parameter in a Databricks SQL dashboard apply to multiple visuals?

I have created a few queries and visualizations in Databricks SQL which use parameters. Each query has the same parameter. But when I pin the visualizations to a dashboard, each of the visuals keeps it's own parameter drop down.I want to have one dro...