- 8822 Views
- 3 replies
- 2 kudos
Resolved! DLT pipeline
Hi all!I have a question about setting a target schema. How to set different targets for 2 different tables in the same delta live table pipeline. We have 2 target schemas in a database Bronze_chema and silver_schema. The pipeline has a streaming ra...
- 8822 Views
- 3 replies
- 2 kudos
- 20974 Views
- 9 replies
- 4 kudos
Kafka unable to read client.keystore.jks.
Below is the error we have received when trying to read the stream Caused by: kafkashaded.org.apache.kafka.common.KafkaException: Failed to load SSL keystore /dbfs/FileStore/Certs/client.keystore.jksCaused by: java.nio.file.NoSuchFileException: /dbfs...
- 20974 Views
- 9 replies
- 4 kudos
- 4 kudos
Ok, scrub that - the problem in my case was that I was using the 14.0 databricks runtime, which appears to have a bug relating to abfss paths here. Switching back to the 13.3 LTS release resolved it for me. So if you're in the same boat finding abfss...
- 4 kudos
- 11783 Views
- 2 replies
- 1 kudos
Resolved! Spark readStream kafka.ssl.keystore.location abfss path
Similar to https://community.databricks.com/t5/data-engineering/kafka-unable-to-read-client-keystore-jks/td-p/23301 - the documentation (https://learn.microsoft.com/en-gb/azure/databricks/structured-streaming/kafka#use-ssl-to-connect-azure-databricks...
- 11783 Views
- 2 replies
- 1 kudos
- 1 kudos
@Retired_mod- quick update - managed to find the cause. It's neither of the above, it's a bug in the DataBricks 14.0 runtime. I had switched back to the 13.3 LTS runtime, and that is what caused the error to disappear.As soon as I try to read directl...
- 1 kudos
- 2588 Views
- 0 replies
- 0 kudos
Spark streaming auto loader wildcard not working
Need som help with an issue loading a subdirectory from S3 bucket using auto-loader. For example:S3://path1/path2/databases*/paths/In databases there are various versions of databases. For examplepath1/path2/database_v1/sub_path/*.parquet path1/path...
- 2588 Views
- 0 replies
- 0 kudos
- 1833 Views
- 1 replies
- 0 kudos
Unable to start Cluster in Databricks because of `BOOTSTRAP_TIMEOUT`
Unable to start the Cluster in AWS-hosted Databricks because of the below reason{ "reason": { "code": "BOOTSTRAP_TIMEOUT", "parameters": { "databricks_error_message": "[id: InstanceId(i-0634ee9c2d420edc8), status: INSTANCE_INITIALIZIN...
- 1833 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Sahha, It may be a DNS issue if that wasn't rolled back, can you check the documentation on troubleshooting guide and see if these configurations were rolled back as well? https://docs.databricks.com/en/administration-guide/cloud-configurations/aw...
- 0 kudos
- 3882 Views
- 0 replies
- 0 kudos
A way to run OPTIMIZE, VACUUM, ANALYZE on all schemas and tables
Hello everyone,I am not sure if this was asked, but I am trying to find a way to create one python (or Scala) script that would basically take a list of all the schemas and then run optimize, vacuum, and then analyze tables on them.I see a lot of web...
- 3882 Views
- 0 replies
- 0 kudos
- 1867 Views
- 0 replies
- 1 kudos
Problems with extending DataFrame-object with custom method in DBR 14.0
I noticed that extending the DataFrame object with a custom method does not work in DBR 14.0 This works fine in when using DBR 13.3. The below code throws a "TypeError: 'Column' object is not callable". from pyspark.sql.functions import col,lit from...
- 1867 Views
- 0 replies
- 1 kudos
- 1331 Views
- 0 replies
- 0 kudos
Spark streaming is not able to assume role
Hello,I am trying to assume an IAM role in spark streaming with "s3-sqs" format. It is giving a 403 error. The code is provided below:spark.readStream .format("s3-sqs") .option("fileFormat", "json") .option("roleArn", roleArn) .option("compressi...
- 1331 Views
- 0 replies
- 0 kudos
- 5141 Views
- 1 replies
- 2 kudos
Databricks Job issue (run was cancelled bydatabricks and spark UI is not available after 10mins)
Hi!We had an issue on 09/19/2023 - we launched job, run was started, but after 10mins it was cancelled with no reasons. The spark ui is not available (which probably means that claster has not been started at all) and I don’t see any logs even.Could ...
- 5141 Views
- 1 replies
- 2 kudos
- 2 kudos
Was it a one time only error or a recurring one?For the former, I'd check if your vCPU quota was not exceeded, or perhaps there was a temporary issue with the cloud provider,... Could be a lot of things (lots of moving parts under the hood).For the ...
- 2 kudos
- 5536 Views
- 2 replies
- 2 kudos
Can't login anymore: Invalid email address or password
Since last Friday i cannot access databricks community any more, which is kinda annoying since my Bachelors dissertation is due in a couple of weeks. I always get the message: "Invalid email address or password Note: Emails/usernames are case-sensiti...
- 5536 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi Team,My community edition databricks cred are locked. I am doing very important project. Please help me resolve the issue Please try that it not gets locked in future as well.Email used for login @Retired_mod @Sujitha I have sent a email to commu...
- 2 kudos
- 3495 Views
- 2 replies
- 2 kudos
Unity Catalog - Writing to PNG Files to Cluster and then using dbutils.fs.cp to send to Azure ADLS2
Hi AllLooking to get some help. We are on Unity Catalog in Azure. We have a requirement to use Python to write out PNG files (several) via Matplotlib and then drop those into an ADLS2 Bucket. With Unity Catalog, we can easily use dbutils.fs.cp or fs....
- 3495 Views
- 2 replies
- 2 kudos
- 2 kudos
Hmm I read something different - someone else had this error because they used a shared cluster - apparently it does not happen on a single user cluster. All those settings are already done and I am a fully admin.
- 2 kudos
- 4432 Views
- 4 replies
- 0 kudos
Resolved! How do you enable verbose logging from with in Workspace Settings using Terraform?
I've searched in the databricks provider and online and couldn't find out if it is possible to set the `Verbose Audit Logs` to `enabled` using Terraform. Can anybody clarify if it is possible?
- 4432 Views
- 4 replies
- 0 kudos
- 0 kudos
The switch you're looking for is enableVerboseAuditLogs in databricks_workspace_confresource: { databricks_workspace_conf: { this: { custom_config: { enableIpAccessLists: true, enableVerboseAuditLogs: true, }, }, },
- 0 kudos
- 2735 Views
- 0 replies
- 0 kudos
Use Python code from a remote Git repository
I'm trying to create a task where the source is a Python script located in remote GitLab repo. I'm following the instructions HERE and this is how I have the task set up:However, no matter what path I specify all I get is the error below:Cannot read ...
- 2735 Views
- 0 replies
- 0 kudos
- 3019 Views
- 3 replies
- 0 kudos
Resolved! Issue with Logging Spark Events to LogAnalytics after Upgrading to Databricks 11.3 LTS
We have recently been in the process of upgrading our Databricks clusters to version 11.3 LTS. As part of this upgrade, we have been working on integrating the logging of Spark events to LogAnalytics using the repository available at https://github.c...
- 3019 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi Ravikumashi, Can you please raise a ticket with us so that we can look deeper in to the issue
- 0 kudos
- 3363 Views
- 0 replies
- 0 kudos
Scheduled job output export
Hi ,I have a Databricks job that results in a dashboard post run , I'm able to download the dashboard as HTML from the view job runs page , but I want to automate the process , so I tried using the Databricks API , but it says {"error_code":"INVALID_...
- 3363 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 482 | |
| 349 | |
| 287 |
