I want to get job launched status in notebook if it is launched by scheduler or manuallyI tried using JobTriggerType property of notebook context but it gives only manual and repair but not by scheduleddbutils.notebook.entry_point.getDbutils().notebo...
I am working as data engineer I was about checking the subscription plan. I would like to know how I can check the subscription plan. I am "Admin" but I cannot "Manage accounts" on Databricks workspace portal.This subscription information is pretty i...
Hey, not really. Update you: if you try to add ACL to a secret scope then you will be sure that your subscription is Enterprise or Standard, because either you succeed then you are working with Enterprise or the API respond with the Standard subscrip...
and it seems that older data (From the "replaced" table) isn't being removed, long after the retention period I'd be glad for clues on how to handle this
Thanks, I know that, but the table history shows 30 days, but the actual data size and number of files and all other indicators , correlate to 170 days.
Like title, i have a delta table with data of 22/9 and today i wanna remove old data and add new data of 23/9, i used 'truncate' and 'copy into' query but after 'truncate', nothing is added to table, what's happened with my table, file of old data st...
Hi all!I have a question about setting a target schema. How to set different targets for 2 different tables in the same delta live table pipeline. We have 2 target schemas in a database Bronze_chema and silver_schema. The pipeline has a streaming ra...
Below is the error we have received when trying to read the stream Caused by: kafkashaded.org.apache.kafka.common.KafkaException: Failed to load SSL keystore /dbfs/FileStore/Certs/client.keystore.jksCaused by: java.nio.file.NoSuchFileException: /dbfs...
Ok, scrub that - the problem in my case was that I was using the 14.0 databricks runtime, which appears to have a bug relating to abfss paths here. Switching back to the 13.3 LTS release resolved it for me. So if you're in the same boat finding abfss...
Similar to https://community.databricks.com/t5/data-engineering/kafka-unable-to-read-client-keystore-jks/td-p/23301 - the documentation (https://learn.microsoft.com/en-gb/azure/databricks/structured-streaming/kafka#use-ssl-to-connect-azure-databricks...
@Retired_mod- quick update - managed to find the cause. It's neither of the above, it's a bug in the DataBricks 14.0 runtime. I had switched back to the 13.3 LTS runtime, and that is what caused the error to disappear.As soon as I try to read directl...
Need som help with an issue loading a subdirectory from S3 bucket using auto-loader. For example:S3://path1/path2/databases*/paths/In databases there are various versions of databases. For examplepath1/path2/database_v1/sub_path/*.parquet path1/path...
Unable to start the Cluster in AWS-hosted Databricks because of the below reason{ "reason": { "code": "BOOTSTRAP_TIMEOUT", "parameters": { "databricks_error_message": "[id: InstanceId(i-0634ee9c2d420edc8), status: INSTANCE_INITIALIZIN...
Hi Sahha, It may be a DNS issue if that wasn't rolled back, can you check the documentation on troubleshooting guide and see if these configurations were rolled back as well? https://docs.databricks.com/en/administration-guide/cloud-configurations/aw...
Hello everyone,I am not sure if this was asked, but I am trying to find a way to create one python (or Scala) script that would basically take a list of all the schemas and then run optimize, vacuum, and then analyze tables on them.I see a lot of web...
I noticed that extending the DataFrame object with a custom method does not work in DBR 14.0 This works fine in when using DBR 13.3. The below code throws a "TypeError: 'Column' object is not callable". from pyspark.sql.functions import col,lit from...
Hello,I am trying to assume an IAM role in spark streaming with "s3-sqs" format. It is giving a 403 error. The code is provided below:spark.readStream .format("s3-sqs") .option("fileFormat", "json") .option("roleArn", roleArn) .option("compressi...
Hi!We had an issue on 09/19/2023 - we launched job, run was started, but after 10mins it was cancelled with no reasons. The spark ui is not available (which probably means that claster has not been started at all) and I don’t see any logs even.Could ...
Was it a one time only error or a recurring one?For the former, I'd check if your vCPU quota was not exceeded, or perhaps there was a temporary issue with the cloud provider,... Could be a lot of things (lots of moving parts under the hood).For the ...
Since last Friday i cannot access databricks community any more, which is kinda annoying since my Bachelors dissertation is due in a couple of weeks. I always get the message: "Invalid email address or password Note: Emails/usernames are case-sensiti...
Hi Team,My community edition databricks cred are locked. I am doing very important project. Please help me resolve the issue Please try that it not gets locked in future as well.Email used for login @Retired_mod @Sujitha I have sent a email to commu...
Hi AllLooking to get some help. We are on Unity Catalog in Azure. We have a requirement to use Python to write out PNG files (several) via Matplotlib and then drop those into an ADLS2 Bucket. With Unity Catalog, we can easily use dbutils.fs.cp or fs....
Hmm I read something different - someone else had this error because they used a shared cluster - apparently it does not happen on a single user cluster. All those settings are already done and I am a fully admin.
| User | Count |
|---|---|
| 1637 | |
| 791 | |
| 517 | |
| 349 | |
| 287 |
