- 9233 Views
- 4 replies
- 0 kudos
Intermittent error databricks job kept running
Hi i have the following error, but the job kept running, is that normal?{ "message": "The service at /api/2.0/jobs/runs/get?run_id=899157004942769 is temporarily unavailable. Please try again later. [TraceId: -]", "error_code": "TEMPORARILY_U...
- 9233 Views
- 4 replies
- 0 kudos
- 0 kudos
@Ayushi_Suthar also when ever it happens the job status does not change to "failed". But it keeps running. Is that normal?
- 0 kudos
- 11165 Views
- 1 replies
- 2 kudos
Change Data Capture with Databricks
How to leverage Change Data Capture (CDC) from your databases to DatabricksChange Data Capture allows you to ingest and process only changed records from database systems to dramatically reduce data processing costs and enable real-time use cases suc...
- 11165 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi, @isaac_gritz can you provide any reference resource to achieve the AWS DynamoDB CDC to Delta Tables.Thank You,
- 2 kudos
- 12259 Views
- 2 replies
- 3 kudos
Resolved! What happens to table created with CTAS statement when data in source table has changed
Hey all - I am sure this has been documented / answered before but what happens to a table created with a CTAS statement when data in the source table has changed? Does the sink table reflect the changes? Or is the data stored when the table is defin...
- 12259 Views
- 2 replies
- 3 kudos
- 3 kudos
CREATE TABLE AS (CTAS) is a "one and done" kind of statement.The new table retains no memory on how it came to be.Therefore it will be oblivious to changes in the source.Views, as you say, stored queries, no data is persisted. And therefore the query...
- 3 kudos
- 13757 Views
- 3 replies
- 3 kudos
List of Nodes Types
Hi Guys,Where can I find the list f nodes types on the Databricks website ? (standard_ds3_v2, etc. etc. etc)Thank you
- 13757 Views
- 3 replies
- 3 kudos
- 15244 Views
- 4 replies
- 1 kudos
Resolved! Managed table overwrites existing location for delta but not for oth
I am working on Azure Databricks, with Databricks Runtime version being - 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12). I am facing the following issue.Suppose I have a view named v1 and a database f1_processed created from the following comman...
- 15244 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi,this is how the delta format work. With overwrite you are not deleting the files in the folder or replacing them. Delta is creating a new file with the overwritten schema and data. This way you are also able to return to former versions of the del...
- 1 kudos
- 2220 Views
- 1 replies
- 0 kudos
Dashboard from Notebook - How to schedule
notebook is created with insight and have created dashboard (Not a SQL) from it.Need to schedule this. I have tried scheduling by workflow - it only takes you to the notebookeven the schedule from dashboard takes me to the notebook and not the dashbo...
- 2220 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Accn , Thanks for bringing up your concerns, always happy to help We understand your concern but right now there is only way to refresh a notebook dashboard is via scheduled jobs. To schedule a dashboard to refresh at a specified interval, click...
- 0 kudos
- 9982 Views
- 1 replies
- 1 kudos
Resolved! After moving mounted s3 bucket under unity catalog control, python file paths no longer work
I have been using a mounted external s3 bucket with json files up until a few days ago, when my company changed to using all file mounts under control of the unity catalog. Suddenly I can no loner run a command like:with open("/mnt/my_files/my_json....
- 9982 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @chrisf_sts ,Thanks for bringing up your concerns, always happy to help May I know which cluster access mode you are using to run the notebook commands? Can you please try to run this below command on Single user cluster access mode. "with open(...
- 1 kudos
- 2359 Views
- 1 replies
- 0 kudos
Running unit tests and hyperopt causes a broadcast variable exception
Hello,We are using hyperopt to train a model with relatively large train dataset.We've experience some performance issues and following the suggestions in this notebook, we broadcasted the dataset.To verify that broadcasting the dataset resolved the ...
- 2359 Views
- 1 replies
- 0 kudos
- 16245 Views
- 3 replies
- 6 kudos
Resolved! How to add I custom logging in Databricks
I want to add custom logs that redirect in the Spark driver logs. Can I use the existing logger classes to have my application logs or progress message in the Spark driver logs.
- 16245 Views
- 3 replies
- 6 kudos
- 6 kudos
1) Is it possible to save all the custom logging to its own file? Currently it is being logging with all other cluster logs (see image) 2) Also Databricks it seems like a lot of blank files are also being created for this. Is this a bug? this include...
- 6 kudos
- 2060 Views
- 1 replies
- 0 kudos
Importing data from S3 to Azure DataBricks Cluster with Unity Catalog in Shared Mode
Environment details:DataBricks on Azure, 13.3 LTS, Unity Catalog, Shared Cluster mode.Currently in the environment I'm in, we run imports from S3 with code like:spark.read.option('inferSchema', 'true').json(s3_path). When running on a cluster in Sha...
- 2060 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello Sha, We usually get such error while working with shared cluster mode assuming this your dev environment just to avoid errors, please use different clusters. However as a alternative solution in case if would like to keep the shared cluster the...
- 0 kudos
- 10460 Views
- 4 replies
- 0 kudos
CREATE TABLE does not overwrite location whereas CREATE OR REPLACE TABLE does
I am working on Azure Databricks, with Databricks Runtime version being - 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12). I am facing the following issue.Suppose I have a view named v1 and a database f1_processed created from the following comman...
- 10460 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Dhruv-22 , Based on the information you shared above, the "CREATE OR REPLACE" and "CREATE" commands in Databricks do have different behaviours, particularly when it comes to handling tables with specific target locations. The "CREATE OR REPLACE"...
- 0 kudos
- 10872 Views
- 1 replies
- 2 kudos
REDACTED_POSSIBLE_SECRET_ACCESS_KEY as part of column value result form aes_encrypt
Hi, i've encountered an error using base64/aes_encrypt, as result the string saved contains 'REDACTED_POSSIBLE_SECRET_ACCESS_KEY' at the end destroying the original data, rendering it useless undecryptable, is there a way to avoid this replacement in...
- 10872 Views
- 1 replies
- 2 kudos
- 2 kudos
I had the same issue, and my usage was similar to OP:base64(aes_encrypt(<clear_text>, unbase64(secret(<scope>, <key>))))Databricks support suggested to not call secret within the insert/update operation that writes to the table. After updating the py...
- 2 kudos
- 10912 Views
- 3 replies
- 1 kudos
Resolved! REPLACE TABLE AS SELECT is not working with parquet whereas it works fine for delta
I am working on Azure Databricks, with Databricks Runtime version being - 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12). I am facing the following issue.Suppose I have a view named v1 and a database f1_processed created from the following comman...
- 10912 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Dhruv-22 We understand that you are facing the following error when using REPLACE TABLE AS SELECT on the Parquet Table but at this moment the REPLACE TABLE AS SELECT operation you're trying to perform is not supported for Parquet tables. Accord...
- 1 kudos
- 2498 Views
- 2 replies
- 0 kudos
Near Real time Solutioning on data from Core System which gets updated
We are trying to build solution , where customer data stored in one of RDBM database SQL server and we are moving this data to delta lake in medallion architecture and want to this to be near real time by using DLT pipeline.Problem is that source tab...
- 2498 Views
- 2 replies
- 0 kudos
- 0 kudos
came across this matrix while reading about DLT what is read from complete and write to Incremental means ?
- 0 kudos
- 1956 Views
- 2 replies
- 0 kudos
SQL warehourse l in AZURE databricks trial account
Can we create SQL warehourse 2xsmall cluster in AZURE databricks trial account
- 1956 Views
- 2 replies
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 573 | |
| 349 | |
| 287 |
