- 1221 Views
- 1 replies
- 0 kudos
Bootstrap Timeout during job cluster start
My job was not able to start because I got this problem in the job cluster.This job is running on a Azure Databricks workspace that has been deployed for almost a year and I have not had this error before. It is deployed in North Europe.After getting...
- 1221 Views
- 1 replies
- 0 kudos
- 0 kudos
We have the same problem randomly occurring since yesterday in two workspaces.The cluster started fine today in the morning at 08:00, but failed again from around 09:00 on.
- 0 kudos
- 4941 Views
- 1 replies
- 1 kudos
One-time backfill for DLT streaming table before apply_changes
Hi,absolute Databricks noob here, but I'm trying to set up a DLT pipeline that processes cdc records from an external sql server instance to create a mirrored table in my databricks delta lakehouse. For this, I need to do some initial one-time backfi...
- 4941 Views
- 1 replies
- 1 kudos
- 1 kudos
So since nobody responded, I decided to try my own suggestion and hack the snapshot data into the table that gathers the change data capture. After some straying I ended up with the notebook as attached.The notebook first creates 2 dlt tables (lookup...
- 1 kudos
- 5266 Views
- 2 replies
- 4 kudos
- 5266 Views
- 2 replies
- 4 kudos
- 4 kudos
Yes it does, !!https://databricks.com/session/secured-kerberos-based-spark-notebook-for-data-science
- 4 kudos
- 1290 Views
- 1 replies
- 1 kudos
Job Claiming NotebooKNotFound Incorrectly (seemingly)
I have the code captured below in the screenshot. When I run this individually it works just fine, when I JOB runs this it fails out with 'ResourceNotFound' - not sure what the issue is... - Checked 'main' branch, which is where this job is pulling f...
- 1290 Views
- 1 replies
- 1 kudos
- 1 kudos
Figured it out:ecw_staging_nb_List = ['nb_UPSERT_stg_ecw_insurance','nb_UPSERT_stg_ecw_facilitygroups']Works just fine.
- 1 kudos
- 2101 Views
- 0 replies
- 0 kudos
Selective Overwrite to a Unity Catalog Table
I have been able to perform a selective overwrite using replace Where to a hive_metastore table, but when I use the same code for the same table in a unity catalog, no data is written.Has anyone else had this issue or is there common mistakes that ar...
- 2101 Views
- 0 replies
- 0 kudos
- 6548 Views
- 6 replies
- 7 kudos
Dbutils.notebook.run command not working with /Repos/
I have two github repo configured in Databricks Repos folder. repo_1 is run using a job and repo_2 is run/called from repo_1 using Dbutils.notebook.run command. dbutils.notebook.run("/Repos/repo_2/notebooks/notebook", 0, args)i am getting the follo...
- 6548 Views
- 6 replies
- 7 kudos
- 7 kudos
I am having a similar issue... ecw_staging_nb_List = ['/Workspace/Repos/PRIMARY/UVVC_DATABRICKS_EDW/silver/nb_UPSERT_stg_ecw_insurance', '/Repos/PRIMARY/UVVC_DATABRICKS_EDW/silver/nb_UPSERT_stg_ecw_facilitygroups'] Adding workspace d...
- 7 kudos
- 935 Views
- 0 replies
- 0 kudos
Optimization failed for timestampNtz
We have a table using timestampNtz type for timestamp, which is also a cluster key for this table using liquid clustering. I ran OPTIMIZE <table-name>, it failed with errorUnsupported datatype 'TimestampNTZType' But the failed optmization also broke ...
- 935 Views
- 0 replies
- 0 kudos
- 3658 Views
- 2 replies
- 0 kudos
Export Users and Groups from Unity Catalog
Hi,I am trying to export the list of users and groups from Unity catalog through databricks workspace but i am seeing only the users/groups created inside the workspace instead of the groups and users coming through scim in unity catalog.How can i ge...
- 3658 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello when you refer to the users and groups in Unity Catalog, do you refer to the ones created at the Account Level?If this is the case you need to run the API call at the account level and not workspace level, you can see the API doc for account le...
- 0 kudos
- 2772 Views
- 1 replies
- 0 kudos
Trigger a job on file update
I'm using AutoLoader to process any new file or update that arrives to my landing area. And then I schedule the job using DB workflows to trigger on file arrival. The issue is that the trigger only executes when new files arrive, not when an exiting ...
- 2772 Views
- 1 replies
- 0 kudos
- 0 kudos
I don't think you can effectively achieve your goal. While it's theoretically somewhat possible, Databricks documentation says there is no guarantee for correctness - Auto Loader FAQ | Databricks on AWS
- 0 kudos
- 8740 Views
- 2 replies
- 1 kudos
When reading a csv file with Spark.read, the data is not loading in the appropriate column while pas
I am trying to read a csv file from storage location using spark.read function. Also, i am explicitly passing the schema to the function. However, the data is not loading in proper column of the dataframe. Following are the code details:from pyspark....
- 8740 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi , i would suggest to approach as suggested by Thomaz Rossito,but maybe you can give it as an try like swapping the struct field order like this followingschema = StructType([StructField('DA_RATE', DateType(), True),StructField('CURNCY_F', StringTy...
- 1 kudos
- 7651 Views
- 3 replies
- 0 kudos
Resolved! OPTIMIZE: Exception thrown in awaitResult: / by zero
We run `OPTIMIZE` on our tables every 24 hours as follows:spark.sql(f'OPTIMIZE {catalog_name}.{schema_name}.`{table_name}`;') This morning one of our hourly jobs started failing on the call to `OPTIMIZE` with the error:org.apache.spark.SparkException...
- 7651 Views
- 3 replies
- 0 kudos
- 6722 Views
- 1 replies
- 1 kudos
Resolved! Cluster pools
Could you help me understand pools? How to know the difference in pricing between running clusters and running clusters with a pool? Since we're saving time to start/stop the cluster when we have a pool. And should we keep Min Idle above 0 or equal t...
- 6722 Views
- 1 replies
- 1 kudos
- 1 kudos
Databricks pools are a set of idle, ready-to-use instances. When a cluster is attached to a pool, cluster nodes are created using the pool’s idle instances. If the pool has no idle instances, the pool expands by allocating a new instance from the ins...
- 1 kudos
- 3127 Views
- 2 replies
- 0 kudos
Resolved! How to enable CDF when saveAsTable from pyspark code?
I'm running this code in databricks notebook and I want the table from dataframe in catalog were created with CDF enables. When I run the code table hasn't exited yet.This code doesn't create a table with enables CDF. It doesn't add:delta.enableChang...
- 3127 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @drag7ter ,I don't see anything wrong with your approach, check my repro:
- 0 kudos
- 8488 Views
- 9 replies
- 0 kudos
Can not access a sql file from Notebook
Hey,I have a repo of notebooks and SQL files, the typical way is to update/create notebooks in the repo then push it and CICD pipeline deploys the notebooks to the Shared workspace.the issue is that I can access the SQL files in the Repo but can not ...
- 8488 Views
- 9 replies
- 0 kudos
- 4650 Views
- 1 replies
- 0 kudos
Exam vochure
Hii I am planning to take Databricks certified data engineer assosciate certification. where can i get the exam vochure.
- 4650 Views
- 1 replies
- 0 kudos
- 0 kudos
The Help Center provides an additional forum for this topic. You can request a voucher by submitting a help request, however, vouchers are not provided in all cases. Other ways to obtain a voucher are participation in training events held throughout ...
- 0 kudos
| User | Count |
|---|---|
| 1629 | |
| 790 | |
| 511 | |
| 349 | |
| 287 |
