- 3219 Views
- 4 replies
- 3 kudos
What's the recommended way of creating tables in Databricks with unity catalog (External/Managed)
I have databricks with unity catalog enables and created an external ADLS location. when I create the catalog/schema it uses the external location. when I try to create the table it uses the extrenal location but they are managed tables. What's the r...
- 3219 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi Shivap,How are you doing today?, As per my understanding, in Unity Catalog, if you want to create an external table, you just need to make sure the external location is registered and approved first. Even though you're specifying a path with LOCAT...
- 3 kudos
- 4591 Views
- 8 replies
- 2 kudos
Resolved! When will DB release runtime with Scala 2.13
When will DB release runtime with Scala 2.13? Thx
- 4591 Views
- 8 replies
- 2 kudos
- 2 kudos
Spark 4 is coming and it's only going to support Scala 2.13, so a Databricks Runtime with Spark 3.5.x and Scala 2.13 should be released soonish.
- 2 kudos
- 4133 Views
- 1 replies
- 1 kudos
Job Retry Wait Policy and Cluster Shutdown
Hi all,I have a Databricks Workflow job in which the final task makes an external API call. Sometimes this API will be overloaded and the call will fail. In the spirit of automation, I want this task to retry the call an hour later if it fails in the...
- 4133 Views
- 1 replies
- 1 kudos
- 1 kudos
What about this? Any reply? Any alternative? I'm facing the same issue.
- 1 kudos
- 3750 Views
- 6 replies
- 0 kudos
Can external tables be created backed by current cloud files without ingesting files in Databricks?
Hi,We have huge amount of parquet files in s3 with the path pattern <bucket>/<customer>/yyyy/mm/dd/hh/.*.parquet.The question is can I create a external table in Unity Catalog from this external location without actually ingesting the files? Like wha...
- 3750 Views
- 6 replies
- 0 kudos
- 0 kudos
i think the issue is that you are trying to create a DELTA table in Unity catalog from an Parquet source without converting it to Delta format first.As Unity catalog will not allow delta table to be created in an non-empty location. Since you want t...
- 0 kudos
- 3571 Views
- 3 replies
- 0 kudos
Maven library installation issue on 15.4 LTS
recently i upgraded from 10.4LTS databricks runtime version to 15.4LTS version. while installing Maven library i was facing issue like :Library installation attempted on the driver node of cluster 0415-115331-dune977 and failed. Library resolution fa...
- 3571 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Rakesh007, Good Day!This seems to be a compatibility issue with the Apache Spark version, as the DBR 15.4LTS supports 3.5.0. Please try installing the below version and let us know how it goes?Version: com.crealytics:spark-excel_2.12:3.5.0_0.20.3...
- 0 kudos
- 8972 Views
- 2 replies
- 0 kudos
Why "rror: Invalid access to Org: xxx"
Hi team, I installed Databricks CLI, and run "databricks auth login --profile xxx" successfully. I can also connect from vscode to Databricks. "databricks clusters list -p xxx" also works. But when I tried to rundatabricks bundle validateI got"Error:...
- 8972 Views
- 2 replies
- 0 kudos
- 0 kudos
I just ran into this issue (in Azure Databricks) and found that it was caused by an incorrect `host` value specified in my databricks.yml file:targets: dev: default: true mode: production workspace: host: https://adb-<workspace-id...
- 0 kudos
- 1472 Views
- 3 replies
- 0 kudos
GCE cluster chokes the secret api server.
Hi,We upgraded the GKE cluster to GCE cluster as per the databricks documentation. It works fine with one or two notebooks in job. Our production job has more than 40 notebooks and each notebook access the secret api and seems like secret api server ...
- 1472 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Sadam97, This looks to be a known issue. I will share more details soon. There is a known issue with NAT and GKE and we’ll follow up offline
- 0 kudos
- 4847 Views
- 4 replies
- 4 kudos
Is it possible to use Liquid Clustering on Delta Live Tables / Materialised Views?
Is it possible to use Liquid Clustering on Delta Live Tables? If it is available what is the Python syntax for adding liquid clustering to a Delta Live Table / Materialised view please?
- 4847 Views
- 4 replies
- 4 kudos
- 4 kudos
@Dlt.table(name=table_name,comment="just_testing",table_properties={"quality": "gold","mergeSchema": "true"},cluster_by=["test_id", "find_date"] # Optimizes for queries filtering on these columns)def testing_table():return create_testing_table(df_fin...
- 4 kudos
- 1978 Views
- 2 replies
- 0 kudos
Access mode for pool compute
Is there a way to set Access Mode: Shared to pool instances similar to All Purpose or Job clusters?We are getting an error reading from a table with a masking set up on a column:Failed to acquire a SAS token for list on /schema1/table1/_delta_log due...
- 1978 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Brahmareddy, thanks for reply. It seems that a higher Runtime version could help: https://learn.microsoft.com/en-us/azure/databricks/compute/access-mode-limitations#fine-grained-access-control-limitations-for-unity-catalog-dedicated-access-mode I...
- 0 kudos
- 6686 Views
- 5 replies
- 0 kudos
Self-joins are blocked on remote tables
In our production databricks workflow, we have been getting this error since yesterday in one of the steps:org.apache.spark.SparkException: Self-joins are blocked on remote tablesWe haven't changed our workflow or made any configurations for the data...
- 6686 Views
- 5 replies
- 0 kudos
- 0 kudos
@TomRenish Yeah, we fixed it by changing it to use a shared compute. It is called "USER_ISOLATION" in the `job.yaml` file:data_security_mode: USER_ISOLATION
- 0 kudos
- 1297 Views
- 1 replies
- 0 kudos
Databricks-Sql-Connector
Hi,i am connecting with databricks sql_warehouse using vs_code and i am running following command: import osfrom databricks import sqlhost = 'adb-xxxxxxxxxxx.xx.azuredatabricks.net'http_path = '/sql/1.0/warehouses/xxxxxxxxxxxxxx'access_token = 'dapib...
- 1297 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Upendra_Dwivedi , This is potentially a missing package in your local Python setup, kindly can you check troubleshooting steps here and let me know In the alternative this didn't work please share the output of the following commands: python ...
- 0 kudos
- 2548 Views
- 1 replies
- 0 kudos
enable dynamic resource allocation on job cluster
I have a databricks job having two task those will run each alone or both parallel (will be controlled by if conditional task). When it runs parallel, one task is running for long time, but the same task finish quick when it runs alone. particularly ...
- 2548 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Abser786, There is a difference between Dynamic Resource Allocation and the Scheduler policy Dynamic Resource Allocation means getting more compute as needed if current compute is totally consumed, this can be achieved by autoscaling feature/c...
- 0 kudos
- 6087 Views
- 3 replies
- 0 kudos
Any known issue with interactive Shared Cluster Driver Memory Cleanup
I am experiencing memory leaks on a Standard (formerly shared) interactive cluster: 1. We run jobs regularly on the cluster2. After each job completes, driver memory usage continues to increase, suggesting resources aren't fully released3. Eventually...
- 6087 Views
- 3 replies
- 0 kudos
- 1459 Views
- 1 replies
- 1 kudos
UC upgrade in Spark Streaming jobs
Kindly share the recommended approach for upgrading from HMS to UC for structured streaming jobs, ensuring seamless execution without any failures or data duplication? I would also appreciate insights into any best practices you have followed during ...
- 1459 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi Vetrivel,How are you doing today?, As per my understanding, Upgrading from Hive Metastore (HMS) to Unity Catalog (UC) for structured streaming jobs needs a careful approach to avoid failures or data duplication. The best way is to first pause all ...
- 1 kudos
- 1198 Views
- 1 replies
- 1 kudos
Resolved! How can I efficiently remove backslashes during a COPY INTO load in Databricks?
I’m using Databricks’ COPY INTO to load data from a CSV file into a Delta table. My input CSV looks like this: CSV filecolumn1(string),column2(string) "[\,\,111\,222\,]","012\"34"After running COPY INTO, my Delta table currently contains:column1(str...
- 1198 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi Yutaro,You're doing great, and your question is very clear! In your case, the most efficient way to remove backslashes during the COPY INTO operation is to first load the raw CSV data into a temporary or staging Delta table, and then insert the cl...
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
