- 982 Views
- 1 replies
- 0 kudos
Unable to create a metastore for Unity Catalog as I don't have Account Admin rights
Hello guys,I just started learning Databricks. I created a Databricks workspace via the Azure Portal using the Trial (Premium - 14-Days Free DBUs) plan. The workspace name is `easewithdata-adb`.However,I do not currently see the option to create a Un...
- 982 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @harika5991 You're right about the root cause of your issue. Creating a Unity Catalog metastore requires Account Admin privileges, which is separate from just creating a workspace in Azure.These are options you can try:When you create a Databricks...
- 0 kudos
- 4977 Views
- 4 replies
- 4 kudos
Resolved! What are your most impactful use cases for schema evolution in Databricks?
Data Engineers, Share Your Experiences with Delta Lake Schema Evolution! We're calling on all data engineers to share their experiences with the powerful schema evolution feature in Delta Lake. This feature allows for seamless adaptation to changin...
- 4977 Views
- 4 replies
- 4 kudos
- 745 Views
- 1 replies
- 0 kudos
Resolved! ConcurrentDeleteDeleteException in liquid cluster table
I am doing a merge in a table in parallel via 2 jobs.The table is a liquid clustered table with the following properties:delta.enableChangeDataFeed=truedelta.enableDeletionVectors=truedelta.enableRowTracking=truedelta.feature.changeDataFeed=supported...
- 745 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @flashmav , keep in mind that operations in Delta Lake often occur at the file level rather than the row level. For example, if two sessions attempt to update data in the same file (even if they’re not updating the same row), you may encounter a...
- 0 kudos
- 9118 Views
- 6 replies
- 6 kudos
Issue with Docker Image connection
Hello, I have created and pushed a docker image to Azure Container Registry . I used that image to start the cluster in Databricks. But the cluster doesn't start and as such when I try to run a Databricks' Jon using that Cluster, I get this error bel...
- 9118 Views
- 6 replies
- 6 kudos
- 9866 Views
- 6 replies
- 4 kudos
DLT | Cluster terminated by System-User | INTERNAL_ERROR: Communication lost with driver. Cluster 0312-140502-k9monrjc was not reachable for 120 seconds
Dear Community, Hope you are doing well.For the last couple of days I am seeing very strange issues with my DLT pipeline, So every 60-70 mins it is getting failed in continuous mode, with the ERROR; INTERNAL_ERROR: Communication lost with driver. Clu...
- 9866 Views
- 6 replies
- 4 kudos
- 4 kudos
We had similar error for one the DLT pipeline, This could be some times because of compute size, we had increased compute size of server in your DLT pipelines, still we were seeing this error while processing very large file. we then added below para...
- 4 kudos
- 1399 Views
- 3 replies
- 1 kudos
Exporting table to GCS bucket using job
Hi all,Usecase: I want to send the result of a query to GCS bucket location in json format.Approach: From my java based application I create a job and that job will be running a notebook`. Notebook will have something like this```query = "SELECT * FR...
- 1399 Views
- 3 replies
- 1 kudos
- 5572 Views
- 6 replies
- 6 kudos
How to infer the online feature store table via an mlflow registered model, which is deployed to a sagemaker endpoint?
Can an mlflow registered model automatically infer the online feature store table, if that model is trained and logged via a databricks feature store table and the table is pushed to an online feature store (like AWS RDS)?
- 5572 Views
- 6 replies
- 6 kudos
- 836 Views
- 2 replies
- 0 kudos
Need access to browse onprem SQL data
Our BI team has started using Databricks and would like to browse our local (onprem) SQL database servers from within Databricks. I'm not sure if that's even possible.So far, I've set up Databricks Secure Cluster Connectivity (SCC), created a privat...
- 836 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi, based on what you’ve shared, it seems you’ve already completed many of the necessary steps. Just a few things to double-check as you move forward:SQL Warehouses used for BI tools need to run in Pro mode, not serverless, since only Pro or Classic ...
- 0 kudos
- 1081 Views
- 1 replies
- 1 kudos
Resolved! Row filtering and Column masking not working even when requirements the are met
I have been trying to use the Row filtering and Column masking feature to redacted columns and and filter rows based on the group a user is in.I have all the necessary permissions and I've used cluster's with version 15.4 and higher.When I run the fo...
- 1081 Views
- 1 replies
- 1 kudos
- 1 kudos
Here are some things to consider/try: The UnityCatalogServiceException error you are encountering, ABORTED.UC_DBR_TRUST_VERSION_TOO_OLD, generally indicates that the Databricks Runtime (DBR) version you are using no longer supports the operation, s...
- 1 kudos
- 882 Views
- 1 replies
- 0 kudos
Column Default Propagation
Hi Today I found I somewhat strange behavior when it comes to default values in columns. Apparently, column defaults are propagated to a new table, when you select the column without any operation on it. This is a bit unexpected for me. Here a short...
- 882 Views
- 1 replies
- 0 kudos
- 0 kudos
The behavior you described regarding the propagation of default column values is expected and is tied to the specific usage of the delta.feature.allowColumnDefaults table property in Delta Lake. Here’s an explanation: Default Propagation Without Tra...
- 0 kudos
- 778 Views
- 2 replies
- 3 kudos
- 778 Views
- 2 replies
- 3 kudos
- 3 kudos
Esse erro ocorre quando sua conta Azure excede a cota regional de núcleos disponÃveis, impedindo a criação do cluster no Databricks. Isso significa que o cluster tentou utilizar mais recursos do que o permitido na sua região.1-Revise a configuração d...
- 3 kudos
- 3134 Views
- 6 replies
- 3 kudos
Resolved! [Databricks Asset Bundles] Triggering Delta Live Tables
I would like to know how to schedule a DLT pipeline using DAB's.I'm trying to trigger a Delta Live Table pipeline using Databricks Asset Bundles. Below is my YAML code:resources: pipelines: data_quality_pipelines: name: data_quality_pipeline...
- 3134 Views
- 6 replies
- 3 kudos
- 3 kudos
As of now, Databricks Asset Bundles do not support direct scheduling of DLT pipelines using cron expressions within the bundle configuration. Instead, you can achieve scheduling by creating a Databricks job that triggers the DLT pipeline and then sch...
- 3 kudos
- 2963 Views
- 4 replies
- 1 kudos
How to use change data feed when schema is changing between delta table versions?
How to use change data feed when delta table schema changes between delta table versions?I tried to read change data feed in parts (in code snippet I read version 1372, because 1371 and 1373 schema versions are different), but getting errorUnsupporte...
- 2963 Views
- 4 replies
- 1 kudos
- 1 kudos
@LasseL When you read from the change data feed in batch mode, Delta Lake always uses a single schema:By default, it uses the latest table version’s schema, even if you’re only reading an older versionOn Delta Runtime ≥ 12.2 LTS with column mapping e...
- 1 kudos
- 1086 Views
- 3 replies
- 0 kudos
ETL pipeline
I have an ETL pipeline in workflows which I am using to create materialized view. I want to schedule the pipeline for 10 hours only starting from 10 am. How can I schedule that? I can only see hourly basis schedule or cron syntax. I want the compute ...
- 1086 Views
- 3 replies
- 0 kudos
- 386 Views
- 0 replies
- 0 kudos
New remote (dbfs) caching python library
I had some problems getting much speedup at all from spark or DB disk cache, which I think is essential when developing PySpark code iteratively in notebooks. So I developed a handy caching-library for this which has recently been open sourced, see h...
- 386 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 493 | |
| 349 | |
| 287 |
