- 6401 Views
- 5 replies
- 0 kudos
Resolved! Problem with SQL Warehouse (Serverless)
I get the following error message on the attempt to use SQL Warehouse (Serverless) compute with Materialized Views (a simple interaction, e.g. DML, UI sample lookup). The MVs are created off the back of Federated Tables (Postgresql), MVs are created ...
- 6401 Views
- 5 replies
- 0 kudos
- 0 kudos
Hey,To clarify, as I think I'm potentially hitting Databricks unintended "functionality".Materialised Views are managed by DLT pipeline, which was deployed with DABs off CI/CD pipeline,DLT Pipeline runs a notebook with Python code creating MVs dynami...
- 0 kudos
- 3379 Views
- 2 replies
- 0 kudos
Parameterized Delta live table pipeline
I'm trying to create an ETL framework on delta live tables and basically use the same pipeline for all the transformation from bronze to silver to gold. This works absolutely fine when I hard code the tables and the SQL transformations as an array wi...
- 3379 Views
- 2 replies
- 0 kudos
- 1161 Views
- 1 replies
- 0 kudos
SparkRuntimeException: [UDF_ERROR.ENV_LOST] the execution environment was lost during execution
Hey everyone,I have been facing a weird error when i upgrade to use Unity Catalog.org.apache.spark.SparkRuntimeException: [UDF_ERROR.ENV_LOST] Execution of function line_string_linear_interp(geometry#1432) failed - the execution environment was lost ...
- 1161 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @calvinchan_iot, How are you doing today?As per my understanding, It sounds like the error may be due to environment instability when running the UDF after enabling Unity Catalog. The [UDF_ERROR.ENV_LOST] error often points to the UDF execution en...
- 0 kudos
- 997 Views
- 2 replies
- 0 kudos
Glue Catalog Metadata Management with Enforced Tagging in Databricks
As part of the data governance team, we're trying to enforce table-level tagging when users create tables in a Databricks environment where metadata is managed by AWS Glue Catalog (non-Unity Catalog). Is there a way to require tagging at table creati...
- 997 Views
- 2 replies
- 0 kudos
- 0 kudos
You can use lakeFS pre-merge hooks to force this. Works great with this stack -> https://lakefs.io/blog/lakefs-hooks/
- 0 kudos
- 845 Views
- 1 replies
- 0 kudos
Comparing Ultrawide Curved Monitors and Dual-Monitor Setups for Databricks Projects
Hello everyone,I’m currently exploring the best setup for my data engineering tasks in Databricks and have been considering the benefits of using an ultrawide curved monitor compared to a standard dual-monitor setup.I’d love to hear from the communit...
- 845 Views
- 1 replies
- 0 kudos
- 0 kudos
Here’s a clearer version:I don’t use a curved monitor; instead, I have two 35" monitors, which work perfectly for my Databricks work. I chose two large monitors over one extra-large one because I frequently share screens, and it’s easier to share an ...
- 0 kudos
- 10121 Views
- 9 replies
- 4 kudos
Resolved! Parsing 5 GB json file is running long on cluster
I was creating delta table from ADLS json input file. but the job was running long while creating delta table from json. Below is my cluster configuration. Is the issue related to cluster config ? Do I need to upgrade the cluster config ?The cluster ...
- 10121 Views
- 9 replies
- 4 kudos
- 4 kudos
with multiline = true, the json is read as a whole and processed as such.I'd try with a beefier cluster.
- 4 kudos
- 963 Views
- 2 replies
- 2 kudos
Dark mode broken by "New version of this app is available" popup
Hello,I have the interface theme set to "Prefer dark" in Databricks. I just got a popup in the Workflow page while reviewing a job run, that said something like "A new version of this app is available, click to refresh". I clicked refresh, and now my...
- 963 Views
- 2 replies
- 2 kudos
- 2 kudos
I just got another "New version of this app is available" popup, and clicking "Refresh" fixedthe dark mode issue. Thanks for the quick response to whichever engineer at Databricks just pushed a hotfix
- 2 kudos
- 6053 Views
- 3 replies
- 1 kudos
How to migrate power bi reports to databricks
I have a sample set of power bi(.pbix) reports with all dropdowns, tables, filters etc. Now I would like to migrate this reports to data bricks. whatever visuals are created in power bi, I would like to create same in data bricks from scratch. I wou...
- 6053 Views
- 3 replies
- 1 kudos
- 1 kudos
We must ensure you are sure that the Databricks cluster is operational. These are the steps needed for integration between Azure Databricks into Power BI Desktop.1. Constructing the URL for the connectionConnect to the cluster, and click the Advanced...
- 1 kudos
- 1804 Views
- 2 replies
- 0 kudos
Dashboard use 'Multiple values' in underlying query
Currently, it appears that the Dashboards functionality does not support linking a 'Multiple Values' widget to a query parameter, nor does it allow the creation of a line plot with multiple lines.We are developing a dashboard where users need to visu...
- 1804 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello,thank you for your reply.Notebooks are not really a workaround for me here but thank you for the walk through.I think this feature is very important so I hope this reaches your backlog somehow. As this is something a Grafana for example is able...
- 0 kudos
- 1046 Views
- 1 replies
- 0 kudos
Difference between "deep clone ..." and as "select * from ..."
Hi all,I was trying to deep clone one of the sample tables provided with a parametrized query:create table if not exists IDENTIFIER(:target_catalog || :target_schema || :table_name) DEEP CLONE IDENTIFIER('samples.tpch.' || :table_name)But Databricks ...
- 1046 Views
- 1 replies
- 0 kudos
- 0 kudos
Deep clone will also clone all the metadata (e.g. indexes, properties, history, etc.), while SELECT * will create a new, fresh Delta Table, with it's own history and properties.
- 0 kudos
- 1337 Views
- 1 replies
- 0 kudos
Unity Catalog
Hi,I have requirement to setup the athena tables. We have a unity catalog setup in databricks workspace and I would like know is there any possibility that Athen can be point to unity catalog so that all the tables are available in athena.whenever we...
- 1337 Views
- 1 replies
- 0 kudos
- 0 kudos
Unfortunately, as of now, there isn't a direct, seamless integration between Unity Catalog and Athena to automatically synchronize table updates.However, here are a few potential approaches to achieve your desired outcome:1. AWS Glue Data Catalog:Man...
- 0 kudos
- 3084 Views
- 1 replies
- 1 kudos
Querying Unity Managed Tables from Redshift
I built a script about 6 months ago to make our Delta Tables accessible in Redshift for another team, but it's a bit nasty...Generate a delta lake manifest each time the databricks delta table is updatedRecreate the redshift external table (incase th...
- 3084 Views
- 1 replies
- 1 kudos
- 1 kudos
Still searching for the same pain point...may be in marketplace to integrate Unity Catalog and Redshift
- 1 kudos
- 987 Views
- 2 replies
- 0 kudos
I need to use the ojdbc jar file which is in workspace>Driver inside my notebook
Hi, Currently I'm working on a project . Where I need to connect with Oracle database from databricks notebook. To do this I need Ojdbc jar file. But my client has installed that jar file in workspace>Driver and not in the cluster library. So to use ...
- 987 Views
- 2 replies
- 0 kudos
- 0 kudos
do this in your notebook:# Path to the ojdbc JAR file in the workspacejar_path = "/Workspace/Driver/ojdbc8.jar"# Install the JAR file from the workspace into the clusterdbutils.library.install(jar_path)# Restart the cluster to ensure the JAR file is ...
- 0 kudos
- 1216 Views
- 5 replies
- 0 kudos
Need help with DLT Pipeline
I have a DLT pipeline running daily for months and recently found out one issue in my silver layer code and as a result of that, now I have faulty data in my silver schema. Please note that the tables in Silver schema are streaming tables handled wit...
- 1216 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi Fatimah, You can delete the records from silver layer as long as those records don't get reloaded again (from bronze). More info is here May I also understand that are you are using CDC apply_changes method for loading data to silver (like SCD 1)...
- 0 kudos
- 889 Views
- 2 replies
- 0 kudos
How to add existing recipient to existing delta share
I have created a recipient using the data bricks in the console, also, I have created data bricks using the console, Now I want to map the existing recipient to the existing delta share, Is there a way to do this using Terraform
- 889 Views
- 2 replies
- 0 kudos
- 0 kudos
@SathyaSDE Thank you for your response. I am looking into a Terraform-based approach for this.
- 0 kudos
| User | Count |
|---|---|
| 1615 | |
| 779 | |
| 349 | |
| 287 | |
| 267 |
