- 2588 Views
- 4 replies
- 0 kudos
Databricks Runtime 13.3 - can I use Databricks Connect without Unity Catalog?
Hello all,The official documentation for Databricks Connect states that, for Databricks Runtime versions 13.0 and above, my cluster needs to have Unity Catalog enabled for me to use Databricks Connect, and use a Databricks cluster through an IDE like...
- 2588 Views
- 4 replies
- 0 kudos
- 0 kudos
Did you configure databricks connect without UnitCatalog at the end? If you managed to do that can you share with me how?
- 0 kudos
- 8194 Views
- 2 replies
- 2 kudos
Identity column value of Databricks delta table is not started with 0 and increaed by 1. It always started with something like 1 or 2 and increased by 2. Below is the sample code and any logical input here is appreciated
spark.sql("CREATE TABLE integrated.TrailingWeeks(ID bigint GENERATED BY DEFAULT AS IDENTITY (START WITH 0 increment by 1) ,Week_ID int NOT NULL) USING delta OPTIONS (path 'dbfs:/<Path in Azure datalake>/delta')")
- 8194 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi,When you define an identity column in Databricks with GENERATED BY DEFAULT AS IDENTITY (START WITH 0 INCREMENT BY 1), it is expected to start at 0 and increment by 1. However, due to Databricks' distributed architecture, the values may not be str...
- 2 kudos
- 307 Views
- 2 replies
- 0 kudos
Cluster is not starting
Cluster 'xxxxxxx' was terminated. Reason: WORKER_SETUP_FAILURE (SERVICE_FAULT). Parameters: databricks_error_message:DBFS Daemomn is not reachable., gcp_error_message:Unable to reach the colocated DBFS Daemon.Can Anyone help me how can we resolve thi...
- 307 Views
- 2 replies
- 0 kudos
- 20101 Views
- 9 replies
- 9 kudos
Cluster occasionally fails to launch
I have a daily running notebook that occasionally fails with the error:"Run result unavailable: job failed with error message Unexpected failure while waiting for the cluster Some((xxxxxxxxxxxxxxx) )to be readySome(: Cluster xxxxxxxxxxxxxxxx is in un...
- 20101 Views
- 9 replies
- 9 kudos
- 9 kudos
Cluster 'xxxxxxx' was terminated. Reason: WORKER_SETUP_FAILURE (SERVICE_FAULT). Parameters: databricks_error_message:DBFS Daemomn is not reachable., gcp_error_message:Unable to reach the colocated DBFS Daemon.Can Anyone help me how can we resolve thi...
- 9 kudos
- 15445 Views
- 9 replies
- 6 kudos
Resolved! Downloading sharepoint lists using python
Hello, I am trying to download lists from SharePoint into a pandas dataframe. However I cannot get any information successfully. I have attempted many solution mentioned in stackoverflow. Below is one of those attempts: # https://pypi.org/project/sha...
- 15445 Views
- 9 replies
- 6 kudos
- 6 kudos
The error "<urlopen error [Errno -2] Name or service not known>" suggests that there's an issue with the server URL or network connectivity. Double-check the server URL to ensure it's correct and accessible. Also, verify that your network connection ...
- 6 kudos
- 2325 Views
- 5 replies
- 0 kudos
Resolved! Problem with SQL Warehouse (Serverless)
I get the following error message on the attempt to use SQL Warehouse (Serverless) compute with Materialized Views (a simple interaction, e.g. DML, UI sample lookup). The MVs are created off the back of Federated Tables (Postgresql), MVs are created ...
- 2325 Views
- 5 replies
- 0 kudos
- 0 kudos
Hey,To clarify, as I think I'm potentially hitting Databricks unintended "functionality".Materialised Views are managed by DLT pipeline, which was deployed with DABs off CI/CD pipeline,DLT Pipeline runs a notebook with Python code creating MVs dynami...
- 0 kudos
- 1935 Views
- 2 replies
- 0 kudos
Parameterized Delta live table pipeline
I'm trying to create an ETL framework on delta live tables and basically use the same pipeline for all the transformation from bronze to silver to gold. This works absolutely fine when I hard code the tables and the SQL transformations as an array wi...
- 1935 Views
- 2 replies
- 0 kudos
- 502 Views
- 1 replies
- 0 kudos
SparkRuntimeException: [UDF_ERROR.ENV_LOST] the execution environment was lost during execution
Hey everyone,I have been facing a weird error when i upgrade to use Unity Catalog.org.apache.spark.SparkRuntimeException: [UDF_ERROR.ENV_LOST] Execution of function line_string_linear_interp(geometry#1432) failed - the execution environment was lost ...
- 502 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @calvinchan_iot, How are you doing today?As per my understanding, It sounds like the error may be due to environment instability when running the UDF after enabling Unity Catalog. The [UDF_ERROR.ENV_LOST] error often points to the UDF execution en...
- 0 kudos
- 420 Views
- 2 replies
- 0 kudos
Glue Catalog Metadata Management with Enforced Tagging in Databricks
As part of the data governance team, we're trying to enforce table-level tagging when users create tables in a Databricks environment where metadata is managed by AWS Glue Catalog (non-Unity Catalog). Is there a way to require tagging at table creati...
- 420 Views
- 2 replies
- 0 kudos
- 0 kudos
You can use lakeFS pre-merge hooks to force this. Works great with this stack -> https://lakefs.io/blog/lakefs-hooks/
- 0 kudos
- 325 Views
- 1 replies
- 0 kudos
Comparing Ultrawide Curved Monitors and Dual-Monitor Setups for Databricks Projects
Hello everyone,I’m currently exploring the best setup for my data engineering tasks in Databricks and have been considering the benefits of using an ultrawide curved monitor compared to a standard dual-monitor setup.I’d love to hear from the communit...
- 325 Views
- 1 replies
- 0 kudos
- 0 kudos
Here’s a clearer version:I don’t use a curved monitor; instead, I have two 35" monitors, which work perfectly for my Databricks work. I chose two large monitors over one extra-large one because I frequently share screens, and it’s easier to share an ...
- 0 kudos
- 8155 Views
- 9 replies
- 4 kudos
Resolved! Parsing 5 GB json file is running long on cluster
I was creating delta table from ADLS json input file. but the job was running long while creating delta table from json. Below is my cluster configuration. Is the issue related to cluster config ? Do I need to upgrade the cluster config ?The cluster ...
- 8155 Views
- 9 replies
- 4 kudos
- 4 kudos
with multiline = true, the json is read as a whole and processed as such.I'd try with a beefier cluster.
- 4 kudos
- 447 Views
- 2 replies
- 2 kudos
Dark mode broken by "New version of this app is available" popup
Hello,I have the interface theme set to "Prefer dark" in Databricks. I just got a popup in the Workflow page while reviewing a job run, that said something like "A new version of this app is available, click to refresh". I clicked refresh, and now my...
- 447 Views
- 2 replies
- 2 kudos
- 2 kudos
I just got another "New version of this app is available" popup, and clicking "Refresh" fixedthe dark mode issue. Thanks for the quick response to whichever engineer at Databricks just pushed a hotfix
- 2 kudos
- 9414 Views
- 12 replies
- 7 kudos
Value array {{QUERY_RESULT_ROWS}} in Databricks SQL alerts custom template
Please include in documentation an example how to incorporate the `QUERY_RESULT_ROWS` variable in the custom template.
- 9414 Views
- 12 replies
- 7 kudos
- 7 kudos
Databricks confirmed this was an issue on their end and it should be resolved now. It is working for me.
- 7 kudos
- 3857 Views
- 3 replies
- 1 kudos
How to migrate power bi reports to databricks
I have a sample set of power bi(.pbix) reports with all dropdowns, tables, filters etc. Now I would like to migrate this reports to data bricks. whatever visuals are created in power bi, I would like to create same in data bricks from scratch. I wou...
- 3857 Views
- 3 replies
- 1 kudos
- 1 kudos
We must ensure you are sure that the Databricks cluster is operational. These are the steps needed for integration between Azure Databricks into Power BI Desktop.1. Constructing the URL for the connectionConnect to the cluster, and click the Advanced...
- 1 kudos
- 770 Views
- 2 replies
- 0 kudos
Dashboard use 'Multiple values' in underlying query
Currently, it appears that the Dashboards functionality does not support linking a 'Multiple Values' widget to a query parameter, nor does it allow the creation of a line plot with multiple lines.We are developing a dashboard where users need to visu...
- 770 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello,thank you for your reply.Notebooks are not really a workaround for me here but thank you for the walk through.I think this feature is very important so I hope this reaches your backlog somehow. As this is something a Grafana for example is able...
- 0 kudos