- 103 Views
- 3 replies
- 0 kudos
SQL Warehouse error: "Cannot read properties of undefined (reading 'data')" when querying system tab
Queries that previously worked started failing in SQL Warehouse (Dashboards) without any changes on our side.The query succeeds, but fails to render results with error:"Cannot read properties of undefined (reading 'data')"This happens with:- system.b...
- 103 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi, Just had a look at this and I'm trying to replicate my end, can you confirm what type of compute you're using? And is it the SQL editor that it's failing in? Also what region and cloud you're using? I ran the cost query on serverless and it ran f...
- 0 kudos
- 125 Views
- 2 replies
- 2 kudos
Resolved! how to update not tracked column only in new row version in create_auto_cdc_flow
Hi, I'm using create_auto_cdc_flow, scd type 2. In source I have a metadata which tells the origin of the row. This column should not trigger new version row, so it is added to track_history_except_column_list. I don't want to add it to exception col...
- 125 Views
- 2 replies
- 2 kudos
- 2 kudos
@rplazaman This is a well-known limitation of create_auto_cdc_flow / AUTO CDC INTO — and unfortunately there is no native way to achieve exactly what you want within the API's parameters. Here's why, and what you can do about it:The Core ProblemThe t...
- 2 kudos
- 81 Views
- 2 replies
- 0 kudos
OversizedAllocationException with transformWithStateInPandas
Hello,I have a process that uses transformWithStateInPandas on a dataframe that has the content on entire files in on of the columns. Recently, the exception OversizedAllocationException has started happening. I have tried setting these configs in th...
- 81 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @twbde This is a genuinely tricky problem. Here's the diagnosis and the best available workarounds:Root Cause: useLargeVarTypes Is Not Wired Into transformWithStateInPandasYour instinct is correct. The spark.sql.execution.arrow.useLargeVarTypes co...
- 0 kudos
- 297 Views
- 5 replies
- 3 kudos
Resolved! Service Principal access notebooks created under /Workspace/Users
What permissions does a Service Principal need to run Databricks jobs that reference notebooks created by a user and stored in Git?Hi everyone,We are exploring the notebooks‑first development approach with Databricks Bundles, and we’ve run into a wor...
- 297 Views
- 5 replies
- 3 kudos
- 3 kudos
Thank you so much Ashwin, this provides a lot of clarity.1. Where to deploy Bundles in the workspaceWe plan to deploy the bundle using a service principal , so the bundle we plan to deploy under /Workspace/<service_principal>1. Create notebooks under...
- 3 kudos
- 223 Views
- 2 replies
- 0 kudos
S3A Connector Trying to Use AWS STS on Non-AWS S3 Endpoint
Hi everyone,I’m trying to write Delta tables to my S3-compatible (non-AWS) endpoint, and it was writing perfectly fine last week with the same setup. Now, without any changes on my end, it’s failing and giving me anUnknownException: (com.amazonaws.se...
- 223 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @demo-user , Can you share more information about your setup: Cluster type and DBR versionS3-compatible storage implementation (MinIO / something else?) AFAIK this is not supposed to work as Delta client in DBR relies on AWS STS to perform S3 comm...
- 0 kudos
- 189 Views
- 4 replies
- 3 kudos
Resolved! How to Convert a Lateral View to a Table Reference
Hi AllI have a view creation script in DataBricks which uses a lateral view to access columns in a structure held within an array field. It is working fine but I have noted that the LATERAL VIEW is now depracated and that I should be using a TABLE RE...
- 189 Views
- 4 replies
- 3 kudos
- 3 kudos
You can useCREATE OR REPLACE VIEW newview AS SELECT t1.field1, item.field2, item.field3 FROM table1 AS t1 INNER JOIN table2 AS t2 ON t1.id = t2.id , LATERAL EXPLODE(t1.structure) AS structureitem(item)
- 3 kudos
- 3697 Views
- 11 replies
- 4 kudos
Lakehouse federation for SQL server: database name with spaces
We're currently using lakehouse federation for various sources (Snowflake, SQL Server); usually succesful. However we've encountered a case where one of the databases on the SQL Server has spaces in its name, e.g. 'My Database Name'. We've tried vari...
- 3697 Views
- 11 replies
- 4 kudos
- 4 kudos
Hello all,We have a good amount of tables from an external ERP system that are being replicated to an existing dwh in an Azure SQL Server database.We have set up a foreign connection for this database and we can connect to the server and database. Sa...
- 4 kudos
- 108 Views
- 5 replies
- 2 kudos
Can't Migrate Auto Loader To File Events
Hi there,I am trying to migrate my auto loader job to use file events, but it's failing with this error:com.databricks.sql.util.UnexpectedHttpStatus: Failed to list objects. There are problems on the location that need to be resolved. Details: Failed...
- 108 Views
- 5 replies
- 2 kudos
- 2 kudos
I'm so sorry for this. Turns out I've been assigning roles to the wrong service account. I recently got access to the Storage Credential in Databricks and noticed the different service account.These roles were enough:Storage Blob Data Contributor (st...
- 2 kudos
- 241 Views
- 3 replies
- 0 kudos
Lakeflow Connect Snapshot Restarts After Gateway Restart
Hi everyoneWe recently started using Lakeflow Connect to ingest data from our on-prem SQL Server environment, and we’ve run into an issue related to gateway restarts.From our understanding, the gateway begins by performing full snapshots of each tabl...
- 241 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @DavidOldelius , If the initial snapshot has been fully ingested, then in case of ingestion gateway restart the connector will start from where it left and not from the beginning. Do you observe the behaviour you described during or after the snap...
- 0 kudos
- 2257 Views
- 4 replies
- 0 kudos
How to pass parameters to a notebook via URL?
I have a Notebook written in Python, and a Notebook's Dashboard that shows its result.The Notebook has a parameters, processed with a text widget: dbutils.widgets.text("p_test", "", "Enter Test Value") p_test = dbutils.widgets.get("p_test") When I en...
- 2257 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi, were you able to figure this out? This is important for dashboard embedding as well.https://<account>.cloud.databricks.com/embed/dashboardsv3/<dashboard>?o=<id>&p_param1=101&p_param_2=500&p_status=activethis is not passing the parameters as expec...
- 0 kudos
- 173 Views
- 4 replies
- 2 kudos
Resolved! Netsuite Data Connector Not Available
I see that the Azure Databricks Data Connector for Netsuite is in Public Preview. Unfortunately I am unable to see it in my instance. I have gone into Preview and selected it as enabled, have downloaded the JAR file from Netsuite and have setup the i...
- 173 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @rwhitepwt, From what I can see, having the NetSuite connector in Public Preview doesn’t automatically guarantee that the tile appears in every workspace. In addition to enabling the preview and creating the UC connection + uploading the SuiteAnal...
- 2 kudos
- 42672 Views
- 4 replies
- 4 kudos
Resolved! How to get the Job ID and Run ID and save into a database
We are having Databricks Job running with main class and JAR file in it. Our JAR file code base is in Scala. Now, when our job starts running, we need to log Job ID and Run ID into the database for future purpose. How can we achieve this?
- 42672 Views
- 4 replies
- 4 kudos
- 4 kudos
i came across a similar requirement and got it through named parameters. Wrote a community blog about it
- 4 kudos
- 476 Views
- 11 replies
- 6 kudos
Resolved! OrderBy is not sorting the results
Hi,I am currently using Lakeflow SDP ,firstly I am creating 2 views and then joining them and creating materialized view and using order by in the materialized view create function , but the results are not sorted does order by not work on materializ...
- 476 Views
- 11 replies
- 6 kudos
- 55 Views
- 0 replies
- 0 kudos
Connect Databrick's cluster with Artifactory
Hello,I'm trying to connect databricks with an own JFrog Artifactory. The objective is to download both PIP/JAR dependencies from it instead of connecting to maven-central/PyPi. Im struggling with JAR's. My aproximation to solve the problem is:1. Cre...
- 55 Views
- 0 replies
- 0 kudos
- 220 Views
- 3 replies
- 0 kudos
Structured streaming error- NON_TIME_WINDOW_NOT_SUPPORTED_IN_STREAMING
Hi,I was using window function row_number(),min,sum in the code, then the Lakeflow SDP pipeline was failing with the error: NON_TIME_WINDOW_NOT_SUPPORTED_IN_STREAMING - Window function is not supported on streaming dataframeswhat is the recommended a...
- 220 Views
- 3 replies
- 0 kudos
- 0 kudos
@Louis_Frolio suppose if I use foreachbatch I might end up with duplicates as the state is not maintainedcan you please share more information on max_by
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
