- 249 Views
- 2 replies
- 0 kudos
Resolved! serverless workflow Compute became unresponsive. Compute is likely out of memory.
I set up run 10 notebooks at same time in serverless workflowI got this error:serverless workflow Compute became unresponsive. Compute is likely out of memory.Is there a quota in serverless compute I can set in zure databricks? These notebooks are pr...
- 249 Views
- 2 replies
- 0 kudos
- 0 kudos
The issue is new, and azure cloud providers are also not quite sure the details, will get more info later
- 0 kudos
- 13617 Views
- 21 replies
- 16 kudos
Resolved! Was DBFS disabled from community edition?
HelloI'm trying to use the Upload data to DBFS... from Databricks community edition but it's disabled. I'm also trying to activate it using the path Settings>Advanced>Other but the option is not longer in the list. Is this a temporary or permanent mo...
- 13617 Views
- 21 replies
- 16 kudos
- 16 kudos
Noticed there a several questions/'workarounds' on this topic. I may have overlooked but again, has this feature been removed or is it currently integrated naturally? Appreciate the assistance, in advance.
- 16 kudos
- 80 Views
- 2 replies
- 1 kudos
dashboard cost databricks - AWS instance
Hello guys, I hope your day going well ! I have some question about your dashboard cost publish on github databricks quick lab: this dashboard describe both the cost of databricks and AWS instance EC2 or just databricks cost ? Thanx
- 80 Views
- 2 replies
- 1 kudos
- 1 kudos
@seefoods, also worth noting that you might not be able to see the system tables without a metastore admin granting the appropriate access. That was something I learned in Azure : https://docs.databricks.com/aws/en/admin/ All the best,BS
- 1 kudos
- 155 Views
- 1 replies
- 0 kudos
Move/Merge Catalogs
HiWe have a couple of unity catalogs. In each are schema's, some used, some old, same named temp_data etc. etc Within those schema are tables with the same erratic approach. The result of a couple of years of bad housekeeping. (who not experienced th...
- 155 Views
- 1 replies
- 0 kudos
- 0 kudos
For moving tables and entire schemas consider looking at the "SYNC" command. The SYNC Command - Your Primary Migration Tool Databricks provides a SYNC command specifically designed for this exact use case - migrating tables from one location to ano...
- 0 kudos
- 123 Views
- 3 replies
- 1 kudos
Permission issue for pandas to read local files
I can use pandas to read local files in a notebook, such as those located in tmp.However, when I run two consecutive notebooks within the same job and read files with pandas in both, I encounter a permission error in the second notebook stating that ...
- 123 Views
- 3 replies
- 1 kudos
- 1 kudos
@liu is there anything preventing you from moving this file into a Volume on the Unity Catalog & then reading it from there?Based on your original message, the "permission" message is making me feel like it's having an issue with opening the file; pe...
- 1 kudos
- 112 Views
- 1 replies
- 0 kudos
Error with AutoLoader pipeline ingesting from external location: LOCATION_OVERLAP
Hello,I am trying to use pipelines in Databricks to ingest data from an external location to the datalake using AutoLoader, and I am facing this issue. I have noticed other posts with similar errors, but in those posts, the error was related to the d...
- 112 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @mattstyl-ff As you can see the error : ErrorClass=INVALID_PARAMETER_VALUE.LOCATION_OVERLAP]Databricks automatically manages the storage location under the UC catalog’s storage root.either you don’t need to (and shouldn’t) set schemaLocation or...
- 0 kudos
- 60 Views
- 1 replies
- 0 kudos
unable to create table on external location
Currently trying to connect a table on external location and it fails with error -[UNAUTHORIZED_ACCESS] Unauthorized access: PERMISSION_DENIED: request not authorized SQLSTATE: 42501which seems like a pretty straight forward error but unable to find ...
- 60 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @PratikRudra Thank you for sharing the error: I think probably there is a component that is missing. Writing table metadata (for example, to the _delta_log directory) requires the CREATE EXTERNAL TABLE capability on the external location; this ...
- 0 kudos
- 303 Views
- 7 replies
- 7 kudos
Resolved! MATERIALZIED VIEW TRIGGER ON UPDATE with external table as upstream table
goal: implement event driven architecture without trigger on file arrivalI would like to know whether materialzied view can update itself when its source table which is external table updated.given that the source external table referencing data in d...
- 303 Views
- 7 replies
- 7 kudos
- 7 kudos
it seems that my idea is a bad idea because it seems that materialzied view doesn't support incremental udpate for external locationIncremental refresh for materialized views - Azure Databricks | Microsoft Learn
- 7 kudos
- 148 Views
- 1 replies
- 0 kudos
Changing paths to tables
HiMy organization has many notebooks that reference tables in schemas with the three part pathcatalog.schema.tablenameWith a lack of foresight we hardcoded all of these paths in the code and now the inevitable is happening and there is a need to rest...
- 148 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @IONA ,Definitely, I would say that’s even a common practice. Create a feature branch and make the necessary changes there. But once a day, merge into that feature branch all the changes that have appeared on your main branch. That way, you will a...
- 0 kudos
- 640 Views
- 10 replies
- 4 kudos
Resolved! Intermittent failures with auto loader on Azure
We've been using auto loader to ingest data from a storage account on Azure (format "cloudFiles").Today, we're starting to see failures during the setup of event notification:25/09/11 19:06:28 ERROR MicroBatchExecution: Non-interrupted exception thro...
- 640 Views
- 10 replies
- 4 kudos
- 4 kudos
Hello @Malthe @Saska @MehdiJafariWhen it was showing such error, did you try any alternative methods like using databricks service credentials. For example stream = spark.readStream.format("cloudFiles") .option("cloudFiles.format", "json") .option...
- 4 kudos
- 87 Views
- 1 replies
- 1 kudos
Simulating Real Time Streaming in Databricks free edition
Use Case:Kafka real time steaming network telemetry logsIn a real use case approx. 40 TB of data can be real time streamed in a dayArchitecture:issue encountered: when i try to simulate kakfa like streaming in databricks itself , as this is a free e...
- 87 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @ManojkMohan ,Unfortunately there is no workaround here. Free Edition supports only serverless compute. And serverless has following streaming limitation - one of which you just encountered - there is no support for default or time-based trigger i...
- 1 kudos
- 250 Views
- 4 replies
- 2 kudos
Resolved! Problems with cluster shutdown in DLT
[Issue] DLT finishes processing, but cluster remains active due to log write errorHi everyone, I'm running into a problem with my DLT pipeline and was hoping someone here could help or has experienced something similar.Problem DescriptionThe pipeline...
- 250 Views
- 4 replies
- 2 kudos
- 2 kudos
Can you please try one more option. If you’re on Preview, move to Current (or vice versa). Sometimes the regression only exists in one channel.
- 2 kudos
- 70 Views
- 1 replies
- 0 kudos
Decoupling power bi reports from schemas
HiI'm sure many of you have power bi reports that use the native Databricks connector to pull data from a schema to fuel a wonderful dashboard. If the source schemes was moved to a different catalog or a table renamed then the pbi connection would be...
- 70 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @IONA ,You can parametrize your data source using power query parameters. Just define parameter for catalog, schema etc. I have such setup in one of the project. We parametrized our PBI semantic models and when we are deploying semantic model to a...
- 0 kudos
- 1752 Views
- 7 replies
- 4 kudos
- 1752 Views
- 7 replies
- 4 kudos
- 4 kudos
Did anyone know when we will be going to receive the partner tech summit badge?
- 4 kudos
- 134 Views
- 3 replies
- 3 kudos
Resolved! Issue with Auto Liquid clustering
I have written data to a table using clusterByAuto set to trueBut the clustering keys are not selected automatically when i do a desc detail on the table.Screenshot belowWhy are clustering columns not being selected automatically?Repro steps:Create ...
- 134 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @RevanthV ,As @K_Anudeep correctly suggested it could be the case that your table is to small to benefit from liquid clustering.Another possibility it that you're using runtime lower than 15.4 LTS.
- 3 kudos
| User | Count |
|---|---|
| 1616 | |
| 784 | |
| 349 | |
| 325 | |
| 287 |
