- 981 Views
- 1 replies
- 1 kudos
Resolved! Import functions in databricks asset bundles using source: WORKSPACE
Hi,We are using Databricks asset bundles, and we create functions which we import in notebooks, for instance:from utils import helperswhere utils is just a folder in our root. When running this with source: WORKSPACE, it will fail to resolve the impo...
- 981 Views
- 1 replies
- 1 kudos
- 1 kudos
In Git folders, the repo root is auto-added to the Python path, so imports like from utils import helpers work, while in workspace folders, only the notebook’s directory is on the path, which is why it breaks. The quick fix is a tiny bootstrap that a...
- 1 kudos
- 1314 Views
- 3 replies
- 3 kudos
Resolved! Serverless Compute Access Restriction Not Supported at User Level
The requirement is to disable serverless compute access for specific users while allowing them to use only their assigned clusters, without restricting serverless compute at the workspace level. After reviewing the available configuration options, th...
- 1314 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @ramsai ,how about a workaround ?setting budget policies at account level.Attribute usage with serverless budget policies | Databricks on AWS
- 3 kudos
- 1227 Views
- 2 replies
- 2 kudos
Resolved! Liquid Clustering and S3 Performance
Are there any performance concerns when using liquid clustering and AWS S3. I believe all the parquet files go in the same folder (Prefix in AWS S3 Terms) verses folders per partition when using "partition by". And there is this note on S3 performa...
- 1227 Views
- 2 replies
- 2 kudos
- 2 kudos
Even though liquid clustering removes Hive-style partition folders, it typically doesn’t cause S3 prefix performance issues on Databricks. Delta tables don’t rely on directory listing for reads; they use the transaction log to locate exact files. In ...
- 2 kudos
- 10480 Views
- 6 replies
- 0 kudos
Resolved! Pass catalog/schema/table name as a parameter to sql task
Hi, i am trying to pass catalog name as a parameter into query for sql task, and it pastes it with single quotes, which results in error. Is there a way to pass raw value or other possible workarounds? query:INSERT INTO {{ catalog }}.pas.product_snap...
- 10480 Views
- 6 replies
- 0 kudos
- 0 kudos
This works USE CATALOG IDENTIFIER({{ catalog_name }});USE SCHEMA IDENTIFIER({{ schema_name }});
- 0 kudos
- 2703 Views
- 12 replies
- 12 kudos
Resolved! Creating Serverless Cluster
Hi everyone,I am trying to create a cluster in Databricks Free Edition, but I keep getting the following error:"Cannot create serverless cluster, please try again later."I have attempted this on different days and at different times, but the issue pe...
- 2703 Views
- 12 replies
- 12 kudos
- 12 kudos
Thank you all ( @Sanjeeb2024 , @Sanjeeb2024, @JAHNAVI , @Manoj12421 ), it works!It was not a DataBricks Free Edition as @Masood_Joukar said.
- 12 kudos
- 1121 Views
- 4 replies
- 2 kudos
Migrating from directory-listing to Autoloader Managed File events
Hi everyone,We are currently migrating from a directory listing-based streaming approach to managed file events in Databricks Auto Loader for processing our data in structured streaming.We have a function that handles structured streaming where we ar...
- 1121 Views
- 4 replies
- 2 kudos
- 2 kudos
Yes, for your setup, Databricks Auto Loader will create a separate event queue for each independent stream running with the cloudFiles.useManagedFileEvents = true option.As you are running - 1 stream per table, 1 unique directory per stream and 1 uni...
- 2 kudos
- 910 Views
- 3 replies
- 2 kudos
Alerts V2 Parameters
Hi, I'm working on using Databricks python SDK to create an alert using a notebook, but it seems with V1 there is no way to add subscribers and with V2 there is no option for adding parameters. Is my understanding correct or am I missing something? A...
- 910 Views
- 3 replies
- 2 kudos
- 2 kudos
Alerts V2 (Public Preview) do not support query parameters yet. This is a documented limitation. Legacy alerts (V1) do support parameters and will use the default values defined in the SQL editor. For notifications, both legacy alerts and Alerts V2 a...
- 2 kudos
- 3955 Views
- 4 replies
- 5 kudos
Resolved! Strange DELTA_MULTIPLE_SOURCE_ROW_MATCHING_TARGET_ROW_IN_MERGE error
I use databricks 17.3 runtime.I try to run following code.CREATE OR REPLACE TABLE default.target_table (key1 INT,key2 INT,key3 INT,val STRING) USING DELTA;INSERT INTO target_table(key1, key2, key3, val) VALUES(1, 1, 1, 'a');CREATE OR REPLACE TABLE de...
- 3955 Views
- 4 replies
- 5 kudos
- 5 kudos
Hi, you need to put all of the keys in the ON part of the clause rather then in the where condition. This code works: MERGE INTO target_table AS target USING source_table AS source ON target.key1 = source.key1 AND target.key2 = source.key2 AND target...
- 5 kudos
- 1610 Views
- 5 replies
- 0 kudos
Installing Custom Packages on Serverless Compute via Databricks Connect
I have a custom Python package that provides a PySpark DataSource implementation. I'm using Databricks Connect (16.4.10) and need to understand package installation options for serverless compute.Works: Traditional Compute ClusterCustom package pre-i...
- 1610 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @ganesh_raskar - If you can provide which custom package and exact code and error, I can try to replicate at my end and explore the suitable option.
- 0 kudos
- 24361 Views
- 9 replies
- 17 kudos
Resolved! MetadataChangedException
A delta lake table is created with identity column and I'm not able to load the data parallelly from four process. i'm getting the metadata exception error.I don't want to load the data in temp table . Need to load directly and parallelly in to delta...
- 24361 Views
- 9 replies
- 17 kudos
- 17 kudos
I'm also having the same problem. I'm using autoloader to load many files into a delta table with an identity column. What used to work now dies with this problem -- after running for a long time!!
- 17 kudos
- 2111 Views
- 6 replies
- 0 kudos
workspace notebook path not recognized by dbutils.notebook.run() when running from a workflow/job
result = dbutils.notebooks.run("/Workspace/YourFolder/NotebookA", timeout_seconds=600, arguments={"param1": "value1"}) print(result)I was able to execute the above code manually from a notebook.But when i run the same notebook as a job, it fails stat...
- 2111 Views
- 6 replies
- 0 kudos
- 0 kudos
@siva_pusarla: We use the following pattern and it works,1) Calling notebook - constant location used by Job. + src/framework + notebook_executor.py2) Callee notebooks - dynamic + src/app/notebooks ...
- 0 kudos
- 4529 Views
- 4 replies
- 1 kudos
pyspark.sql.utils.AnalysisException: Non-time-based windows are not supported on streaming DataFrames/Datasets
pyspark.sql.utils.AnalysisException: Non-time-based windows are not supported on streaming DataFrames/DatasetsGetting this error while writing can any one please tell how we can resolve it
- 4529 Views
- 4 replies
- 1 kudos
- 1936 Views
- 4 replies
- 2 kudos
Azure Databricks UI consuming way too much memory & laggy
This especially happens when the notebook is large with many cells. Even if I clear all the outputs scrolling the notebook is way too laggy. When I start running the code the memory consumption is 3-4GB minimum even if I am not displaying any data/ta...
- 1936 Views
- 4 replies
- 2 kudos
- 2 kudos
@dj4: Are you in a corporate proxy environment?Databricks Browser UI uses Web Sockets and sometimes the performance issues happen due to the security checks in the traffic.
- 2 kudos
- 5022 Views
- 12 replies
- 1 kudos
restarting the cluster always running doesn't free the memory?
Hello community,I was working on optimising the driver memory, since there are code that are not optimised for spark, and I was planning temporary to restart the cluster to free up the memory.that could be a potential solution, since if the cluster i...
- 5022 Views
- 12 replies
- 1 kudos
- 1 kudos
@jeremy98 : Please review the cluster's event logs to understand the trend of the GC related issues. Example in below snapshot.Typically, productive jobs are executed using Job clusters; and they stop as soon as the work is completed. Could you pleas...
- 1 kudos
- 656 Views
- 1 replies
- 1 kudos
How to not make tab headers show when embedding dashboard
When embedding the AI BI dashboard, is there a way to not make the tabs show and instead use our own UI tab to navigate the tabs?Currently, there are two tab headers - one in the databricks dashboard and then another tab section in our embedding webp...
- 656 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @amekojc At the moment, Databricks AI BI Dashboards do not support hiding or disabling the native dashboard tabs when embedding. The embedded dashboard always renders with its own tab headers, and there is no configuration or API to control tab vi...
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 576 | |
| 349 | |
| 287 |
