- 3728 Views
- 2 replies
- 3 kudos
Resolved! Databricks Apps - Auto Terminate Option
Hi Everyone,I’m exploring Databricks Apps and have two questions:Is there a way to automatically terminate an app after a certain period of inactivity?Does Databricks provide any scheduling mechanism for apps, similar to how Jobs can be scheduled?Any...
- 3728 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @vamsi_simbus - One option you can explore to start and stop apps using Databricks API. Have a look on the below document link - https://docs.databricks.com/api/workspace/apps/stop
- 3 kudos
- 1610 Views
- 3 replies
- 1 kudos
Resolved! DAB Job - Serverless Cluster using configured base environment
I have configured a base serverless environment for my workspace that includes libraries from a private repositoryThis base environment has been set to default, and behaves as expected when running notebooks manually in the workspace with Serverless ...
- 1610 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @slangenborg According to the official Databricks Jobs REST API documentation, notebook tasks use the notebook environment only implicitly when no environment_key is provided. The API lets you explicitly configure environments only via an environ...
- 1 kudos
- 1097 Views
- 1 replies
- 0 kudos
Migrate on-premise delta tables to Databricks (Azure)
Hi There,I have the situation that we've decided to migrate our on-premise delta-lake to Azure Databricks.Because of networking I can only "push" the data from on-prem to cloud.What would be the best way to replicate all tables: schema+partitioning i...
- 1097 Views
- 1 replies
- 0 kudos
- 0 kudos
The correct solution is not SQL based.Delta tables are defined by the contents of the delta log directory, not by CREATE TABLE statements. That is why SHOW CREATE TABLE cannot reconstruct partitions, properties or constraints.The only reliable migrat...
- 0 kudos
- 2790 Views
- 4 replies
- 1 kudos
Resolved! Issues Creating Genie Space via API Join Specs Are Not Persisted
Hi,I’m experimenting with the new API to create a Genie Space.I’m able to successfully create the space, but the join definitions are not created, even though I’m passing a join_specs object in the same format returned by GET /spaces/{id} for an exis...
- 2790 Views
- 4 replies
- 1 kudos
- 1 kudos
The serialized space JSON is incorrect. It has `join_specs` and `sql_snippets` nested under `data_sources`, but they should be nested under `instructions` instead. There they apply as expected.
- 1 kudos
- 1261 Views
- 1 replies
- 1 kudos
Resolved! Import functions in databricks asset bundles using source: WORKSPACE
Hi,We are using Databricks asset bundles, and we create functions which we import in notebooks, for instance:from utils import helperswhere utils is just a folder in our root. When running this with source: WORKSPACE, it will fail to resolve the impo...
- 1261 Views
- 1 replies
- 1 kudos
- 1 kudos
In Git folders, the repo root is auto-added to the Python path, so imports like from utils import helpers work, while in workspace folders, only the notebook’s directory is on the path, which is why it breaks. The quick fix is a tiny bootstrap that a...
- 1 kudos
- 1788 Views
- 3 replies
- 3 kudos
Resolved! Serverless Compute Access Restriction Not Supported at User Level
The requirement is to disable serverless compute access for specific users while allowing them to use only their assigned clusters, without restricting serverless compute at the workspace level. After reviewing the available configuration options, th...
- 1788 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @ramsai ,how about a workaround ?setting budget policies at account level.Attribute usage with serverless budget policies | Databricks on AWS
- 3 kudos
- 1616 Views
- 2 replies
- 2 kudos
Resolved! Liquid Clustering and S3 Performance
Are there any performance concerns when using liquid clustering and AWS S3. I believe all the parquet files go in the same folder (Prefix in AWS S3 Terms) verses folders per partition when using "partition by". And there is this note on S3 performa...
- 1616 Views
- 2 replies
- 2 kudos
- 2 kudos
Even though liquid clustering removes Hive-style partition folders, it typically doesn’t cause S3 prefix performance issues on Databricks. Delta tables don’t rely on directory listing for reads; they use the transaction log to locate exact files. In ...
- 2 kudos
- 15811 Views
- 6 replies
- 0 kudos
Resolved! Pass catalog/schema/table name as a parameter to sql task
Hi, i am trying to pass catalog name as a parameter into query for sql task, and it pastes it with single quotes, which results in error. Is there a way to pass raw value or other possible workarounds? query:INSERT INTO {{ catalog }}.pas.product_snap...
- 15811 Views
- 6 replies
- 0 kudos
- 0 kudos
This works USE CATALOG IDENTIFIER({{ catalog_name }});USE SCHEMA IDENTIFIER({{ schema_name }});
- 0 kudos
- 4538 Views
- 12 replies
- 12 kudos
Resolved! Creating Serverless Cluster
Hi everyone,I am trying to create a cluster in Databricks Free Edition, but I keep getting the following error:"Cannot create serverless cluster, please try again later."I have attempted this on different days and at different times, but the issue pe...
- 4538 Views
- 12 replies
- 12 kudos
- 12 kudos
Thank you all ( @Sanjeeb2024 , @Sanjeeb2024, @JAHNAVI , @Manoj12421 ), it works!It was not a DataBricks Free Edition as @Masood_Joukar said.
- 12 kudos
- 1539 Views
- 4 replies
- 2 kudos
Migrating from directory-listing to Autoloader Managed File events
Hi everyone,We are currently migrating from a directory listing-based streaming approach to managed file events in Databricks Auto Loader for processing our data in structured streaming.We have a function that handles structured streaming where we ar...
- 1539 Views
- 4 replies
- 2 kudos
- 2 kudos
Yes, for your setup, Databricks Auto Loader will create a separate event queue for each independent stream running with the cloudFiles.useManagedFileEvents = true option.As you are running - 1 stream per table, 1 unique directory per stream and 1 uni...
- 2 kudos
- 1247 Views
- 3 replies
- 2 kudos
Alerts V2 Parameters
Hi, I'm working on using Databricks python SDK to create an alert using a notebook, but it seems with V1 there is no way to add subscribers and with V2 there is no option for adding parameters. Is my understanding correct or am I missing something? A...
- 1247 Views
- 3 replies
- 2 kudos
- 2 kudos
Alerts V2 (Public Preview) do not support query parameters yet. This is a documented limitation. Legacy alerts (V1) do support parameters and will use the default values defined in the SQL editor. For notifications, both legacy alerts and Alerts V2 a...
- 2 kudos
- 4804 Views
- 4 replies
- 5 kudos
Resolved! Strange DELTA_MULTIPLE_SOURCE_ROW_MATCHING_TARGET_ROW_IN_MERGE error
I use databricks 17.3 runtime.I try to run following code.CREATE OR REPLACE TABLE default.target_table (key1 INT,key2 INT,key3 INT,val STRING) USING DELTA;INSERT INTO target_table(key1, key2, key3, val) VALUES(1, 1, 1, 'a');CREATE OR REPLACE TABLE de...
- 4804 Views
- 4 replies
- 5 kudos
- 5 kudos
Hi, you need to put all of the keys in the ON part of the clause rather then in the where condition. This code works: MERGE INTO target_table AS target USING source_table AS source ON target.key1 = source.key1 AND target.key2 = source.key2 AND target...
- 5 kudos
- 2289 Views
- 5 replies
- 0 kudos
Installing Custom Packages on Serverless Compute via Databricks Connect
I have a custom Python package that provides a PySpark DataSource implementation. I'm using Databricks Connect (16.4.10) and need to understand package installation options for serverless compute.Works: Traditional Compute ClusterCustom package pre-i...
- 2289 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @ganesh_raskar - If you can provide which custom package and exact code and error, I can try to replicate at my end and explore the suitable option.
- 0 kudos
- 25531 Views
- 9 replies
- 17 kudos
Resolved! MetadataChangedException
A delta lake table is created with identity column and I'm not able to load the data parallelly from four process. i'm getting the metadata exception error.I don't want to load the data in temp table . Need to load directly and parallelly in to delta...
- 25531 Views
- 9 replies
- 17 kudos
- 17 kudos
I'm also having the same problem. I'm using autoloader to load many files into a delta table with an identity column. What used to work now dies with this problem -- after running for a long time!!
- 17 kudos
- 2854 Views
- 6 replies
- 0 kudos
workspace notebook path not recognized by dbutils.notebook.run() when running from a workflow/job
result = dbutils.notebooks.run("/Workspace/YourFolder/NotebookA", timeout_seconds=600, arguments={"param1": "value1"}) print(result)I was able to execute the above code manually from a notebook.But when i run the same notebook as a job, it fails stat...
- 2854 Views
- 6 replies
- 0 kudos
- 0 kudos
@siva_pusarla: We use the following pattern and it works,1) Calling notebook - constant location used by Job. + src/framework + notebook_executor.py2) Callee notebooks - dynamic + src/app/notebooks ...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
