- 1989 Views
- 3 replies
- 3 kudos
Resolved! Serverless Compute
Is there a way to create multiple serverless compute instances in a Databricks workspace?I have a use case where multiple all-purpose clusters are allocated to different user groups. I want to replace these all-purpose clusters with serverless comput...
- 1989 Views
- 3 replies
- 3 kudos
- 3 kudos
No — Databricks does not currently offer a native way to set hard quotas or throttles on serverless compute per user group (for example, “Team A is limited to X DBUs/hour on serverless notebooks”). Serverless compute is intentionally a shared, autosc...
- 3 kudos
- 1275 Views
- 1 replies
- 2 kudos
Resolved! Why do I need MANAGE privilege on a connection to run a pipeline?
I have a notebook that I am able to run in itself, but when I run it within a Databricks job, it fails with the following error:[UNAUTHORIZED_ACCESS] PERMISSION_DENIED: User does not have MANAGE on Connection 'dev_postgres'. SQLSTATE: 42501The notebo...
- 1275 Views
- 1 replies
- 2 kudos
- 2 kudos
When you ran the job, was your data_security/access mode set to Dedicated/Single-User or Standard/Shared. I assume it was the former. If it's the latter, you wouldn't have need to give that permission. You would get the same experience trying to do a...
- 2 kudos
- 3726 Views
- 7 replies
- 4 kudos
Resolved! How can I manage the code on using a Spot Instance?
Hello community,In the near future, I need to use spot instances to reduce the cost of running a batch processing job.My question is: how can I manage my code to properly handle and capture a SIGTERM signal?Is there any documentation or guidance you ...
- 3726 Views
- 7 replies
- 4 kudos
- 4 kudos
@Kirankumarbs — in the case of a single-node cluster, well… you get what you get . A single-node cluster can only run on an on-demand instance. That one node hosts both the driver and the executor, so if the node goes away, there’s no recovery path....
- 4 kudos
- 3554 Views
- 4 replies
- 3 kudos
When accessing a view in Unity Catalog; access to underlying tables of the view is also needed.
My goal is that i want to provide users access to a view but not the underlying tables. I only want them to see specific columns and rows of the table. When i just give the select permissions on the view the user gets an error that they also need acc...
- 3554 Views
- 4 replies
- 3 kudos
- 3 kudos
you may need to check if the compute you are using met what mentioned in https://docs.databricks.com/aws/en/views/#requirements-for-querying-views
- 3 kudos
- 4109 Views
- 2 replies
- 1 kudos
Can not open socket to local (127.0.0.1)
Hi, I'm running a databricks pipeline hourly using python notebooks checked out from git with on-demand compute (using r6gd.xlarge 32GB + 4 CPU's Gravaton). Most of the times the pipeline runs without problems. However, sometimes the first notebook f...
- 4109 Views
- 2 replies
- 1 kudos
- 1 kudos
HI I am facing similar error, the cluster runs 24/7 and this issue is observed for a few runs in the day. The data volume being processed is not huge but the logic that this needs to go though is complex. I do not want to include the display in produ...
- 1 kudos
- 2214 Views
- 1 replies
- 2 kudos
Resolved! JDBC with serverless compute
Hi community,We have a scenario where we need to ingest data into Lakebase. Currently, we are trying to use JDBC to write data in a notebook with serverless compute. However, the documentation on serverless limitations (link) mentions that JAR librar...
- 2214 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello @Dedescoat , I did some poking around in our documentation and would like to offer some tips/tricks to help you further diagnose the issue. Yes — using a Unity Catalog JDBC connection to load a driver from a UC volume and write from serverless...
- 2 kudos
- 2331 Views
- 3 replies
- 1 kudos
Resolved! How to access artifacts from job run?
In a workflow, is there a way to access task artifacts from within the run?I have a job with a task TasksA, which is a dbt task that creates some artifacts. I want to store these artifacts, but the job artifacts seems to be saved in a location I cann...
- 2331 Views
- 3 replies
- 1 kudos
- 942 Views
- 3 replies
- 1 kudos
Resolved! Connect to spark session and uc tables in python file
How to Connect to spark session and uc tables in python file. I want to read uc tables in python modules in databricks workspace. How to access the current sparksession
- 942 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @ajay_wavicle ,Azure Databricks automatically creates a SparkContext for each compute cluster, and creates an isolated SparkSession for each notebook or job executed against the cluster. So following should work in python module in Databricks Work...
- 1 kudos
- 47653 Views
- 15 replies
- 6 kudos
Error writing parquet files
Hi, we are having this chain of errors every day in different files and processes:An error occurred while calling o11255.parquet.: org.apache.spark.SparkException: Job aborted.Caused by: org.apache.spark.SparkException: Job aborted due to stage failu...
- 47653 Views
- 15 replies
- 6 kudos
- 5295 Views
- 1 replies
- 2 kudos
Resolved! Database schema migration tools: Flyway vs Liquibase
I'm working on comparison between these tools for database schema migration on Databricks Delta tables.Any experience with the tools? Pros/cons?I've been through most of the blogs on how to implement them. I seek for comparison or real production exp...
- 5295 Views
- 1 replies
- 2 kudos
- 2 kudos
Pick Flyway if you prefer a simple, lightweight solution with sequential migrations and minimal overhead. It’s well suited for teams managing straightforward schema changes—mainly add, drop, or alter operations—through small, explicit, versioned scri...
- 2 kudos
- 1630 Views
- 1 replies
- 1 kudos
Resolved! External to Managed
We are looking to migrate to managed tables using ALTER TABLE fq_table_name SET MANAGED During migration process we need to have ability to switch between external & managed tables & vice versa.UNSET MANAGED works for 14 days. But I'm unable to just ...
- 1630 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello, Thanks for sharing your investigation. You’re correct: you can’t immediately recreate the converted table as an external table on the same original path after dropping it. The Unity Catalog table object remains in a soft‑deleted state for 7 da...
- 1 kudos
- 630 Views
- 2 replies
- 2 kudos
- 630 Views
- 2 replies
- 2 kudos
- 2 kudos
@mj — can you share a bit more context on what you’re ultimately trying to accomplish? As @balajij8 pointed out, Unity Catalog is the right foundation for sharing data, but with a little more detail I can be much more prescriptive about the best ap...
- 2 kudos
- 1446 Views
- 3 replies
- 6 kudos
Resolved! Lakeflow Spark Declarative Pipelines do not support read_files with format excel yet
Error: Failed to find the data source: excel. Make sure the provider name is correct and the package is properly registered and compatible with your Spark version.Upon checking the cluster details, it is running with 16.4 yet. When will it be updated...
- 1446 Views
- 3 replies
- 6 kudos
- 6496 Views
- 4 replies
- 1 kudos
Resolved! Cluster terminated. Reason: Npip Tunnel Setup Failure
Hi, I have recently deployed a new Workspace in AWS and getting the following error when trying to start the cluster:"NPIP tunnel setup failure during launch. Please try again later and contact Databricks if the problem persists. Instance bootstrap f...
- 6496 Views
- 4 replies
- 1 kudos
- 1 kudos
Finally, this error was fixed by changing the DHCP configuration of the VPC.
- 1 kudos
- 1356 Views
- 2 replies
- 0 kudos
Lakeflow pipeline (formerly DLT pipeline) performance progressively degrades on a persistent cluster
I have a small (under 20 tables, all streaming) DLT pipeline running in triggered mode, scheduled every 15min during the workday. For development I've set `pipelines.clusterShutdown.delay` to avoid having to start a cluster every update.I've noticed...
- 1356 Views
- 2 replies
- 0 kudos
- 0 kudos
I'm facing this exact issue, only with a standard job instead of a DLT pipeline. I can't use serverless or restart the cluster periodically due to things out of my control. Any specific advice on diagnosis and resolving? I don't think it can be check...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
