- 3689 Views
- 4 replies
- 3 kudos
When accessing a view in Unity Catalog; access to underlying tables of the view is also needed.
My goal is that i want to provide users access to a view but not the underlying tables. I only want them to see specific columns and rows of the table. When i just give the select permissions on the view the user gets an error that they also need acc...
- 3689 Views
- 4 replies
- 3 kudos
- 3 kudos
you may need to check if the compute you are using met what mentioned in https://docs.databricks.com/aws/en/views/#requirements-for-querying-views
- 3 kudos
- 4180 Views
- 2 replies
- 1 kudos
Can not open socket to local (127.0.0.1)
Hi, I'm running a databricks pipeline hourly using python notebooks checked out from git with on-demand compute (using r6gd.xlarge 32GB + 4 CPU's Gravaton). Most of the times the pipeline runs without problems. However, sometimes the first notebook f...
- 4180 Views
- 2 replies
- 1 kudos
- 1 kudos
HI I am facing similar error, the cluster runs 24/7 and this issue is observed for a few runs in the day. The data volume being processed is not huge but the logic that this needs to go though is complex. I do not want to include the display in produ...
- 1 kudos
- 2460 Views
- 1 replies
- 2 kudos
Resolved! JDBC with serverless compute
Hi community,We have a scenario where we need to ingest data into Lakebase. Currently, we are trying to use JDBC to write data in a notebook with serverless compute. However, the documentation on serverless limitations (link) mentions that JAR librar...
- 2460 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello @Dedescoat , I did some poking around in our documentation and would like to offer some tips/tricks to help you further diagnose the issue. Yes — using a Unity Catalog JDBC connection to load a driver from a UC volume and write from serverless...
- 2 kudos
- 2452 Views
- 3 replies
- 1 kudos
Resolved! How to access artifacts from job run?
In a workflow, is there a way to access task artifacts from within the run?I have a job with a task TasksA, which is a dbt task that creates some artifacts. I want to store these artifacts, but the job artifacts seems to be saved in a location I cann...
- 2452 Views
- 3 replies
- 1 kudos
- 1088 Views
- 3 replies
- 1 kudos
Resolved! Connect to spark session and uc tables in python file
How to Connect to spark session and uc tables in python file. I want to read uc tables in python modules in databricks workspace. How to access the current sparksession
- 1088 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @ajay_wavicle ,Azure Databricks automatically creates a SparkContext for each compute cluster, and creates an isolated SparkSession for each notebook or job executed against the cluster. So following should work in python module in Databricks Work...
- 1 kudos
- 48232 Views
- 15 replies
- 6 kudos
Error writing parquet files
Hi, we are having this chain of errors every day in different files and processes:An error occurred while calling o11255.parquet.: org.apache.spark.SparkException: Job aborted.Caused by: org.apache.spark.SparkException: Job aborted due to stage failu...
- 48232 Views
- 15 replies
- 6 kudos
- 5538 Views
- 1 replies
- 2 kudos
Resolved! Database schema migration tools: Flyway vs Liquibase
I'm working on comparison between these tools for database schema migration on Databricks Delta tables.Any experience with the tools? Pros/cons?I've been through most of the blogs on how to implement them. I seek for comparison or real production exp...
- 5538 Views
- 1 replies
- 2 kudos
- 2 kudos
Pick Flyway if you prefer a simple, lightweight solution with sequential migrations and minimal overhead. It’s well suited for teams managing straightforward schema changes—mainly add, drop, or alter operations—through small, explicit, versioned scri...
- 2 kudos
- 1720 Views
- 1 replies
- 1 kudos
Resolved! External to Managed
We are looking to migrate to managed tables using ALTER TABLE fq_table_name SET MANAGED During migration process we need to have ability to switch between external & managed tables & vice versa.UNSET MANAGED works for 14 days. But I'm unable to just ...
- 1720 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello, Thanks for sharing your investigation. You’re correct: you can’t immediately recreate the converted table as an external table on the same original path after dropping it. The Unity Catalog table object remains in a soft‑deleted state for 7 da...
- 1 kudos
- 694 Views
- 2 replies
- 2 kudos
- 694 Views
- 2 replies
- 2 kudos
- 2 kudos
@mj — can you share a bit more context on what you’re ultimately trying to accomplish? As @balajij8 pointed out, Unity Catalog is the right foundation for sharing data, but with a little more detail I can be much more prescriptive about the best ap...
- 2 kudos
- 1551 Views
- 3 replies
- 6 kudos
Resolved! Lakeflow Spark Declarative Pipelines do not support read_files with format excel yet
Error: Failed to find the data source: excel. Make sure the provider name is correct and the package is properly registered and compatible with your Spark version.Upon checking the cluster details, it is running with 16.4 yet. When will it be updated...
- 1551 Views
- 3 replies
- 6 kudos
- 6603 Views
- 4 replies
- 1 kudos
Resolved! Cluster terminated. Reason: Npip Tunnel Setup Failure
Hi, I have recently deployed a new Workspace in AWS and getting the following error when trying to start the cluster:"NPIP tunnel setup failure during launch. Please try again later and contact Databricks if the problem persists. Instance bootstrap f...
- 6603 Views
- 4 replies
- 1 kudos
- 1 kudos
Finally, this error was fixed by changing the DHCP configuration of the VPC.
- 1 kudos
- 1433 Views
- 2 replies
- 0 kudos
Lakeflow pipeline (formerly DLT pipeline) performance progressively degrades on a persistent cluster
I have a small (under 20 tables, all streaming) DLT pipeline running in triggered mode, scheduled every 15min during the workday. For development I've set `pipelines.clusterShutdown.delay` to avoid having to start a cluster every update.I've noticed...
- 1433 Views
- 2 replies
- 0 kudos
- 0 kudos
I'm facing this exact issue, only with a standard job instead of a DLT pipeline. I can't use serverless or restart the cluster periodically due to things out of my control. Any specific advice on diagnosis and resolving? I don't think it can be check...
- 0 kudos
- 803 Views
- 1 replies
- 1 kudos
Resolved! ingest unstructure data like sharepoint with databricks
Hello everyone, Actually, i am using lakeflow connector to load sharepoint into delta table but the connector is not mature. Someone knows how can i use rest Api sharepoint to load into delta table? Cordially, Seefoods
- 803 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @seefoods curious to know what issues you are facing with the connector assuming you are using the standard SharePoint connector with a Unity Catalog connection and then use spark.read / Auto Loader / COPY INTO against SharePoint URLs.
- 1 kudos
- 6009 Views
- 10 replies
- 0 kudos
Triggering DLT Pipelines with Dynamic Parameters
Hi Team,We have a scenario where we need to pass a dynamic parameter to a Spark job that will trigger a DLT pipeline in append mode. Can you please suggest an approach for this?Regards,Phani
- 6009 Views
- 10 replies
- 0 kudos
- 0 kudos
If you’re looking to build a dynamic, configuration-driven DLT pipeline, a better approach is to use a configuration table. This table should include fields such as table_name, pipeline_name, table_properties, and other relevant settings. Your notebo...
- 0 kudos
- 918 Views
- 1 replies
- 1 kudos
Resolved! Bigquery using foreign catalog change behavior from runtime 15.4 to 16.4
Hello,I wanted to update my all-purpose cluster from 15.4 to higher (like 16.4) but I noticed a change in the behavior of my query on a BigQuery catalog.Using 15.4 my query runs only 1 job Using 16.4, same query on same resources produce 35 jobs (and...
- 918 Views
- 1 replies
- 1 kudos
- 1 kudos
In DBR 16.1+ Databricks switched the BigQuery federation connector from JDBC to the BigQuery Storage API,which parallelizes reads (more jobs) and transfers data directly from BigQuery to Databricks compute—so cross‑cloud queries (Azure → GCP) can inc...
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
