- 5988 Views
- 9 replies
- 0 kudos
Spark aws s3 folder partition pruning doesn't work
Hi, I have a use case where my spark job running on EMR AWS, and it is reading from a s3 path: some-bucket/some-path/region=na/days=1during my read, I pass DataFrame df = sparkSession.read().option("mergeSchema", true).parquet("some-bucket/some-path...
- 5988 Views
- 9 replies
- 0 kudos
- 2217 Views
- 4 replies
- 0 kudos
Resolved! Bundled Wheel Task with Serverless Compute
I am trying to run a wheel task as part of a bundle on serverless compute. My databricks.yml includes an artifact being constructed:artifacts: nexusbricks: type: whl build: python -m build path: .I then am trying to set up a job to cons...
- 2217 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @mlrichmond-mill This is how we do it. environments: - environment_key: serverless_env_v4 spec: environment_version: '4' dependencies: dist/*.whl becasue your bundle whl gets placed into dist/*.whl...
- 0 kudos
- 5137 Views
- 8 replies
- 2 kudos
Pushing data from databricks (cloud) to Oracle (on-prem) instance?
Pushing data from databricks (cloud) to Oracle (on-prem) instance?===================================================Thanks for reviewing my threads. I find some threads on this subject dated in 2022 by @ Ajay-PandeyDatabricks to Oracle We find many...
- 5137 Views
- 8 replies
- 2 kudos
- 2 kudos
Thanks for weighing in and proving an interesting insight. Here are some questions coming to my mind upon reviewing this thread.I like to see a way to write to on-prem linux folder directly from DBX without going thru S3. As we see many features adde...
- 2 kudos
- 1627 Views
- 3 replies
- 3 kudos
Resolved! Serverless Compute
Is there a way to create multiple serverless compute instances in a Databricks workspace?I have a use case where multiple all-purpose clusters are allocated to different user groups. I want to replace these all-purpose clusters with serverless comput...
- 1627 Views
- 3 replies
- 3 kudos
- 3 kudos
No — Databricks does not currently offer a native way to set hard quotas or throttles on serverless compute per user group (for example, “Team A is limited to X DBUs/hour on serverless notebooks”). Serverless compute is intentionally a shared, autosc...
- 3 kudos
- 1009 Views
- 1 replies
- 2 kudos
Resolved! Why do I need MANAGE privilege on a connection to run a pipeline?
I have a notebook that I am able to run in itself, but when I run it within a Databricks job, it fails with the following error:[UNAUTHORIZED_ACCESS] PERMISSION_DENIED: User does not have MANAGE on Connection 'dev_postgres'. SQLSTATE: 42501The notebo...
- 1009 Views
- 1 replies
- 2 kudos
- 2 kudos
When you ran the job, was your data_security/access mode set to Dedicated/Single-User or Standard/Shared. I assume it was the former. If it's the latter, you wouldn't have need to give that permission. You would get the same experience trying to do a...
- 2 kudos
- 2927 Views
- 7 replies
- 4 kudos
Resolved! How can I manage the code on using a Spot Instance?
Hello community,In the near future, I need to use spot instances to reduce the cost of running a batch processing job.My question is: how can I manage my code to properly handle and capture a SIGTERM signal?Is there any documentation or guidance you ...
- 2927 Views
- 7 replies
- 4 kudos
- 4 kudos
@Kirankumarbs — in the case of a single-node cluster, well… you get what you get . A single-node cluster can only run on an on-demand instance. That one node hosts both the driver and the executor, so if the node goes away, there’s no recovery path....
- 4 kudos
- 3077 Views
- 4 replies
- 3 kudos
When accessing a view in Unity Catalog; access to underlying tables of the view is also needed.
My goal is that i want to provide users access to a view but not the underlying tables. I only want them to see specific columns and rows of the table. When i just give the select permissions on the view the user gets an error that they also need acc...
- 3077 Views
- 4 replies
- 3 kudos
- 3 kudos
you may need to check if the compute you are using met what mentioned in https://docs.databricks.com/aws/en/views/#requirements-for-querying-views
- 3 kudos
- 3949 Views
- 2 replies
- 1 kudos
Can not open socket to local (127.0.0.1)
Hi, I'm running a databricks pipeline hourly using python notebooks checked out from git with on-demand compute (using r6gd.xlarge 32GB + 4 CPU's Gravaton). Most of the times the pipeline runs without problems. However, sometimes the first notebook f...
- 3949 Views
- 2 replies
- 1 kudos
- 1 kudos
HI I am facing similar error, the cluster runs 24/7 and this issue is observed for a few runs in the day. The data volume being processed is not huge but the logic that this needs to go though is complex. I do not want to include the display in produ...
- 1 kudos
- 1609 Views
- 1 replies
- 2 kudos
Resolved! JDBC with serverless compute
Hi community,We have a scenario where we need to ingest data into Lakebase. Currently, we are trying to use JDBC to write data in a notebook with serverless compute. However, the documentation on serverless limitations (link) mentions that JAR librar...
- 1609 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello @Dedescoat , I did some poking around in our documentation and would like to offer some tips/tricks to help you further diagnose the issue. Yes — using a Unity Catalog JDBC connection to load a driver from a UC volume and write from serverless...
- 2 kudos
- 2113 Views
- 3 replies
- 1 kudos
Resolved! How to access artifacts from job run?
In a workflow, is there a way to access task artifacts from within the run?I have a job with a task TasksA, which is a dbt task that creates some artifacts. I want to store these artifacts, but the job artifacts seems to be saved in a location I cann...
- 2113 Views
- 3 replies
- 1 kudos
- 758 Views
- 3 replies
- 1 kudos
Resolved! Connect to spark session and uc tables in python file
How to Connect to spark session and uc tables in python file. I want to read uc tables in python modules in databricks workspace. How to access the current sparksession
- 758 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @ajay_wavicle ,Azure Databricks automatically creates a SparkContext for each compute cluster, and creates an isolated SparkSession for each notebook or job executed against the cluster. So following should work in python module in Databricks Work...
- 1 kudos
- 46541 Views
- 15 replies
- 6 kudos
Error writing parquet files
Hi, we are having this chain of errors every day in different files and processes:An error occurred while calling o11255.parquet.: org.apache.spark.SparkException: Job aborted.Caused by: org.apache.spark.SparkException: Job aborted due to stage failu...
- 46541 Views
- 15 replies
- 6 kudos
- 4676 Views
- 1 replies
- 2 kudos
Resolved! Database schema migration tools: Flyway vs Liquibase
I'm working on comparison between these tools for database schema migration on Databricks Delta tables.Any experience with the tools? Pros/cons?I've been through most of the blogs on how to implement them. I seek for comparison or real production exp...
- 4676 Views
- 1 replies
- 2 kudos
- 2 kudos
Pick Flyway if you prefer a simple, lightweight solution with sequential migrations and minimal overhead. It’s well suited for teams managing straightforward schema changes—mainly add, drop, or alter operations—through small, explicit, versioned scri...
- 2 kudos
- 1368 Views
- 1 replies
- 1 kudos
Resolved! External to Managed
We are looking to migrate to managed tables using ALTER TABLE fq_table_name SET MANAGED During migration process we need to have ability to switch between external & managed tables & vice versa.UNSET MANAGED works for 14 days. But I'm unable to just ...
- 1368 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello, Thanks for sharing your investigation. You’re correct: you can’t immediately recreate the converted table as an external table on the same original path after dropping it. The Unity Catalog table object remains in a soft‑deleted state for 7 da...
- 1 kudos
- 499 Views
- 2 replies
- 2 kudos
- 499 Views
- 2 replies
- 2 kudos
- 2 kudos
@mj — can you share a bit more context on what you’re ultimately trying to accomplish? As @balajij8 pointed out, Unity Catalog is the right foundation for sharing data, but with a little more detail I can be much more prescriptive about the best ap...
- 2 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 576 | |
| 349 | |
| 287 |
