- 630 Views
- 1 replies
- 3 kudos
Resolved! Way to send SQL query results from a Databricks notebook via email
Hi everyone,I’m working with a Databricks notebook where I run a SQL query using spark.sql. The query returns a small result set (mainly counts or summary values).Once the notebook finishes running, I need to automatically send the query result from ...
- 630 Views
- 1 replies
- 3 kudos
- 3 kudos
Hi @faizan_111111,I would suggest setting up an email destination around a SQL Alert which should give you the ability to see returned results in the QUERY_RESULT_TABLEvariable. See: https://docs.databricks.com/aws/en/sql/user/alerts/If this doesn't ...
- 3 kudos
- 772 Views
- 3 replies
- 2 kudos
Resolved! Serverless budget policies auto-application and resource limiting in Databricks
Hi Team,I am exploring serverless compute in Databricks and had a few questions related to governance and cost control.We have multiple user groups in a workspace and are planning to move from all-purpose clusters to serverless compute. We understand...
- 772 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi there, That's great to hear that you're looking to use Serverless. It's way less overhead and a better use experience than classic compute. To answer your questions: Going forward, users should be required to select a budget policy on all new note...
- 2 kudos
- 522 Views
- 1 replies
- 2 kudos
Resolved! Databricks to powerbi connection issues
Creating a connection with powerbi and databricks. Did all necessary things to assigning permissions all the stuff but no issue and odbc drivers are also in place, cannot use other authentications methods and desktop due to restrictions. Need urgent ...
- 522 Views
- 1 replies
- 2 kudos
- 2 kudos
Could you be more specific as to what permissions you assigned and where? Are you using a Service Principal? There's two main options for the Power BI Service. The recommended being Machine to Machine OAuth. If you could be a bit more specific about ...
- 2 kudos
- 562 Views
- 2 replies
- 3 kudos
Resolved! Intermittent PERMISSION_DENIED Error in Genie
Hello everyone,We're experiencing an intermittent issue with our Genie implementation that started this afternoon after months of stable operation.Error message:"We're sorry, but an unexpected internal error has occurred. Please try again in a few mi...
- 562 Views
- 2 replies
- 3 kudos
- 3 kudos
Genie falls under Partner-powered AI features, so there was a change recently to align model hosting with Azure Geographies: Partner-powered AI features are now supported in the Canada, Brazil, and United Kingdom Azure Geographies
- 3 kudos
- 4094 Views
- 9 replies
- 0 kudos
Spark aws s3 folder partition pruning doesn't work
Hi, I have a use case where my spark job running on EMR AWS, and it is reading from a s3 path: some-bucket/some-path/region=na/days=1during my read, I pass DataFrame df = sparkSession.read().option("mergeSchema", true).parquet("some-bucket/some-path...
- 4094 Views
- 9 replies
- 0 kudos
- 729 Views
- 4 replies
- 0 kudos
Resolved! Bundled Wheel Task with Serverless Compute
I am trying to run a wheel task as part of a bundle on serverless compute. My databricks.yml includes an artifact being constructed:artifacts: nexusbricks: type: whl build: python -m build path: .I then am trying to set up a job to cons...
- 729 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @mlrichmond-mill This is how we do it. environments: - environment_key: serverless_env_v4 spec: environment_version: '4' dependencies: dist/*.whl becasue your bundle whl gets placed into dist/*.whl...
- 0 kudos
- 2621 Views
- 8 replies
- 2 kudos
Pushing data from databricks (cloud) to Oracle (on-prem) instance?
Pushing data from databricks (cloud) to Oracle (on-prem) instance?===================================================Thanks for reviewing my threads. I find some threads on this subject dated in 2022 by @ Ajay-PandeyDatabricks to Oracle We find many...
- 2621 Views
- 8 replies
- 2 kudos
- 2 kudos
Thanks for weighing in and proving an interesting insight. Here are some questions coming to my mind upon reviewing this thread.I like to see a way to write to on-prem linux folder directly from DBX without going thru S3. As we see many features adde...
- 2 kudos
- 673 Views
- 3 replies
- 3 kudos
Resolved! Serverless Compute
Is there a way to create multiple serverless compute instances in a Databricks workspace?I have a use case where multiple all-purpose clusters are allocated to different user groups. I want to replace these all-purpose clusters with serverless comput...
- 673 Views
- 3 replies
- 3 kudos
- 3 kudos
No — Databricks does not currently offer a native way to set hard quotas or throttles on serverless compute per user group (for example, “Team A is limited to X DBUs/hour on serverless notebooks”). Serverless compute is intentionally a shared, autosc...
- 3 kudos
- 495 Views
- 1 replies
- 2 kudos
Resolved! Why do I need MANAGE privilege on a connection to run a pipeline?
I have a notebook that I am able to run in itself, but when I run it within a Databricks job, it fails with the following error:[UNAUTHORIZED_ACCESS] PERMISSION_DENIED: User does not have MANAGE on Connection 'dev_postgres'. SQLSTATE: 42501The notebo...
- 495 Views
- 1 replies
- 2 kudos
- 2 kudos
When you ran the job, was your data_security/access mode set to Dedicated/Single-User or Standard/Shared. I assume it was the former. If it's the latter, you wouldn't have need to give that permission. You would get the same experience trying to do a...
- 2 kudos
- 946 Views
- 7 replies
- 4 kudos
Resolved! How can I manage the code on using a Spot Instance?
Hello community,In the near future, I need to use spot instances to reduce the cost of running a batch processing job.My question is: how can I manage my code to properly handle and capture a SIGTERM signal?Is there any documentation or guidance you ...
- 946 Views
- 7 replies
- 4 kudos
- 4 kudos
@Kirankumarbs — in the case of a single-node cluster, well… you get what you get . A single-node cluster can only run on an on-demand instance. That one node hosts both the driver and the executor, so if the node goes away, there’s no recovery path....
- 4 kudos
- 2457 Views
- 4 replies
- 3 kudos
When accessing a view in Unity Catalog; access to underlying tables of the view is also needed.
My goal is that i want to provide users access to a view but not the underlying tables. I only want them to see specific columns and rows of the table. When i just give the select permissions on the view the user gets an error that they also need acc...
- 2457 Views
- 4 replies
- 3 kudos
- 3 kudos
you may need to check if the compute you are using met what mentioned in https://docs.databricks.com/aws/en/views/#requirements-for-querying-views
- 3 kudos
- 3612 Views
- 2 replies
- 1 kudos
Can not open socket to local (127.0.0.1)
Hi, I'm running a databricks pipeline hourly using python notebooks checked out from git with on-demand compute (using r6gd.xlarge 32GB + 4 CPU's Gravaton). Most of the times the pipeline runs without problems. However, sometimes the first notebook f...
- 3612 Views
- 2 replies
- 1 kudos
- 1 kudos
HI I am facing similar error, the cluster runs 24/7 and this issue is observed for a few runs in the day. The data volume being processed is not huge but the logic that this needs to go though is complex. I do not want to include the display in produ...
- 1 kudos
- 553 Views
- 1 replies
- 2 kudos
Resolved! JDBC with serverless compute
Hi community,We have a scenario where we need to ingest data into Lakebase. Currently, we are trying to use JDBC to write data in a notebook with serverless compute. However, the documentation on serverless limitations (link) mentions that JAR librar...
- 553 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello @Dedescoat , I did some poking around in our documentation and would like to offer some tips/tricks to help you further diagnose the issue. Yes — using a Unity Catalog JDBC connection to load a driver from a UC volume and write from serverless...
- 2 kudos
- 1627 Views
- 3 replies
- 1 kudos
Resolved! How to access artifacts from job run?
In a workflow, is there a way to access task artifacts from within the run?I have a job with a task TasksA, which is a dbt task that creates some artifacts. I want to store these artifacts, but the job artifacts seems to be saved in a location I cann...
- 1627 Views
- 3 replies
- 1 kudos
- 354 Views
- 3 replies
- 1 kudos
Resolved! Connect to spark session and uc tables in python file
How to Connect to spark session and uc tables in python file. I want to read uc tables in python modules in databricks workspace. How to access the current sparksession
- 354 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @ajay_wavicle ,Azure Databricks automatically creates a SparkContext for each compute cluster, and creates an isolated SparkSession for each notebook or job executed against the cluster. So following should work in python module in Databricks Work...
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 555 | |
| 349 | |
| 287 |
