- 1297 Views
- 2 replies
- 1 kudos
Does Databricks have a data processing agreement?
Does Databricks have a data processing agreement?
- 1297 Views
- 2 replies
- 1 kudos
- 1 kudos
Yes, Databricks typically provides a Data Processing Agreement (DPA) to comply with data protection regulations like GDPR. It's important for businesses to thoroughly review these agreements to ensure alignment with their data privacy policies. You c...
- 1 kudos
- 1250 Views
- 3 replies
- 1 kudos
How to define DLT table with cyclic reference
@Dlt.table def table_A(): return ( dlt.read_stream(...) ) @dlt.table def table_join_A_and_C(): df_A = dlt.read_stream(table_A) df_C = dlt.read_stream(table_C) return ( ....df_A.join(df_C) ) @dlt.table def table_C(): return ( ...
- 1250 Views
- 3 replies
- 1 kudos
- 323 Views
- 3 replies
- 0 kudos
DLT Append Flow Parameterization
Hi All,I'm currently using DLT append flow to merge multiple streaming flows into one output.While trying to make the append flow into a dynamic function for scalability, the dlt append flow seem to have some errors.stat_table = f"{catalog}.{bronze_s...
- 323 Views
- 3 replies
- 0 kudos
- 0 kudos
The error you're encountering occurs because Delta Live Tables (DLT) append flows currently do not support streaming aggregations or other transformations on streaming DataFrames unless a watermark is applied properly to handle late data. Based on yo...
- 0 kudos
- 2274 Views
- 1 replies
- 0 kudos
Oracle Lakehouse Federaton with CA Certificate
Hi!We have been pulling data from Oracle to Databricks by installing Oracle Driver and certificates directly in the cluster. We are now looking into using Lakehouse Federation for Oracle instead, but it seems like the connection doesn't pick up the c...
- 2274 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @AntonDBUser ,Thanks for sharing your experience, we’re looking into using Lakehouse Federation with Oracle too.I haven’t tested this myself, but one idea that came to mind is whether switching from a serverless cluster to a standard (Pro) cluster...
- 0 kudos
- 631 Views
- 6 replies
- 1 kudos
external table not being written to data lake
I was following a tutorial to learn databricks from https://youtu.be/7pee6_Sq3VYGreat video btwI am stuck here at 2:52:24I am trying to create an external table directly to data lake but i am facing some weird issue saying no such location exists.I h...
- 631 Views
- 6 replies
- 1 kudos
- 1 kudos
Hey @mridultuteja To register an external location, you have to first create a Storage Credential, and then create the External Location.This process allows Databricks to securely access data stored in Azure Data Lake Storage Gen2 (ADLS Gen2), while ...
- 1 kudos
- 256 Views
- 1 replies
- 0 kudos
Unity Catalog Enablement
Hello,After scouring documentation yesterday, I was finally able to get unity catalog enabled and assigned to my workspace. Or so I thought. When I run the CURRENT METASTORE() command I get the below error:However, when I look at my catalog I can see...
- 256 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi,Please check if the cluster you are using to run the query as well upgraded to Unity Catalog. Also, follow the best practices outlined here for enablement: https://docs.databricks.com/aws/en/data-governance/unity-catalog/enable-workspacesBest Rega...
- 0 kudos
- 539 Views
- 1 replies
- 0 kudos
Unity Catalog Python UDF to Send Messages to MS Teams
Good Morning All - This didn't seem like such a daunting task until I tried it. Of course, it's my very first function in Unity Catalog. Attached are images of both the UDF and example usage I created to send messages via the Python requests library ...
- 539 Views
- 1 replies
- 0 kudos
- 0 kudos
This has been dropped in favor of using a function defined internally within a notebook. If anyone has occasion to get a similar process set up - please let me know.Thanks
- 0 kudos
- 349 Views
- 3 replies
- 0 kudos
how to restrict creation serving endpoints in databricks to a user
Is it possible somehow to restrict creation of the serving endpoints to specific users? I want to grant Workspace access under the Entitlements of the specific group, but I want not to allow users of this group create serving endpoints.The only way I...
- 349 Views
- 3 replies
- 0 kudos
- 0 kudos
@drag7ter While I mentioned Workspace-level entitlements as an approach, I need to clarify something important: Databricks doesn't currently support creating fully customized entitlements where you can exclude specific permissions like "serving endpo...
- 0 kudos
- 274 Views
- 1 replies
- 0 kudos
How to get response from API call made via executor
Hi Guys,I'm trying to call multiple APIs via executor using foreach partition, However as API response is getting returned at executor level I'm unable to see the response of API weather its 200 or 500.I dont want my APIs to execute on driver so I'm ...
- 274 Views
- 1 replies
- 0 kudos
- 0 kudos
Vaibhavaher2025 - I recommend trying the following: 1. Write logs from executors to persist storage insideprocess_partition. 2. Use mapPartitions instead offoreachPartition to return responses back to the driver as a Dataframe 3. Check executor log...
- 0 kudos
- 20031 Views
- 14 replies
- 3 kudos
Resolved! Databricks SQL - Unable to Escape Dollar Sign ($) in Column Name
It seems that due to how Databricks processes SQL cells, it's impossible to escape the $ when it comes to a column name.I would expect the following to work:%sql SELECT 'hi' `$id`The backticks ought to escape everything. And indeed that's exactly wha...
- 20031 Views
- 14 replies
- 3 kudos
- 519 Views
- 6 replies
- 0 kudos
Resolved! Serverless: ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied
Hello guys!I am getting this error when running a job:ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/some-python-package'I have lis...
- 519 Views
- 6 replies
- 0 kudos
- 3708 Views
- 22 replies
- 1 kudos
Resolved! BQ partition data deleted fully even though 'spark.sql.sources.partitionOverwriteMode' is DYNAMIC
We have a date (DD/MM/YYYY) partitioned BQ table. We want to update a specific partition data in 'overwrite' mode using PySpark. So to do this, I applied 'spark.sql.sources.partitionOverwriteMode' to 'DYNAMIC' as per the spark bq connector documentat...
- 3708 Views
- 22 replies
- 1 kudos
- 1 kudos
@soumiknow , Just checking if there are any further questions, and did my last comment help?
- 1 kudos
- 271 Views
- 2 replies
- 2 kudos
Dataframe is getting empty during execution of daily job with random pattern
Hello, I have a daily ETL job that adds new records to a table for the previous day. However, from time to time, it does not produce any output.After investigating, I discovered that one table is sometimes loaded as empty during execution. As a resul...
- 271 Views
- 2 replies
- 2 kudos
- 2 kudos
Thank you very much, @BigRoux , for such a detailed and insightful answer!All tables used in this processing are managed Delta tables loaded through Unity Catalog.I will try running it with spark.databricks.io.cache.enabled set to false just to see i...
- 2 kudos
- 635 Views
- 6 replies
- 4 kudos
Databricks warehouse table optimization
Hi everyone,I just started using the Databricks and wanted to evaluate the reading speeds when using the Databricks warehouse.So I've generated the dataset of 100M records, which contains name, surname, date of birth, phone number and an address. Dat...
- 635 Views
- 6 replies
- 4 kudos
- 4 kudos
Hi Brahmareddy and AndrewN,Thank you on your answers.I first need to apologize as I accidentally wrote wrong that I got 270ms with hashing the date of birth, surname and name and then using the z ordering.I actually achieved around 290ms with hashing...
- 4 kudos
- 316 Views
- 4 replies
- 0 kudos
API Definition File
Hello. We are in the process of setting up Azure APIM to Databricks Rest API(s). Is there an official definition file available for download?Any help would be greatly appreciated
- 316 Views
- 4 replies
- 0 kudos
- 0 kudos
Thank you for the feedback. The postman collection would be ideal but the link is a no longer active
- 0 kudos
| User | Count |
|---|---|
| 1612 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
