- 725 Views
- 2 replies
- 0 kudos
Delta live tables - ignore updates on some columns
Hello Team,I have scenario where in apply_changes, i want to ignore updates on 1 column. Is there any way we can achieve this in Delta live tables?
- 725 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi there @Anish_2 , Yes you can do that Here is the doc link : https://docs.databricks.com/aws/en/dlt/cdc?language=PythonFor python you can simply add an attribute except_columns_list like thisdlt.apply_changes( target = "target", source = "users...
- 0 kudos
- 404 Views
- 1 replies
- 0 kudos
Unable to export object using /api/2.0/workspace/export API
Hi ,I was using /api/2.0/workspace/export API in power automate workflow to export to excel from data brick to sharepoint. This functionality was working fine till yesterday. Today onwards it is throwing below error while calling the APIAction 'HTTP_...
- 404 Views
- 1 replies
- 0 kudos
- 0 kudos
@hims_2021 This error indicates an encoding issue when trying to export an Excel file from Databricks to SharePoint via Power Automate. The specific error message about being "Unable to translate bytes [9A] at index 11" suggests that Power Automate i...
- 0 kudos
- 5013 Views
- 8 replies
- 2 kudos
Resolved! DLT Direct Publishing Mode
Hello,I'm working on a DLT pipeline and have a block of SQL that runs...USE CATALOG catalog_a; USE SCHEMA schema_a; CREATE OR REFRESH MATERIALIZED VIEW table_a AS SELECT ... FROM catalog_b.schema_b.table_b;Executing this block returns the following.....
- 5013 Views
- 8 replies
- 2 kudos
- 2 kudos
I'm in EastUS and i don't have that option on my previews page. Also it only works with serverless?
- 2 kudos
- 12510 Views
- 9 replies
- 8 kudos
Databricks short cut to split a cell
Is there a shortcut to split a cell into two in Dtabricks notebook as in Jupiter notebook? in jupyter notebook it is Shift/Ctr/-
- 12510 Views
- 9 replies
- 8 kudos
- 8 kudos
Hi @mundy Jim​ / All, Attached are two snapshots so first snapshot with one cell if pressed Ctrl+Alt+Minus split into two.
- 8 kudos
- 1011 Views
- 3 replies
- 1 kudos
Resolved! How to Update Identity Column for a Databricks Table
Hi All,I have a databricks table with the below DDL:CREATE TABLE default.Test ( ID BIGINT GENERATED ALWAYS AS IDENTITY (START WITH 1 INCREMENT BY 1), StopFromDateTime TIMESTAMP, StopToDateTime TIMESTAMP, User STRING) USING delta TBLPROPERTIE...
- 1011 Views
- 3 replies
- 1 kudos
- 1 kudos
If you recreate the table using BIGINT GENERATED BY DEFAULT instead of BIGINT GENERATED ALWAYS you can manipulate the column values."When using the clause GENERATED BY DEFAULT AS IDENTITY, insert operations can specify values for the identity column...
- 1 kudos
- 770 Views
- 1 replies
- 0 kudos
add custom logs and save in a folder logs
Hi,I am trying to add custom logging functionality for my code. Please refer to the code I am using, I am trying to save my log files by creating a logs folder in my users workspace. My intent is to store dynamic custom log files each time I run my n...
- 770 Views
- 1 replies
- 0 kudos
- 0 kudos
Here are some suggestions for your consideration. The issue with your custom logging setup seems to stem from attempting to save the log files in a path under "/Workspace/Users/ramya.v@point32health.org/CD/", which is not directly writable by your ...
- 0 kudos
- 9425 Views
- 4 replies
- 1 kudos
UDF not working inside transform() & lambda (SQL)
Below is toy example of what I'm trying to achieve, but don't understand why it fails. Can anyone explain why, and suggest a fix or not overly bloated workaround?%sqlcreate or replace function status_map(status int)returns stringreturn map(10, "STATU...
- 9425 Views
- 4 replies
- 1 kudos
- 1 kudos
Scope of lambda implementation within transform function only ,which fails with udf because lambda variable bindings (e in your case) are not available for use within the UDF's scope. we can use below workaround for above example which generate same ...
- 1 kudos
- 1682 Views
- 2 replies
- 1 kudos
Does Databricks have a data processing agreement?
Does Databricks have a data processing agreement?
- 1682 Views
- 2 replies
- 1 kudos
- 1 kudos
Yes, Databricks typically provides a Data Processing Agreement (DPA) to comply with data protection regulations like GDPR. It's important for businesses to thoroughly review these agreements to ensure alignment with their data privacy policies. You c...
- 1 kudos
- 1657 Views
- 3 replies
- 1 kudos
How to define DLT table with cyclic reference
@Dlt.table def table_A(): return ( dlt.read_stream(...) ) @dlt.table def table_join_A_and_C(): df_A = dlt.read_stream(table_A) df_C = dlt.read_stream(table_C) return ( ....df_A.join(df_C) ) @dlt.table def table_C(): return ( ...
- 1657 Views
- 3 replies
- 1 kudos
- 974 Views
- 3 replies
- 0 kudos
DLT Append Flow Parameterization
Hi All,I'm currently using DLT append flow to merge multiple streaming flows into one output.While trying to make the append flow into a dynamic function for scalability, the dlt append flow seem to have some errors.stat_table = f"{catalog}.{bronze_s...
- 974 Views
- 3 replies
- 0 kudos
- 0 kudos
The error you're encountering occurs because Delta Live Tables (DLT) append flows currently do not support streaming aggregations or other transformations on streaming DataFrames unless a watermark is applied properly to handle late data. Based on yo...
- 0 kudos
- 4129 Views
- 1 replies
- 0 kudos
Oracle Lakehouse Federaton with CA Certificate
Hi!We have been pulling data from Oracle to Databricks by installing Oracle Driver and certificates directly in the cluster. We are now looking into using Lakehouse Federation for Oracle instead, but it seems like the connection doesn't pick up the c...
- 4129 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @AntonDBUser ,Thanks for sharing your experience, we’re looking into using Lakehouse Federation with Oracle too.I haven’t tested this myself, but one idea that came to mind is whether switching from a serverless cluster to a standard (Pro) cluster...
- 0 kudos
- 2025 Views
- 6 replies
- 1 kudos
external table not being written to data lake
I was following a tutorial to learn databricks from https://youtu.be/7pee6_Sq3VYGreat video btwI am stuck here at 2:52:24I am trying to create an external table directly to data lake but i am facing some weird issue saying no such location exists.I h...
- 2025 Views
- 6 replies
- 1 kudos
- 1 kudos
Hey @mridultuteja To register an external location, you have to first create a Storage Credential, and then create the External Location.This process allows Databricks to securely access data stored in Azure Data Lake Storage Gen2 (ADLS Gen2), while ...
- 1 kudos
- 442 Views
- 1 replies
- 0 kudos
Unity Catalog Enablement
Hello,After scouring documentation yesterday, I was finally able to get unity catalog enabled and assigned to my workspace. Or so I thought. When I run the CURRENT METASTORE() command I get the below error:However, when I look at my catalog I can see...
- 442 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi,Please check if the cluster you are using to run the query as well upgraded to Unity Catalog. Also, follow the best practices outlined here for enablement: https://docs.databricks.com/aws/en/data-governance/unity-catalog/enable-workspacesBest Rega...
- 0 kudos
- 2440 Views
- 1 replies
- 0 kudos
Unity Catalog Python UDF to Send Messages to MS Teams
Good Morning All - This didn't seem like such a daunting task until I tried it. Of course, it's my very first function in Unity Catalog. Attached are images of both the UDF and example usage I created to send messages via the Python requests library ...
- 2440 Views
- 1 replies
- 0 kudos
- 0 kudos
This has been dropped in favor of using a function defined internally within a notebook. If anyone has occasion to get a similar process set up - please let me know.Thanks
- 0 kudos
- 556 Views
- 1 replies
- 0 kudos
How to get response from API call made via executor
Hi Guys,I'm trying to call multiple APIs via executor using foreach partition, However as API response is getting returned at executor level I'm unable to see the response of API weather its 200 or 500.I dont want my APIs to execute on driver so I'm ...
- 556 Views
- 1 replies
- 0 kudos
- 0 kudos
Vaibhavaher2025 - I recommend trying the following: 1. Write logs from executors to persist storage insideprocess_partition. 2. Use mapPartitions instead offoreachPartition to return responses back to the driver as a Dataframe 3. Check executor log...
- 0 kudos
| User | Count |
|---|---|
| 1615 | |
| 780 | |
| 349 | |
| 287 | |
| 275 |
