- 852 Views
- 1 replies
- 0 kudos
UDF already defined error when using it into a DLT pipeline
I'm using Unity catalog and defined some UDFs in my catalog.database, as reported by show functions in main.default main.default.getgender main.default.tointlist main.default.tostrlistI can use them from a start warehouse pro:SELECT main.default.get_...
- 852 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @6502, This is a common issue that many users have faced when using UDFs in Unity Catalog. There are a few possible solutions that you can try to fix this error: You can drop the existing function main.default.get_gender from your catalog databa...
- 0 kudos
- 1180 Views
- 1 replies
- 0 kudos
DLT || Python || Aggregate Functions recomputing all the records
Hi all, I am building a realtime dashboard using Databricks Delta Live Tables Pipeline and using the following steps : - Bronze Table : Using the autoloader functionality provided by databricks, its incrementally ingesting new files records into a br...
- 1180 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @shivam-singh, One option is to use change data capture (CDC) in Databricks Delta Live Tables Pipeline to update tables based on changes in source data. CDC lets you capture and apply changes from streaming sources like Kafka, S3, or Delta Lake t...
- 0 kudos
- 624 Views
- 1 replies
- 0 kudos
Registered for Wrong certification
Hi, I am registered for databricks certification today at 23:00hr and I accidentally choose databricks data analyst certification instead of databricks data engineer associate.I have left several mails and submitted several forms to be able to fix th...
- 624 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Pavani22, Thank you for posting your concern on Community! To expedite your request, please list your concerns on our ticketing portal. Our support staff would be able to act faster on the resolution (our standard resolution time is 24-48 hours).
- 0 kudos
- 9750 Views
- 8 replies
- 0 kudos
error after updating delta table com.databricks.sql.transaction.tahoe.DeltaUnsupportedOperationException: Detected a data update
Hi,I have pipeline running. I have updated one file in delta table which is already processed. Now I am getting errorcom.databricks.sql.transaction.tahoe.DeltaUnsupportedOperationException: Detected a data update. This is currently not supported. If ...
- 9750 Views
- 8 replies
- 0 kudos
- 1127 Views
- 2 replies
- 0 kudos
Materialize View creation not supported from DLT Pipeline
Hi Team, I have a very basic scenario where I am using my custom catalog and want materialize view to get created from DLT table at the end of pipeline. The SQL used as below for the same.where "loom_data_transform" is a Streaming table. But pipeline...
- 1127 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey @abhijitnag are you sure your loom_data_transform was created as a STREAMING table? docs:https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-streaming-table.html
- 0 kudos
- 1600 Views
- 1 replies
- 1 kudos
Issue with Reading MongoDB Data in Unity Catalog Cluster
I am encountering an issue while trying to read data from MongoDB in a Unity Catalog Cluster using PySpark. I have shared my code below: from pyspark.sql import SparkSession database = "cloud" collection = "data" Scope = "XXXXXXXX" Key = "XXXXXX-YYY...
- 1600 Views
- 1 replies
- 1 kudos
- 1 kudos
Few points 1. chce if you installed exactly same driver version as you are pointing this in code (2.12:3.2.0) it has to match 100percentorg.mongodb.spark:mongo-spark-connector_2.12:3.2.02. I have seen people configuring connction to atlas in two way...
- 1 kudos
- 1641 Views
- 1 replies
- 0 kudos
DeltaRuntimeException: Keeping the source of the MERGE statement materialized has failed repeatedly.
I'm using Autoloader (in Azure Databricks) to read parquet files and write their data into the Delta table.schemaEvolutionMode is set to 'rescue'.In foreach_batch I do1) Transform of read dataframe;2) Create temp view based on read dataframe and merg...
- 1641 Views
- 1 replies
- 0 kudos
- 0 kudos
Hmm, you can't have duplicated data in source dataframe/batch but it should error out with diffrent erro like "Cannot perform Merge as multiple source rows matched and attempted to modify the same target row...".Also this behaviour after rerun is str...
- 0 kudos
- 1529 Views
- 1 replies
- 0 kudos
Slow stream static join in Spark Structured Streaming
SituationRecords are streamed from an input Delta table via a Spark Structured Streaming job. The streaming job performs the following.Read from input Delta table (readStream)Static join on small JSONStatic join on big Delta tableWrite to three Delta...
- 1529 Views
- 1 replies
- 0 kudos
- 0 kudos
You have quite small machines that you are using, please take into consideration that a lot of memory of machine is occupied by other processes https://kb.databricks.com/clusters/spark-shows-less-memoryThis is not good idea to broadcast huge data fra...
- 0 kudos
- 9365 Views
- 6 replies
- 3 kudos
Resolved! How to run code-formating on the notebooks
Has anyone found a nice way to run code-formating (like black) on the notebooks **in the workspace**? My current workflow is to commit the file, pull it locally, format, repush and pull. It would be nice if it was some relatively easy way to run blac...
- 9365 Views
- 6 replies
- 3 kudos
- 3 kudos
Hi Erik,I don't know if you are aware of this feature, currently there is an option to format the code in your databricks notebooks using the black code style formatter.Just you need to either have a version of your DBR equal to or greater than 11.2 ...
- 3 kudos
- 1546 Views
- 2 replies
- 1 kudos
Resolved! Are there system tables that are customer accessible for setting up job run health monitoring in GCP Databricks?
Is Overwatch still an active project, is there anything equivalent for GCP Databricks or any plans for Overwatch to be available in GCP?
- 1546 Views
- 2 replies
- 1 kudos
- 455 Views
- 0 replies
- 0 kudos
Help design my streaming pipeline
###Data Source- AWS RDS- Database migration tasks have been created using AWS DMS- Relevant cdc information is being stored in a specific bucket in S3### Data frequency- Once a day (but not sure when, sometime after 6pm)### Development environment- d...
- 455 Views
- 0 replies
- 0 kudos
- 1468 Views
- 1 replies
- 0 kudos
Handling data close to SCD2 with Delta tables
Hello, stack used: pyspark and delta tablesI'm working with some data that look a bit like SCD2 data.Basically, the data has columns that represent an id, a rank column and other informations, here's an example:login, email, business_timestamp => the...
- 1468 Views
- 1 replies
- 0 kudos
- 0 kudos
Your problem is exactly like SCD2 . You just add one more column with valid to date ( optionals you can add flag is actual to tag current records)You can use DLT apply changes syntax. Alternatively Merge statement .On the top of that table you can bu...
- 0 kudos
- 16914 Views
- 6 replies
- 6 kudos
- 16914 Views
- 6 replies
- 6 kudos
- 6961 Views
- 3 replies
- 1 kudos
Resolved! Databricks approaches to CDC
I'm interested in learning more about Change Data Capture (CDC) approaches with Databricks. Can anyone provide insights on the best practices and recommendations for utilizing CDC effectively in Databricks? Are there any specific connectors or tools ...
- 6961 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi, first of all thank you all in advance! I am very interested on this topic!My question is beyond what it is described here. As well as @Pektas , I am using debezium to send data from Postgres to a Kafka topic (in fact, Azure EventHub). My question...
- 1 kudos
- 4519 Views
- 4 replies
- 0 kudos
BINARY data type
Hello everyone.I'm trying to understand how BINARY data type works in spark sql. According to examples in the documentation, using cast or literal 'X' should return HEX representation of the binary data type, but when I try the same code, I see base6...
- 4519 Views
- 4 replies
- 0 kudos
- 0 kudos
If you are confused , please look at this thread, they explain that Databricks use base64 as binary default. This is not documented but can be tracked at source code level.https://stackoverflow.com/questions/75753311/not-getting-binary-value-in-datab...
- 0 kudos