- 2013 Views
- 0 replies
- 0 kudos
- 2013 Views
- 0 replies
- 0 kudos
Forum Posts
- 3483 Views
- 2 replies
- 0 kudos
Materialize View creation not supported from DLT Pipeline
Hi Team, I have a very basic scenario where I am using my custom catalog and want materialize view to get created from DLT table at the end of pipeline. The SQL used as below for the same.where "loom_data_transform" is a Streaming table. But pipeline...
- 3483 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hey @abhijitnag are you sure your loom_data_transform was created as a STREAMING table? docs:https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-streaming-table.html
- 0 kudos
- 2197 Views
- 0 replies
- 0 kudos
DLT || Python || Aggregate Functions recomputing all the records
Hi all, I am building a realtime dashboard using Databricks Delta Live Tables Pipeline and using the following steps : - Bronze Table : Using the autoloader functionality provided by databricks, its incrementally ingesting new files records into a br...
- 2197 Views
- 0 replies
- 0 kudos
- 1266 Views
- 0 replies
- 0 kudos
Registered for Wrong certification
Hi, I am registered for databricks certification today at 23:00hr and I accidentally choose databricks data analyst certification instead of databricks data engineer associate.I have left several mails and submitted several forms to be able to fix th...
- 1266 Views
- 0 replies
- 0 kudos
- 4489 Views
- 1 replies
- 1 kudos
Issue with Reading MongoDB Data in Unity Catalog Cluster
I am encountering an issue while trying to read data from MongoDB in a Unity Catalog Cluster using PySpark. I have shared my code below: from pyspark.sql import SparkSession database = "cloud" collection = "data" Scope = "XXXXXXXX" Key = "XXXXXX-YYY...
- 4489 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
Few points 1. chce if you installed exactly same driver version as you are pointing this in code (2.12:3.2.0) it has to match 100percentorg.mongodb.spark:mongo-spark-connector_2.12:3.2.02. I have seen people configuring connction to atlas in two way...
- 1 kudos
- 3228 Views
- 1 replies
- 0 kudos
Unable to locate "Configure databricks" in visual studio code
Need help in locating "configure databricks" option to connect to a cluster in visual studio code. Not sure which settings need to enabled to view/access the configuration. Below is the screen shot of it.
- 3228 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
- 0 kudos
- 2418 Views
- 0 replies
- 0 kudos
Dashboard HTML Export: DisplayHTML Formatting Lost
I am building an analytics dashboard that we plan to export into HTML to provide to clients. We are running into major issues specifically with DisplayHTML visualizations. We are using DisplayHTML for help text, sub-headings, and for a KPI banner at ...
- 2418 Views
- 0 replies
- 0 kudos
- 5157 Views
- 1 replies
- 0 kudos
DeltaRuntimeException: Keeping the source of the MERGE statement materialized has failed repeatedly.
I'm using Autoloader (in Azure Databricks) to read parquet files and write their data into the Delta table.schemaEvolutionMode is set to 'rescue'.In foreach_batch I do1) Transform of read dataframe;2) Create temp view based on read dataframe and merg...
- 5157 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hmm, you can't have duplicated data in source dataframe/batch but it should error out with diffrent erro like "Cannot perform Merge as multiple source rows matched and attempted to modify the same target row...".Also this behaviour after rerun is str...
- 0 kudos
- 4445 Views
- 1 replies
- 0 kudos
Slow stream static join in Spark Structured Streaming
SituationRecords are streamed from an input Delta table via a Spark Structured Streaming job. The streaming job performs the following.Read from input Delta table (readStream)Static join on small JSONStatic join on big Delta tableWrite to three Delta...
- 4445 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
You have quite small machines that you are using, please take into consideration that a lot of memory of machine is occupied by other processes https://kb.databricks.com/clusters/spark-shows-less-memoryThis is not good idea to broadcast huge data fra...
- 0 kudos
- 17549 Views
- 4 replies
- 3 kudos
Resolved! How to run code-formating on the notebooks
Has anyone found a nice way to run code-formating (like black) on the notebooks **in the workspace**? My current workflow is to commit the file, pull it locally, format, repush and pull. It would be nice if it was some relatively easy way to run blac...
- 17549 Views
- 4 replies
- 3 kudos
Latest Reply
- 3 kudos
Hi Erik,I don't know if you are aware of this feature, currently there is an option to format the code in your databricks notebooks using the black code style formatter.Just you need to either have a version of your DBR equal to or greater than 11.2 ...
- 3 kudos
- 2823 Views
- 2 replies
- 1 kudos
Resolved! Are there system tables that are customer accessible for setting up job run health monitoring in GCP Databricks?
Is Overwatch still an active project, is there anything equivalent for GCP Databricks or any plans for Overwatch to be available in GCP?
- 2823 Views
- 2 replies
- 1 kudos
Latest Reply
- 993 Views
- 0 replies
- 0 kudos
Help design my streaming pipeline
###Data Source- AWS RDS- Database migration tasks have been created using AWS DMS- Relevant cdc information is being stored in a specific bucket in S3### Data frequency- Once a day (but not sure when, sometime after 6pm)### Development environment- d...
- 993 Views
- 0 replies
- 0 kudos
- 3121 Views
- 1 replies
- 0 kudos
Handling data close to SCD2 with Delta tables
Hello, stack used: pyspark and delta tablesI'm working with some data that look a bit like SCD2 data.Basically, the data has columns that represent an id, a rank column and other informations, here's an example:login, email, business_timestamp => the...
- 3121 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Your problem is exactly like SCD2 . You just add one more column with valid to date ( optionals you can add flag is actual to tag current records)You can use DLT apply changes syntax. Alternatively Merge statement .On the top of that table you can bu...
- 0 kudos
- 26607 Views
- 5 replies
- 6 kudos
- 26607 Views
- 5 replies
- 6 kudos
Latest Reply
- 9585 Views
- 4 replies
- 0 kudos
BINARY data type
Hello everyone.I'm trying to understand how BINARY data type works in spark sql. According to examples in the documentation, using cast or literal 'X' should return HEX representation of the binary data type, but when I try the same code, I see base6...
- 9585 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
If you are confused , please look at this thread, they explain that Databricks use base64 as binary default. This is not documented but can be tracked at source code level.https://stackoverflow.com/questions/75753311/not-getting-binary-value-in-datab...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
