- 3173 Views
- 1 replies
- 0 kudos
Unable to locate "Configure databricks" in visual studio code
Need help in locating "configure databricks" option to connect to a cluster in visual studio code. Not sure which settings need to enabled to view/access the configuration. Below is the screen shot of it.
- 3173 Views
- 1 replies
- 0 kudos
- 0 kudos
- 0 kudos
- 2343 Views
- 0 replies
- 0 kudos
Dashboard HTML Export: DisplayHTML Formatting Lost
I am building an analytics dashboard that we plan to export into HTML to provide to clients. We are running into major issues specifically with DisplayHTML visualizations. We are using DisplayHTML for help text, sub-headings, and for a KPI banner at ...
- 2343 Views
- 0 replies
- 0 kudos
- 5085 Views
- 1 replies
- 0 kudos
DeltaRuntimeException: Keeping the source of the MERGE statement materialized has failed repeatedly.
I'm using Autoloader (in Azure Databricks) to read parquet files and write their data into the Delta table.schemaEvolutionMode is set to 'rescue'.In foreach_batch I do1) Transform of read dataframe;2) Create temp view based on read dataframe and merg...
- 5085 Views
- 1 replies
- 0 kudos
- 0 kudos
Hmm, you can't have duplicated data in source dataframe/batch but it should error out with diffrent erro like "Cannot perform Merge as multiple source rows matched and attempted to modify the same target row...".Also this behaviour after rerun is str...
- 0 kudos
- 4380 Views
- 1 replies
- 0 kudos
Slow stream static join in Spark Structured Streaming
SituationRecords are streamed from an input Delta table via a Spark Structured Streaming job. The streaming job performs the following.Read from input Delta table (readStream)Static join on small JSONStatic join on big Delta tableWrite to three Delta...
- 4380 Views
- 1 replies
- 0 kudos
- 0 kudos
You have quite small machines that you are using, please take into consideration that a lot of memory of machine is occupied by other processes https://kb.databricks.com/clusters/spark-shows-less-memoryThis is not good idea to broadcast huge data fra...
- 0 kudos
- 17435 Views
- 4 replies
- 3 kudos
Resolved! How to run code-formating on the notebooks
Has anyone found a nice way to run code-formating (like black) on the notebooks **in the workspace**? My current workflow is to commit the file, pull it locally, format, repush and pull. It would be nice if it was some relatively easy way to run blac...
- 17435 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi Erik,I don't know if you are aware of this feature, currently there is an option to format the code in your databricks notebooks using the black code style formatter.Just you need to either have a version of your DBR equal to or greater than 11.2 ...
- 3 kudos
- 2774 Views
- 2 replies
- 1 kudos
Resolved! Are there system tables that are customer accessible for setting up job run health monitoring in GCP Databricks?
Is Overwatch still an active project, is there anything equivalent for GCP Databricks or any plans for Overwatch to be available in GCP?
- 2774 Views
- 2 replies
- 1 kudos
- 975 Views
- 0 replies
- 0 kudos
Help design my streaming pipeline
###Data Source- AWS RDS- Database migration tasks have been created using AWS DMS- Relevant cdc information is being stored in a specific bucket in S3### Data frequency- Once a day (but not sure when, sometime after 6pm)### Development environment- d...
- 975 Views
- 0 replies
- 0 kudos
- 3048 Views
- 1 replies
- 0 kudos
Handling data close to SCD2 with Delta tables
Hello, stack used: pyspark and delta tablesI'm working with some data that look a bit like SCD2 data.Basically, the data has columns that represent an id, a rank column and other informations, here's an example:login, email, business_timestamp => the...
- 3048 Views
- 1 replies
- 0 kudos
- 0 kudos
Your problem is exactly like SCD2 . You just add one more column with valid to date ( optionals you can add flag is actual to tag current records)You can use DLT apply changes syntax. Alternatively Merge statement .On the top of that table you can bu...
- 0 kudos
- 26365 Views
- 5 replies
- 6 kudos
- 26365 Views
- 5 replies
- 6 kudos
- 9481 Views
- 4 replies
- 0 kudos
BINARY data type
Hello everyone.I'm trying to understand how BINARY data type works in spark sql. According to examples in the documentation, using cast or literal 'X' should return HEX representation of the binary data type, but when I try the same code, I see base6...
- 9481 Views
- 4 replies
- 0 kudos
- 0 kudos
If you are confused , please look at this thread, they explain that Databricks use base64 as binary default. This is not documented but can be tracked at source code level.https://stackoverflow.com/questions/75753311/not-getting-binary-value-in-datab...
- 0 kudos
- 1573 Views
- 1 replies
- 0 kudos
Shutdown Cluster in script if there is any failure
I am working on incremental load from sql server to Delta lake tables stored in ADLS gen2. DUring the script i need to qrite a logic toShut down the DB cluster on failure (there needs to be logging added to ensure that shutdown happens promptly to pr...
- 1573 Views
- 1 replies
- 0 kudos
- 0 kudos
If you run your notebook via workflow and error happen and there are no retires on job, then job cluster will be terminated immidietly after failure.You can add python block of try catch and if error occurs , you catch the error and log somewhere bef...
- 0 kudos
- 1794 Views
- 1 replies
- 0 kudos
Remove description from job
How do I remove a description from a job completely? When I try to just remove the text in the edit window, the same text shows up afterwards, even though it says "Successfully updated job". Also I had to write this twice, because on the first try I ...
- 1794 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi,this is not possible form UI You have to replace content with e.g. white space. I think this is bug.But you can do it using job api !Below example in PowerShell, just reaplce:job_istokenworkspaceURL$body = @' { "job_id": 123456789, "new_setti...
- 0 kudos
- 7980 Views
- 5 replies
- 5 kudos
How to change cluster size using a script
I want to change instance type or number of max workers via a python script. Does anyone know how to do it/is it possible? I have a lot of background jobs when I want to scale down my workers, so autoscaling is not an option. I was getting an error t...
- 7980 Views
- 5 replies
- 5 kudos
- 5 kudos
Hi ksenija, this is just my guess but maybe you are using Cluster Policy in your cluster that only allows you to use specific cluster size ? E.g. below cluster policy that limits to some cluster sizes only.
- 5 kudos
- 7836 Views
- 6 replies
- 3 kudos
Change DLT table type from streaming to 'normal'
I have a DLT streaming live table, and after watching a QA session, I saw that it is advised to only use streaming tables for your raw landing. I attempted to modify my pipeline to have my silver table be a regular LIVE TABLE, but an error was throw...
- 7836 Views
- 6 replies
- 3 kudos
- 3 kudos
Just curious, could you point me to said QA session if it's a video or something? I'm not aware of such a limitation. You can use DLT's live streaming tables anywhere in the Medallion architecture, just make sure not to break stream composability by ...
- 3 kudos
- 10077 Views
- 2 replies
- 0 kudos
Delta Lake, CFD & SCD2
HiWhat's the best way to deal with SCD2-styled tables in silver and/or gold layer while streaming.From what I've seen in the Professional Data Engineer videos, they usually go for SCD1 tables (simple updates or deletes)In a SCD2 scenario, we need to ...
- 10077 Views
- 2 replies
- 0 kudos
- 0 kudos
I did some further reading and got the same conclusion. APPLY CHANGES might to the trick. However, I don't like the limitations. From Bronze to Silver I might need .foreachBatch to implement the JSON-logic and the attribute names (__start_at / __end_...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 580 | |
| 349 | |
| 287 |
