- 22095 Views
- 6 replies
- 11 kudos
How to remove widgets from a notebook dashboard?
I'm creating a dashboard from the output of a notebook cell, but noticing that the dashboard displays the the widgets of the notebook in addition to the cell output. How can I remove the widgets from the dashboard?
- 22095 Views
- 6 replies
- 11 kudos
- 11 kudos
Did you find any solution for this? I am facing a similar issue wanting to create multiple dashboads on a single notebook where not all widgets are relevant for both dashboards. this makes it difficult for users to understand the dahsboard.
- 11 kudos
- 9825 Views
- 1 replies
- 1 kudos
Schema Deletion -Structured Streaming
Hi,I have a Structured Stream which reads data from my silver layer and creates a gold layer using foreachBatch. The stream has been working fine, but now I have change where there are deletions to the schema and some of the columns from the silver l...
- 9825 Views
- 1 replies
- 1 kudos
- 1 kudos
@Retired_mod Thank you so much for a detailed explanation
- 1 kudos
- 14596 Views
- 2 replies
- 2 kudos
encryption
Hi Databricks, Could you please guide me on the below scenario?Here is the use case we are trying to solve forCurrently environment is using “Voltage” as an encryption tool for encrypting the data in S3 in conjunction with business business-provided ...
- 14596 Views
- 2 replies
- 2 kudos
- 2 kudos
To replace "Voltage" with Databricks encryption, follow these steps: set up a Customer Managed Key in AWS, configure the S3 bucket, read data in Databricks, and implement custom UDFs for AES encryption/decryption.
- 2 kudos
- 12435 Views
- 1 replies
- 2 kudos
Resolved! DLT pipeline evolution schema error
Hello everyone, I am currently working on my first dlt pipeline, and I stumped on a problem which I am struggling to solve.I am working on several tables where I have a column called "my_column" with an array of json with two keys : 1 key : score, 2n...
- 12435 Views
- 1 replies
- 2 kudos
- 5901 Views
- 4 replies
- 3 kudos
Resolved! delta live table udf not known when defined in python module
Hi I have the problem that my "module" is not known when used in a user defined function. The precise message is posted below. I have a repo structure as follows: analytics_pipelines │ ├── __init__.py │ ├── coordinate_transformation.py │ ├── d...
- 5901 Views
- 4 replies
- 3 kudos
- 3 kudos
Hiyes, I discovered three working possibilities:Define the pandas functions as inline function as pointed out aboveDefine the pandas function in the same script that is imported as "library" in the dlt config ( libraries: - notebook: path: ./pipeline...
- 3 kudos
- 10130 Views
- 2 replies
- 0 kudos
Exporting delta table to one CSV
Process to export a delta table is taking ~2hrs.Delta table has 66 partitions with total size of ~6gb, 4million rows and 270 columns.Used below commanddf.coalesce(1).write.csv("path")what are my options to reduce the time?
- 10130 Views
- 2 replies
- 0 kudos
- 0 kudos
A very interesting task in front of you.... let me know how you solve it!
- 0 kudos
- 15616 Views
- 4 replies
- 0 kudos
Resolved! Explode is giving unexpected results.
I have a dataframe with a schema similar to the following:id: stringarray_field: array element: struct field1: string field2: string array_field2: array element: struct nested_field: stri...
- 15616 Views
- 4 replies
- 0 kudos
- 0 kudos
It turns out that if the exploded fields don't match the schema that was defined when reading the JSON in the first place that all the data that doesn't match is silently dropped. This is not really nice default behaviour.
- 0 kudos
- 7138 Views
- 2 replies
- 2 kudos
Change data feed from target tables of APPLY CHANGES
Up until yesterday I was (sort of) able to read changes from target tables of apply changes operations (either through tables_changes() or using readChangeFeed). I say sort of because the meta columns (_change_type, _commit_version, _commit_timestamp...
- 7138 Views
- 2 replies
- 2 kudos
- 3183 Views
- 1 replies
- 0 kudos
DLT concurrent pipeline updates.
Hi!Regarding this info "An Azure Databricks workspace is limited to 100 concurrent pipeline updates." (Release 2023.16 - Azure Databricks | Microsoft Learn), what is considered as an update? Changes in pipeline logic or each pipeline run?
- 3183 Views
- 1 replies
- 0 kudos
- 7734 Views
- 1 replies
- 0 kudos
How to resolve the column name in s3 path saved as UUID format
our managed databricks tables stored in s3 as default, while i am reading that s3 path directly i am getting the column value as UUIDeg: column name ID in databricks tablewhile checking the S3 Path, the column name looks like COL- b400af61-9tha-4565-...
- 7734 Views
- 1 replies
- 0 kudos
- 0 kudos
hi @Retired_mod Thank you for you are reply but the issue is i am not able to map ID with COL- b400af61-9tha-4565-89c4-d6ba43f948b7. i useDESCRIBE TABLE EXTENDED table_namea query to get the list of UUID column names. and for real column name fettin...
- 0 kudos
- 3600 Views
- 2 replies
- 1 kudos
How to call a table created with create_table using dlt in a separate notebook?
I created a separate pipeline notebook to generate the table via DLT, and a separate notebook to write the entire output to redshift at the end. The table created via DLT is called spark.read.table("{schema}.{table}").This way, I can import[MATERIALI...
- 3600 Views
- 2 replies
- 1 kudos
- 13211 Views
- 10 replies
- 15 kudos
All-purpose clusters not remembering custom tags
Hi, we have several clusters used with Notebooks, we don't delete them, just start-stop according to the "minutes of inactivity" set.I'm trying to set a custom tag, so I wait until the cluster shuts down, add a tag, check that the tag is among then "...
- 13211 Views
- 10 replies
- 15 kudos
- 15 kudos
@alejandrofm the behavior you're describing, where the custom tag disappears after the cluster restarts, might be related to the cluster configuration or the specific settings of your Databricks environment. To troubleshoot this, ensure that the cust...
- 15 kudos
- 1800 Views
- 0 replies
- 0 kudos
Request for reattempt voucher. Databricks Certified Data Engineer Associate exam
HiOn Nov 29th ,I attempted the Databricks Certified Data Engineer Associate exam for 1st time , unfortunately I ended up by failing grade. My passing grade was 70%, and I received 64.00%.I am planning to reattempt the exam, Could you kindly give me a...
- 1800 Views
- 0 replies
- 0 kudos
- 1954 Views
- 0 replies
- 0 kudos
Flattening a Nested Recursive JSON Structure into a Struct List
This is from Spark Event log on Event SparkListenerSQLExecutionStart.How to flatten the sparkPlanInfo struct into an array of the same struct, then later explode it. Note that the element children is an array containing the parent struct, and the lev...
- 1954 Views
- 0 replies
- 0 kudos
- 4111 Views
- 4 replies
- 1 kudos
Resolved! "Your workspace is hosted on infrastructure that cannot support serverless compute."
Hello,I wanted to try out Lakehouse Monitoring, but I receive the following message during setup: "Your workspace is hosted on infrastructure that cannot support serverless compute."I meet all requirements outlined in the documentation. My workspace ...
- 4111 Views
- 4 replies
- 1 kudos
- 1 kudos
Lakehouse MonitoringThis feature is in Public Preview in the following regions: eu-central-1, eu-west-1, us-east-1, us-east-2, us-west-2, ap-southeast-2. Not all workspaces in the regions listed are supported. If you see the error “Your workspace is ...
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
