- 5002 Views
- 2 replies
- 1 kudos
Nulls in Merge
If you are going to handle any null values in your MERGE condition, better watch out for your syntax #databricks
- 5002 Views
- 2 replies
- 1 kudos
- 2560 Views
- 3 replies
- 0 kudos
Ingest __databricks_internal catalog - PERMISSION DENIED
Good morning, I have a DLT process with CDC incremental load and I need to ingest the history as CDC transactions are only recent. To do this I need to ingest data in the __databricks_internal catalog. In my case, as I am full admin, I can do it, how...
- 2560 Views
- 3 replies
- 0 kudos
- 0 kudos
The tables do not have specific configuration, and the user who is receiving the error is in a group that has full permission in the INTERNAL catalog
- 0 kudos
- 1301 Views
- 2 replies
- 0 kudos
How can I throw an exception when a .json.gz file has multiple roots?
I have a situation where source files in .json.gz sometimes arrive with invalid syntax containing multiple roots separated by empty braces []. How can I detect this and thrown an exception? Currently the code runs and picks up only record set 1, and ...
- 1301 Views
- 2 replies
- 0 kudos
- 1177 Views
- 1 replies
- 0 kudos
Multiple Tables Migration from one workspace to another.
Hi all!I need to copy multiple tables from one workspace to another with metadata information. Is there any way to do it?Please reply as soon as possible.
- 1177 Views
- 1 replies
- 0 kudos
- 0 kudos
@HaripriyaP - Depends on your use case, Either of the below approach can be chosen. 1) DELTA CLONE(DEEP CLONE) to clone them to the new workspace. 2) Have the same cluster policy/Instance profile of the old workspace to access them in the new worksp...
- 0 kudos
- 2435 Views
- 3 replies
- 0 kudos
Delta Live Tables Materialised View Column Comment Error
While materialised view doc says MVs support columns comments, this does not seem like the case for MVs created by DLT. For example, when trying to add a comment to a MV created by DLT, it errors:Any ideas on when this will be fixed/supported?
- 2435 Views
- 3 replies
- 0 kudos
- 0 kudos
Just to close the loop here that it seems DLT generated MVs now support column comments.
- 0 kudos
- 931 Views
- 1 replies
- 0 kudos
Installing dlt causing error
I'm trying to use the example in big book of engineering 2nd edition-final.pdf and I had an issue with the statementimport dltSo I created another cell and installed it and I noticed I was getting this error:"dataclass_transform() got an unexpected k...
- 931 Views
- 1 replies
- 0 kudos
- 0 kudos
could you get the full error stack trace please
- 0 kudos
- 2405 Views
- 1 replies
- 0 kudos
Update DeltaTable on column type ArrayType(): add element to array
Hi all,I need to perform an Update on a Delta Table adding elements to a column of ArrayType(StringType()) which is initialized empty.Before UpdateCol_1 StringType()Col_2 StringType()Col_3 ArrayType()ValVal[ ]After UpdateCol_1 StringType()Col_2 Strin...
- 2405 Views
- 1 replies
- 0 kudos
- 4921 Views
- 8 replies
- 73 kudos
Trying to claim reward points but its not reflecting my points
Hi Team,Can anyone help me why my reward point still showing 0 balance. My databricks community points is not reflecting on reward claim portal.i was login first time . Also I was wait for 3 business days but its still not reflecting
- 4921 Views
- 8 replies
- 73 kudos
- 1434 Views
- 0 replies
- 0 kudos
Ephemeral storage how to create/mount.
Hi,I'm looking for information how to create/mount ephemeral storage to Databricks driver node in Azure Cloud. Does anyone have any experience working with ephemeral storage?Thanks,
- 1434 Views
- 0 replies
- 0 kudos
- 2150 Views
- 2 replies
- 3 kudos
DataBricks Rust client and/or OpenAPI spec
Hi,I'm looking for a DataBricks client for Rust. I could only find these SDK implementations.Alternatively, I would be very happy with the OpenAPI spec. Clearly one exists: the Go SDK implementation contains code to generate itself from such a spec...
- 2150 Views
- 2 replies
- 3 kudos
- 3 kudos
Databricks REST API referenceThis reference contains information about the Databricks application programming interfaces (APIs). Each API reference page is presented primarily from a representational state transfer (REST) perspective. Databricks REST...
- 3 kudos
- 1462 Views
- 1 replies
- 0 kudos
Unable to create a record_id column via DLT - Autoloader
Hi Community,I'm trying to load data from the landing zone to the bronze layer via DLT- Autoloader, I want to add a column record_id to the bronze table while I fetch my data. I'm also using file arrival trigger in the workflow to update my table inc...
- 1462 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @Retired_mod - could you or any body from the community team help me here, please? I've been stuck since quite some time now.
- 0 kudos
- 21061 Views
- 3 replies
- 2 kudos
Resolved! Flatten Deep Nested Struct
Hi All,I have a deeply nested spark dataframe struct something similar to below |-- id: integer (nullable = true) |-- lower: struct (nullable = true) | |-- field_a: integer (nullable = true) | |-- upper: struct (containsNull = true) | | ...
- 21061 Views
- 3 replies
- 2 kudos
- 2 kudos
@Aidonis You can try this as well:flatten-spark-dataframe · PyPIThis also allows for specific level of flattening.
- 2 kudos
- 1398 Views
- 1 replies
- 0 kudos
Passing Parameters from Azure Synapse
Hey Community!Just curious if anyone has tried using Azure Synapse for orchestration and passing parameters from Synapse to a Databricks Notebook. My team is testing out Databricks, and I'm replacing Synapse Notebooks with Databricks Notebooks, but I...
- 1398 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @SPres You can definitely pass these parameters to databricks notebook also.Please refer below docs - Run a Databricks Notebook with the activity - Azure Data Factory | Microsoft Learn
- 0 kudos
- 840 Views
- 0 replies
- 0 kudos
Databricks Model Registry Notification
Hi community,Currently, I am training models on databricks cluster and use mlflow to log and register models. My goal is to send notification to me when a new version of registered model happens (if the new run achieves some model performance baselin...
- 840 Views
- 0 replies
- 0 kudos
- 2174 Views
- 2 replies
- 0 kudos
Databricks connecting SQL Azure DW - Confused between Polybase and Copy Into
I see two articles on databricks documentationshttps://docs.databricks.com/en/archive/azure/synapse-polybase.html#language-pythonhttps://docs.databricks.com/en/connect/external-systems/synapse-analytics.html#service-principal Polybase one is legacy o...
- 2174 Views
- 2 replies
- 0 kudos
| User | Count |
|---|---|
| 1611 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
