- 3919 Views
- 2 replies
- 0 kudos
Row-level Concurrency and Liquid Clustering compatibility
The documentation is a little ambiguous:"Row-level concurrency is only supported on tables without partitioning, which includes tables with liquid clustering."https://docs.databricks.com/en/release-notes/runtime/14.2.html Tables with liquid clusterin...
- 3919 Views
- 2 replies
- 0 kudos
- 0 kudos
Cluster-on-write is something being worked on. The limitations at the moment have to do with accommodating streaming workloads.I found the following informative:https://www.youtube.com/watch?v=5t6wX28JC_M
- 0 kudos
- 3367 Views
- 1 replies
- 1 kudos
IDENTIFIER clause
Hi all. Just trying to implement adb sql scripts using identifier clause but I have errors like that using an example: DECLARE mytab = 'tab1'; CREATE TABLE IDENTIFIER(mytab) (c1 INT); [UNSUPPORTED_FEATURE.TEMP_VARIABLE_ON_DBSQL] The feature is not...
- 3367 Views
- 1 replies
- 1 kudos
- 1 kudos
@RobsonNLPT - Engineering is still working on the feature that allow DECLARE statements in DBSQL. This is with a tentative ETA of Feb 20 available on preview channel.
- 1 kudos
- 10890 Views
- 4 replies
- 0 kudos
DTL - Delta Live Tables & Handling De-Duplication from Source Data
Hello,Could anyone please help regarding the scenario below?Scenario• I'm using the DLT SQL Language• Parquet files are landed each day from a source system.• Each day, the data contains the 7 previous days of data. The source system can have very la...
- 10890 Views
- 4 replies
- 0 kudos
- 0 kudos
Yes, it is available in DLT. Check this document: https://docs.databricks.com/en/delta-live-tables/cdc.html
- 0 kudos
- 3918 Views
- 2 replies
- 1 kudos
Snowflake connector
Hi Team, Databricks recommends storing data in a cloud storage location, but if we directly connect to Snowflake using the Snowflake connector, will we face any performance issues?Could you please suggest the best way to read a large volume of data f...
- 3918 Views
- 2 replies
- 1 kudos
- 2007 Views
- 1 replies
- 1 kudos
Calling a .py Function using DF from another file
I have created a file NBF_TextTranslationspark = SparkSession.builder.getOrCreate() df_TextTranslation = spark.read.format('delta').load(textTranslation_path) def getMediumText(TextID, PlantName): df1 = spark.sql("SELECT TextID, PlantName, Langu...
- 2007 Views
- 1 replies
- 1 kudos
- 1 kudos
You should create a udf on top of getMediumText function and then use the udf in the sql statement.
- 1 kudos
- 13507 Views
- 4 replies
- 2 kudos
Persisting and managing tables and table schemas in Unity Catalog
Hello Databricks Community,we are currently looking for a way to persist and manage our unity catalog tables in an IaC manner. That is, we want to trace any changes to a table's schema and properties and ideally be able to roll back those changes sea...
- 13507 Views
- 4 replies
- 2 kudos
- 2 kudos
As you mentioned, using notebooks with Data Definition Language (DDL) scripts is a viable option. You can create notebooks that contain the table creation scripts and version control these notebooks along with your application code.
- 2 kudos
- 1512 Views
- 1 replies
- 0 kudos
Handling files used more than once in a streaming pipeline
I am implementing Structured Streaming using Delta Live Table. I want to delete the parquet files once they are used. What options should I set so that the files loaded in S3 are not deleted?
- 1512 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, It sounds like your Structured Streaming source is S3, in which case the easiest solution is likely to manage the stream source using an S3 Lifecycle Configuration (https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html)...
- 0 kudos
- 2446 Views
- 0 replies
- 0 kudos
Unable to Install Cluster-Scoped Libraries on Runtime >13.3
Hello team,I'm trying to upgrade our databricks runtime to 13.3 from 9.1, but i've been having issues installing libraries on the compute from our internal artifactoryHowever, when I tried this on a unity-catalog enabled workspace, it works seamless...
- 2446 Views
- 0 replies
- 0 kudos
- 11323 Views
- 7 replies
- 2 kudos
Run Task as Service Principal with Code in Azure DevOps Repo
Hi allI have a task of type Notebook, source is Git (Azure DevOps). This task runs fine with my user, but if I change the Owner to a service principal, I get the following error:Run result unavailable: run failed with error message Failed to checkout...
- 11323 Views
- 7 replies
- 2 kudos
- 2 kudos
@pgruetter​ :To enable a service principal to access a specific Azure DevOps repository, you need to grant it the necessary permissions at both the organization and repository levels.Here are the steps to grant the service principal the necessary per...
- 2 kudos
- 3030 Views
- 2 replies
- 1 kudos
how to i able to read column mapping metadata for delta tables
i want to read column mapping metadatahttps://github.com/delta-io/delta/blob/master/PROTOCOL.md#column-mappingin above link we can able to find the code block with json data. the same data i want to read in pyspark.. is there any option to read that ...
- 3030 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi,Information about the delta table such as history information could be found by running a `describe history table_name`. A `rename column` operation could be found in the `operation` column with a value of `RENAME COLUMN`. If you then look at the ...
- 1 kudos
- 5384 Views
- 2 replies
- 0 kudos
Resolved! How to fix (AWS SSO) Test Connection failed
What did I configure incorrectly? About SSO settings in my Databricks account. What troubleshooting should I do? I don't see any error message.I follow step instruction from AWS View step-by-step instructions. link belowView step-by-step instructions...
- 5384 Views
- 2 replies
- 0 kudos
- 0 kudos
I got solution I forgot to create SaaS users with the same subject as the AD users.Preprovisioned usersPreprovisioned users, means users must already exist in the downstream SaaS application. For instance, you may need to create SaaS users with the s...
- 0 kudos
- 3516 Views
- 0 replies
- 0 kudos
Delta Live Tables are dropped when pipeline is deleted
I created simplistic DLT pipeline that create one table. When I delete the pipeline the tables is dropped as well. That's not really desired behavior. Since I remember there was a strong distinction between data (stored in tables) and processing (spa...
- 3516 Views
- 0 replies
- 0 kudos
- 3280 Views
- 1 replies
- 1 kudos
UC Volume access for spark and other config files
Hi All,Wondering if anyone else getting this problem:We trying to host krb5.conf and jaas.conf for our compute to be able to connect to Kerberised JDBC sources, we attempting to store these files in Catalog volumes, but at run time when initiating th...
- 3280 Views
- 1 replies
- 1 kudos
- 1 kudos
Haven't been able to access volume path when using jdbc format.
- 1 kudos
- 1917 Views
- 1 replies
- 0 kudos
Deltalke performance
HiI am new to databricks and i am trying to understand the use case of deta lakehouse. Is it good idea to build datawarehouse using deltalake architecture. Is it going to give same performance as that of RDBMS clouse datawarehous like snowflake? Whic...
- 1917 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Sas , One of the benefits of the Data Lakehouse architecture, is that it combines the best of both Data Warehouses and Data Lakes all on one unified platform to help you reduce costs and deliver on your data and AI initiatives faster. It brings t...
- 0 kudos
- 4745 Views
- 1 replies
- 0 kudos
Resolved! Installing a private pypi package from Gitlab on a cluster
I have published a pypi package in a private Gitlab repository and I want to install it in my notebook but I don't know how and the documentation doesn't help me much either. I have created a Gitlab token that I use in the index url and I try to inst...
- 4745 Views
- 1 replies
- 0 kudos
- 0 kudos
This problem was solved by removing the `python>=3.11` requirement.
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
