Databricks Community

RiyazAli

Hi there,As a compliance measure, I'm tasked with masking the PII data starting from bronze to silver and all the tables and views downstream. I suggested my clients to use row filters and column masks as mentioned in the doc.However, when a user who...

RiyazAli · 04-10-2024

Hi Community,I'm trying to load data from the landing zone to the bronze layer via DLT- Autoloader, I want to add a column record_id to the bronze table while I fetch my data. I'm also using file arrival trigger in the workflow to update my table inc...

RiyazAli · 11-09-2022

Hello all,I've been tasked to convert a Scala Spark code to PySpark code with minimal changes (kinda literal translation).I've come across some code that claims to be a list comprehension. Look below for code snippet:%scala val desiredColumn = Seq("f...

RiyazAli · 07-03-2022

The very first notebook of the dbc notebook which is a setup cell fails.

RiyazAli · 06-21-2022

HI there,I'm following the course mentioned from Databricks Academy. I downloaded the .dbc archiive and working along side the videos from academy. In ML-08 - Hyperopt notebook, I see the following error in cmd 13. best_hyperparam = fmin(fn=objectiv...

RiyazAli

Hi @Shivaprasad If you think the above response answered your question, please mark it as a solution. Best,

RiyazAli

Hello @Shivaprasad Expanding on the code snippet provided above. Once you run the Rest Api code provided above, you will get the statement id. Use this statement_id to get the query results using the same statements api.code snippet is as below:impor...

RiyazAli

Hello @Shivaprasad Looks like the error is associated with the way you're trying to retrieve the data from the Unity Storage location.As an alternative, let's try to use the 3 level namespace UC has to submit a query and retrieve the data.e.g: instea...

RiyazAli

Hi @theron If you have enabled LC while table creation, you must have already specified the cluster column. Thus I don't see a reason to mention .clusterBy(key) again.Let me know if any questionsCheers!If you want to create a brand new table with LC...

RiyazAli

Hi @David_Billa Before ingesting the csv data into the delta table, you could create delta table using the table properties as shown below:CREATE TABLE catalog_name.schema_name.table_name (`a` string, `b c` string) TBLPROPERTIES ( 'delta.minReade...

Databricks Community

User Stats

User Activity

Requirement to remove/skip column(s) in the downstream tables/views while PII data masking

Unable to create a record_id column via DLT - Autoloader

Converting a transformation written in Spark Scala to PySpark

Issue with .dbc in the Advanced Data Engineering course in Databricks Academy

Errors in notebooks of Scalable Machine Learning with Apache Spark course in Databricks academy.

Re: Accessing delta tables using API outside azure (Workiva)

Re: Accessing delta tables using API outside azure (Workiva)

Re: Accessing delta tables using API outside azure (Workiva)

Re: Liquid Clustering - Implementing with Spark Streaming’s foreachBatch Upsert

Re: Create delta table with space in field names per source CSV file