Databricks Community

SRK · 01-23-2023

I have a requirement, where I need to apply inverse DQ rule on a table to track the invalid data. For which I can use the following approach:import dltrules = {}quarantine_rules = {}rules["valid_website"] = "(Website IS NOT NULL)"rules["valid_locatio...

SRK · 12-21-2022

We are using Databricks SQL Alerts to handle one scenario. We have written the queries for the same, also we have created the SQL Alert. However, I was looking for the best way to deploy it on Higher Environments like Pre-Production and Production.I ...

SRK · 12-21-2022

I am trying to read messages from kafka topic using spark.readstream, I am using the following code to read it.My CODE:df = spark.readStream .format("kafka") .option("kafka.bootstrap.servers", "192.1xx.1.1xx:9xx") .option("subscr...

SRK · 12-08-2022

In this blog I can see for dimension and fact tables, the primary key constraint has been applied. Following is the example:-- Store dimensionCREATE OR REPLACE TABLE dim_store( store_id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, business_key ...

SRK · 12-01-2022

I met with an issue when I was trying to use autoloader to read json files from Azure ADLS Gen2. I am getting this issue for specific files only. I checked the file are good and not corrupted.Following is the issue:java.lang.IllegalArgumentException:...

SRK · 01-25-2023

You can use this to get the time zone of your choosing:

SRK · 01-24-2023

Hi TH,If you look at the code which I have shared, there I am using append to write the data in Json first then I read the Json file using autoloader. df_table.write.mode("append").json("/mnt/temp_table/ Employee ",ignoreNullFields=False)So, it's onl...

SRK · 01-23-2023

Thanks for the reply. I will check, how this helps in my case.

SRK · 01-23-2023

Thanks for the reply. I will check if that helps.

SRK · 01-23-2023

Option 1:You can use Terraform. with Azure Devops to automate the deployments:https://www.databricks.com/blog/2022/12/5/databricks-workflows-through-terraform.htmlOption 2:You can use Databricks to automate the deployments: Jobs CLI | Databricks on A...

Databricks Community

User Stats

User Activity

Delta Live Tables data quality rules application.

How to deploy Databricks SQL queries and SQL Alerts from lower environment to higher environment?

How to get the count of dataframe rows when reading through spark.readstream using batch jobs?

How to apply Primary Key constraint in Delta Live Table?

I met with an issue when I was trying to use autoloader to read json files from Azure ADLS Gen2. I am getting this issue for specific files only. I checked the file are good and not corrupted.

Re: How to convert string to datetime with correct timezone?

Re: How to activate ignoreChanges in Delta Live Table read_stream ?

Re: Delta Live Tables data quality rules application.

Re: Delta Live Tables data quality rules application.

Re: Move whole workflow from Dev to Prod