Databricks Community

Hoviedo

Hi, is there any way to apply a expectations only if that column exists? I am creating multiple dlt tables with the same python function so i would like to create diferent expectations based in the table name, currently i only can create expectations...

Hoviedo · 07-12-2024

Hello Community, I wrote a post about Data Management with the objective of linking the theoretical part of books like DAMA-DMBOK with the practical application in modern data platforms (using databricks as an example) link: data is not the new oil T...

Hoviedo · 07-05-2023

Hi, i have a problem with my on-premise sql conection from databricks.i have my python code wich use pymssql and it is working well but my spark code doest not and i am using the same credential for both, my spark code is """# Read data from SQL Serv...

Hoviedo

def get_changes_from_raw(table_name): @Dlt.table( name=f"{table_name}_changes", comment=f"New {table_name} data incrementally ingested from cloud object storage landing zone", ) @Dlt.expect("valid_rescued_data", "_rescued_data ...

Hoviedo

Hi Walter, sorry what would it look like in the code?i can do this in the python function (it does nothing)if "Distance" in df.columns: dlt.expect("Distance is positive", "Distance >= 0") but i am now sure how can i apply the same with the...

Hoviedo · 06-06-2023

I have the same problem, did you find any solution? thanks

Databricks Community

User Stats

User Activity

Apply expectations only if column exists

New Article : Data is not the new oil

load data from sql server : python works well but spark does not

Re: Apply expectations only if column exists

Re: Apply expectations only if column exists

Re: Unable to read data from ElasticSearch using Databricks (AWS) Cannot detect ES version - Caused by: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [IP:PORT]