Here is how I define the UDF inside the file udf_define.py:from pyspark.sql.functions import length, udf
from pyspark.sql.types import IntegerType
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
def strlen(s):
ret...
I try to follow the instructions of Monitor Delta Live Tables pipelines to query dlt.expection log.Here is the simple code version I copied from the Querying the event log section:CREATE TEMPORARY LIVE VIEW event_log_raw AS (SELECT * FROM event_log(T...
I need a DLT pipeline to create a materialized view for fetching event logs. All the ways below I tried are failed:Attach a notebook with pure SQL inside: No magic cell like `%sql` are failedAttach a notebook with `spark.sql` python code: Failed beca...
I know how to validate the column level constraint, like checking whether the specified column value is larger than target value.Can I validate some table level constraints? For example, validate whether the total records count of a table is larger t...
Here is the policy I just created: {
"node_type_id": {
"defaultValue": "Standard_D8s_v3",
"type": "allowlist",
"values": [
"Standard_D8s_v3",
"Standard_D16s_v3"
]
},
"num_workers": {...
Hi @szymon_dybczak , Thank you for your advice, actually it is works. I find the correct document by following your code. The code I show above is the scalar function version which is not what I wanted.
H @szymon_dybczak , thank you for the advice. After investigating the problem has been solved. The error message mentioned in the title is not the key, the below error message “A pipeline with a different id is already registered in this Credential S...
After just finishing my last reply, I realized what’s wrong with my code: I should use “file” property instead of “notebook” in the libraries section.It works now. Thank you guys, you are my rubber duck!
Hi @szymon_dybczak @filipniziol Here is how I implement the 3rd option and how it failed:I create a JSON file with the DLT pipeline definition inside{
"name": "query_data_quality_event_log_pipeline",
"clusters": [
{
"label...
