Databricks Community

guangyi

All my questions is around this code block@Dlt.append_flow(target=”target_table”): def flow_01(): df = spark.readStream.table(“table_01”) @dlt.append_flow(target=”target_table”): def flow_02(): df = spark.readStream.table(“table_02”)The first qu...

guangyi · 11-14-2024

Here is my code for converting one column field of a data frame to time data type: col_value = df.select(df.columns[0]).first()[0] start_time = time.time() col_value = datetime.strftime(col_value, "%Y-%m-%d %H:%M:%S") \ if isinstance(co...

guangyi · 10-22-2024

I noticed there is a numUpdateRetryAttempts property mentioned in the the document https://learn.microsoft.com/en-us/azure/databricks/delta-live-tables/properties used for configing the retry times of any DLT pipeline, but I cannot find it in the DL...

guangyi · 09-11-2024

Here is how I define the UDF inside the file udf_define.py:from pyspark.sql.functions import length, udf from pyspark.sql.types import IntegerType from pyspark.sql import SparkSession spark = SparkSession.builder.getOrCreate() def strlen(s): ret...

guangyi · 09-08-2024

I try to follow the instructions of Monitor Delta Live Tables pipelines to query dlt.expection log.Here is the simple code version I copied from the Querying the event log section:CREATE TEMPORARY LIVE VIEW event_log_raw AS (SELECT * FROM event_log(T...

guangyi

Thank you for your advice. I found some learning material about how to do performance profiing in Python. The cProfile get correct result for me.

guangyi · 10-15-2024

Congratulations! You helps me a lot!

guangyi · 09-12-2024

And I tried getActiveSession() it is not working

guangyi · 09-12-2024

Hi @szymon_dybczak , Thank you for your advice, actually it is works. I find the correct document by following your code. The code I show above is the scalar function version which is not what I wanted.

guangyi · 09-09-2024

H @szymon_dybczak , thank you for the advice. After investigating the problem has been solved. The error message mentioned in the title is not the key, the below error message “A pipeline with a different id is already registered in this Credential S...

Databricks Community

User Stats

User Activity

DLT pipeline observability questions (and maybe suggestions)

What is the correct way to measure the performance of a Databrick notebook?

Has the numUpdateRetryAttempts property been deprecated?

Unable to call UDF inside the Spark SQL: RuntimeError: SparkSession should be create

Error when query event log: Cannot read more than one event logs in the same query

Re: What is the correct way to measure the performance of a Databrick notebook?

Re: Databricks Community Champion - September 2024 - Szymon Dybczak

Re: Unable to call UDF inside the Spark SQL: RuntimeError: SparkSession should be create

Re: Unable to call UDF inside the Spark SQL: RuntimeError: SparkSession should be create

Re: Error when query event log: Cannot read more than one event logs in the same query