But following excerpt from old version of documentation is interesting:

@databrickseroI wonder if the following workaround could work. I haven’t tested it, and there might be some typos since I wrote it from memory, but I hope you get the idea.

def pandas_function(spark_df):
  pdf = spark_df.toPandas()
  pdf_filtered = pdf[pdf["year"].notnull()]
  return spark.createDataFrame(pdf_filtered )


@dlt.table(name="silver_primary")
def silver_primary():
    df = dlt.read("silver_intermediate")
    df = df.withColumn("year", F.substring(F.col("date_col"), 0, 4))
    df_transformed = pandas_function(df)
    return df_transformed.to_spark()