I am in a situation where I have a notebook that runs in a pipeline that creates a "live streaming table". So, I cannot use a language other than sql in the pipeline. I would like to format a certain column in the pipeline using a scala code (it's a complicated formatting and difficult to replicate in SQL).
Spark allows you to register scala methods as udf and access those registered methods in SQL.
But given my current situation (pipeline with DLT), I cannot include the scala method and the statement to register the method in spark context in the notebook.
Is there any work around here?
afaik you can create python udf's, but somehow I do not find the docs anymore.
But they seem to be removed. If someone knows where to find these...
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!