Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
So there are functions, widgets, or just combining Python with SQL. Probably I would choose SQL functions as they are permanent and stay in metastore (but can not be used for everything).
CREATE OR REPLACE FUNCTION my_name(name STRING COMMENT 'my name')
RETURNS STRING
COMMENT 'just first name'
CONTAINS SQL DETERMINISTIC
RETURN name || 'Hubert'
As you have mentioned that it can not be used for everything , in my case also it doesn't suit as I have a lot variables declaration and having a function created for each variable doesn't look good.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.