topic Re: UDFs for working with date ranges in Data Engineering

UDFs for working with date ranges

hobrob — Sun, 23 Nov 2025 18:03:09 GMT

Hi bricklayers,

Originally from a Teradata background and relatively new to Databricks, I was in need of brushing up on my Python and Github CI/CD skills so I’ve spun up a repo for a project I’m calling Terabricks.

The aim is to provide a space for making things that are useful in Teradata but don’t yet exist in bricks, available in bricks. Starting with a port of Teradata’s temporal functions and operators using Databricks UC UDFs written in Python.

These functions are designed to make your life easier if you’re working with type-2, 4 and 6 SCD data structures, or any scenario where you have periods of time that are bookended by a start and end date. For example, you can easily compare such periods for overlap, chronological order, and contiguousness.

If that sounds like it might be of use to you, please do head on over to the repo to find out more and let me know what you think. What awaits is an initial MVP with room for improvement - it needs some work to enhance scalability for example - and things like that can be prioritised if there is demand. If there are any other recovering Teradata-holics out there with ideas for other stuff they would like to see on bricks then I would love to hear it, and as a relative novice to CI/CD workflows, any contributions or advice in this space are also welcome.

Happy hunting!

Rob

Re: UDFs for working with date ranges

bianca_unifeye — Mon, 24 Nov 2025 14:11:07 GMT

Hi @hobrob ,

This is such a cool idea and honestly exactly the kind of thing that helps people crossing the bridge from Teradata to Databricks without losing their sanity on temporal logic every time 😄

Those type 2/4/6 SCD and period-comparison helpers are pain points in almost every migration, so having a reusable, UC-backed set of functions is genuinely valuable, especially for teams who are still thinking in Teradata patterns but want to “land well” on the Lakehouse.

A few thoughts / ideas from my side:

It would be great to see a couple of end-to-end examples:
- classic SCD2 customer dimension
- overlapping policy / tariff periods
- “find gaps” / “find overlaps” style queries using your UDFs
Long-term, this could easily become a mini “Teradata to Databricks” toolkit:
- temporal helpers (what you’ve started)
- compatibility functions/operators people miss most
- maybe even some “Teradata to Spark SQL” patterns documented.
On the CI/CD side, even a simple GitHub Actions pipeline that runs unit tests against a small Delta test dataset would be a great reference for others starting out.

Keep up with the great work!

Re: UDFs for working with date ranges

Raman_Unifeye — Mon, 24 Nov 2025 14:32:36 GMT

Fantastic Initiative @hobrob.

I have used Teradata for good 5+ years but pre-2014/5. So I will be closely following it and very happy to contribute to it. Thanks.