UDFs for working with date ranges

[hobrob]

Hi bricklayers,

Originally from a Teradata background and relatively new to Databricks, I was in need of brushing up on my Python and Github CI/CD skills so I’ve spun up a repo for a project I’m calling Terabricks.

The aim is to provide a space for making things that are useful in Teradata but don’t yet exist in bricks, available in bricks. Starting with a port of Teradata’s temporal functions and operators using Databricks UC UDFs written in Python.

These functions are designed to make your life easier if you’re working with type-2, 4 and 6 SCD data structures, or any scenario where you have periods of time that are bookended by a start and end date. For example, you can easily compare such periods for overlap, chronological order, and contiguousness.

If that sounds like it might be of use to you, please do head on over to the repo to find out more and let me know what you think. What awaits is an initial MVP with room for improvement - it needs some work to enhance scalability for example - and things like that can be prioritised if there is demand. If there are any other recovering Teradata-holics out there with ideas for other stuff they would like to see on bricks then I would love to hear it, and as a relative novice to CI/CD workflows, any contributions or advice in this space are also welcome.

Happy hunting!

Rob

[bianca_unifeye]

Hi @hobrob ,

This is such a cool idea  and honestly exactly the kind of thing that helps people crossing the bridge from Teradata to Databricks without losing their sanity on temporal logic every time 😄

Those type 2/4/6 SCD and period-comparison helpers are pain points in almost every migration, so having a reusable, UC-backed set of functions is genuinely valuable, especially for teams who are still thinking in Teradata patterns but want to “land well” on the Lakehouse.

A few thoughts / ideas from my side:

• It would be great to see a couple of end-to-end examples:

  • classic SCD2 customer dimension

  • overlapping policy / tariff periods

  • “find gaps” / “find overlaps” style queries using your UDFs

• Long-term, this could easily become a mini “Teradata to Databricks” toolkit:

  • temporal helpers (what you’ve started)

  • compatibility functions/operators people miss most

  • maybe even some “Teradata to Spark SQL” patterns documented.

• On the CI/CD side, even a simple GitHub Actions pipeline that runs unit tests against a small Delta test dataset would be a great reference for others starting out.

Keep up with the great work!

[Raman_Unifeye]

Fantastic Initiative @hobrob.

I have used Teradata for good 5+ years but pre-2014/5. So I will be closely following it and very happy to contribute to it. Thanks.