topic Re: What's the best way to manage multiple versions of the same datasets? in Data Engineering

What's the best way to manage multiple versions of the same datasets?

Kyle — Tue, 15 Feb 2022 21:57:15 GMT

We have use cases that require multiple versions of the same datasets to be available. For example, we have a knowledge graph made of entities of relations, and we have multiple versions of the knowledge graph that's distinguished by schema names right now:

knowledge_graph__1_3 (version 1.3)
    - entities
    - relations
knowledge_graph__1_4 (version 1.4)
    - entities
    - relations
knowledge_graph__2_1 (version 2.1)
    - entities
    - relations

While we can shoehorn the use cases by using names to version datasets, it doesn't feel like an elegant way. I'm also aware of the Delta Lake versioning capabilities, but we would like to keep each version a top-level artifact rather than historical versions as in the Delta Lake case. Any recommendations of best practices?

Re: What's the best way to manage multiple versions of the same datasets?

Anonymous — Wed, 16 Feb 2022 16:23:00 GMT

Howdy, @Kyle Gao. My name is Piper, and I'm a moderator for Databricks. Welcome to the community! Let's give the community some time to respond, and then we'll find an SME if we need to.

Thanks in advance for your patience. 🙂

Re: What's the best way to manage multiple versions of the same datasets?

-werners- — Thu, 17 Feb 2022 07:18:06 GMT

is it an option to add a version number? you did not mention the format in which the data is stored in the end.

Re: What's the best way to manage multiple versions of the same datasets?

Kyle — Thu, 17 Feb 2022 18:49:05 GMT

> is it an option to add a version number?

Where do you suggest the version number to be added? We append the version number to a database name right now, but it doesn't feel very elegant.

The data is stored in delta format, I'm not sure how it's relevant though.

Re: What's the best way to manage multiple versions of the same datasets?

jose_gonzalez — Wed, 02 Mar 2022 02:05:10 GMT

Have you explore Delta's CDF? https://docs.databricks.com/delta/delta-change-data-feed.html

Re: What's the best way to manage multiple versions of the same datasets?

Anonymous — Wed, 27 Apr 2022 16:00:03 GMT

Hey there @Kyle Gao

Hope you are doing well. Thank you for posting your query.

Just wanted to check in if you were able to resolve your issue or do you need more help? We'd love to hear from you.

Cheers!