Hi All,
I have an issue. I have several databases with the same schemas I need to source data from. Those databases are going to end up aggregated in a data warehouse. The problem is the id column in each means different things. Example: a client id in one database e.g. clientid 1 is not the same client in the other databases. With Data Factory, I can add the source database as a column. I want to move away from Data Factory for this project. I think now Databricks may be more cost effective as a total solution of ETL and data warehouse.
I have created a connection to the source databases under External Data and have an associated schema in the catalog. This looks like a simple solution in itself, but I can't see as I copy data into other tables how I would identify which connection, schema the data is being sourced from using SQL. Perhaps this is a python job.
Any recommendations appreciated
