Databricks Community

thewfhengineer

I'm starting a project to migrate our Compliance model - Python code (Pandas-based) from AWS SageMaker to the Azure ecosystem.Source: AWS (SageMaker, Airflow)Target: Azure (Databricks, ADLS)I'm evaluating the high-level approach and would appreciate ...

thewfhengineer · 09-01-2022

Schema Design :Source : Miltiple CSV Files like (SourceFile1 ,SourceFile2)Target : Delta Table like (Target_Table)Excel File : ETL_Mapping_SheetFile Columns : SourceTable ,SourceColumn, TargetTable, TargetColum , MappingLogicMappingLogic columns cont...

thewfhengineer · 09-20-2022

@Aman Sehgal thanks for your response , I update the sample mapping example I already have mapping logic in the mappingsheet so do I still need this extra function now ... can I directly store this SQL logic in a variable and directly exeute like b...

thewfhengineer · 09-20-2022

Thanks for your response.your understanding is correct.I updated the sample etl mapping in the que.As you can see , this mapping sheet contains sql statement to get target values and I have 500 mappings like this so I was thinking to use this mapping...

thewfhengineer · 09-19-2022

@Aman Sehgal @Hubert Dudek @Piper Wilson @Werner Stinckens Can someone pls check this query ??

Databricks Community

User Stats

User Activity

AWS SageMaker to the Azure Databricks.

How to implement Source to Target ETL Mapping sheet in PySpark using Delta tables

Re: How to implement Source to Target ETL Mapping sheet in PySpark using Delta tables

Re: How to implement Source to Target ETL Mapping sheet in PySpark using Delta tables

Re: How to implement Source to Target ETL Mapping sheet in PySpark using Delta tables