I have 2 questions about Lakebridge solution,
- Synapse with dedicated pool Conversion
- We were conducting a PoC for Synapse to DBX migration using Lakebridge. What we have observed is that the conversions are not correct. I was anticipating all tables will be converted fine at least but that is not the case. See below source and converted outputs. For complex stored procedures to notebooks I expected manual conversion but you guys can correct me.
Synapse:
CREATE TABLE [extra1].[dimension_City2] (
[City Key] INT NOT NULL,
[WWI City ID] INT NULL,
[City] NVARCHAR (255) NULL,
[Latest Recorded Population] BIGINT NULL,
[Valid From] DATETIME2 (7) NULL,
[Valid To] DATETIME2 (7) NULL
)
WITH (CLUSTERED COLUMNSTORE INDEX, DISTRIBUTION = HASH([City Key]));
GO
CREATE OR REPLACE TABLE `extra1`.`dimension_City2` (
`City Key` INT NOT NULL,
`WWI City ID` INT,
`City` STRING,
`Latest Recorded Population` BIGINT,
`Valid From` TIMESTAMP ,
`Valid To` TIMESTAMP
)
WITH(CLUSTERED COLUMNSTORE INDEX, DISTRIBUTION = HASH(`City Key`));
- Note: for source_dialect we have tried both mssql and synapse. Sharing one of the config generated post following the steps.
catalog_name: remorph
error_file_path: ./another_try/errors.log
input_source: ./mssql/DatabaseProjectsqlpool-dwh
output_folder:./another_try/output
schema_name: transpiler
skip_validation: true
source_dialect: mssql
transpiler_config_path: ./.databricks/labs/remorph-transpilers/bladebridge/lib/config.yml
transpiler_options:
overrides-file: Bladebridge
version: 3
- Custom Transpiler: I was thinking of building a custom transpiler setup. Can you share where I can find documentation on how to start building with it and what I need to take into account.
Documentation referred from official link
