Lakeflow Connect SchemaParseException: Illegal character

[absan]

Hi,

i'm trying to setup Lakeflow Connect for SQL Server. The created gateway is failing with 

"org.apache.avro.SchemaParseException: Illegal character in: LN.FWH-ID"

Unfortunately, don't have control over the source database to change the column names.

Is there a way around this or should we consider alternatives to Lakeflow Connect?

Thank you

[absan]

Sorry, not sure i understand how this is relevant?

[hippo]

Running into the same issue here. Part of our ingestion pattern is to have the source team replicate into an Azure SQL DB we manage, turn CDC on, and then ingest into Databricks. I tried enabling column mapping multiple times with no success, does anyone know how to address this?

[hippo]

Okay so there are a few options I could find:

1. Create a store procedure on that when creating CDC tables, which creates an intermediary clean table and runs CDC off of that. That table can use triggers to keep data in sync (better for lower volume).

2. If creating pipelines through DABS/API, you can actually change your compute in more detail. Although I have not tested it since doing option #1, you might be able to enable column mapping in the spark_conf section, and run with that.

[absan]

Thanks for your response. I ended up solving this with federation. It's not ideal but given the data volumes and the minimal impact on source database, it works for us for now.