topic Re: Lakeflow Connect SchemaParseException: Illegal character in Data Engineering

Lakeflow Connect SchemaParseException: Illegal character

absan — Fri, 08 Aug 2025 06:56:03 GMT

Hi,

i'm trying to setup Lakeflow Connect for SQL Server. The created gateway is failing with

"org.apache.avro.SchemaParseException: Illegal character in: LN.FWH-ID"

Unfortunately, don't have control over the source database to change the column names.

Is there a way around this or should we consider alternatives to Lakeflow Connect?

Thank you

Re: Lakeflow Connect SchemaParseException: Illegal character

absan — Fri, 08 Aug 2025 15:27:13 GMT

Sorry, not sure i understand how this is relevant?

Re: Lakeflow Connect SchemaParseException: Illegal character

hippo — Sun, 24 Aug 2025 23:45:46 GMT

Running into the same issue here. Part of our ingestion pattern is to have the source team replicate into an Azure SQL DB we manage, turn CDC on, and then ingest into Databricks. I tried enabling column mapping multiple times with no success, does anyone know how to address this?

Re: Lakeflow Connect SchemaParseException: Illegal character

hippo — Tue, 02 Sep 2025 02:08:12 GMT

Okay so there are a few options I could find:

1. Create a store procedure on that when creating CDC tables, which creates an intermediary clean table and runs CDC off of that. That table can use triggers to keep data in sync (better for lower volume).

2. If creating pipelines through DABS/API, you can actually change your compute in more detail. Although I have not tested it since doing option #1, you might be able to enable column mapping in the spark_conf section, and run with that.

Re: Lakeflow Connect SchemaParseException: Illegal character

absan — Tue, 09 Sep 2025 11:05:18 GMT

Thanks for your response. I ended up solving this with federation. It's not ideal but given the data volumes and the minimal impact on source database, it works for us for now.