a month ago
Hi all,
We're planning a move from legacy PowerCenter to Databricks.
Creating Delta Tables in simple enough, but manually converting dozens of PowerCenter mappings, nested mapplets, & expression logic into PySpark is eating up way too much dev time.
Beffore our team spends weeks opening .xml files to document evverything line by line, has anyonw used a parser to extract informatiob repository logic into baseline PySpark scripts automatically?
How are others speeding up the conversion steps ? Thanks!
2 weeks ago
We hit this exact same issue moving PowerCenter logic over to Databricks. Hand translating all those nested mapplets and expression transformations into PySpark is a huge drain on time.
We used a tool named Pulse Convert to read our .xml exports and generate the Pyspark scripts automatically. It take care of most of the repetitive mapping logic for us.
If you want to do it in-house your best bet is writing a quick Python script to parse the XML tags and dump the transformation logic into Excel first.
a month ago
There are already a few Databricks-supported approaches worth exploring before looking for a custom PowerCenter XML parser.
1. Genie Code + Lakebridge Switch (Beta)
Databricks documents Informatica-to-Lakeflow pipeline migration using Genie Code, check this.
Upload the Informatica project XML to a Unity Catalog Volume or Workspace, create an empty Lakeflow pipeline, and prompt Genie Code:
Migrate the project at /Volumes/catalog/schema/volume/projectIt analyses dependencies, constructs an intermediate representation, generates Python/SQL pipeline code and attempts validation/repair.
2. Lakebridge + BladeBridge
There is similar Community post recommending Bladebridge for PowerCenter XML Mapping, session and workflow conversion. Check this post.
3. Alternative: informatica-sparker
An existing Python package designed to parse Informatica PowerCenter XML and generate PySpark code. Worth evaluating for a quick POC.
pip install informatica-sparker4. Custom XML-to-PySpark workaround
If existing converters don't fully support complex transformations or nested mapplets, a metadata-driven approach using Python's lxml could be another alternative.
from lxml import etree
tree = etree.parse("mapping.xml")Hope this helps.
a month ago
Hi @Office_Solution,
Just to add to @data_pulse
Lakebridge is a Databricks Labs project, provided as-is with no support SLA. For large estates, contact your Databricks account team, they should be able to help you to scope the migration.
3 weeks ago
Iโd start by testing informatica-sparker, a Python tool that aims to convert PowerCenter XML exports into baseline PySpark scripts. It could save time extracting mappings and translating common transformations, but Iโd test it on a few real workflows first, especially ones with nested mapplets, lookups, and complex expressions. If it doesnโt cover everything, you can build a Python parser using lxml to extract the mapping logic and generate PySpark for the unsupported transformations. Then run the original PowerCenter workflows and the generated Databricks jobs against the same sample data and compare the results. That way, you can automate most of the repetitive conversion work without trusting generated code blindly.
2 weeks ago
We hit this exact same issue moving PowerCenter logic over to Databricks. Hand translating all those nested mapplets and expression transformations into PySpark is a huge drain on time.
We used a tool named Pulse Convert to read our .xml exports and generate the Pyspark scripts automatically. It take care of most of the repetitive mapping logic for us.
If you want to do it in-house your best bet is writing a quick Python script to parse the XML tags and dump the transformation logic into Excel first.