cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Tools to parse Informatica PowerCenter XML to Databrics PySpark ?

Office_Solution
New Contributor III

Hi all,

We're planning a move from legacy PowerCenter to Databricks.

Creating Delta Tables in simple enough, but manually converting dozens of PowerCenter mappings, nested mapplets, & expression logic into PySpark is eating up way too much dev time.

Beffore our team spends weeks opening .xml files to document evverything line by line, has anyonw used a parser to extract informatiob repository logic into baseline PySpark scripts automatically?

How are others speeding up the conversion steps ? Thanks!

1 ACCEPTED SOLUTION

Accepted Solutions

Abhishek20
New Contributor II

We hit this exact same issue moving PowerCenter logic over to Databricks. Hand translating all those nested mapplets and expression transformations into PySpark is a huge drain on time.

We used a tool named Pulse Convert to read our .xml exports and generate the Pyspark scripts automatically. It take care of most of the repetitive mapping logic for us.

If you want to do it in-house your best bet is writing a quick Python script to parse the XML tags and dump the transformation logic into Excel first.

View solution in original post

4 REPLIES 4

data_pulse
Contributor

@Office_Solution 

There are already a few Databricks-supported approaches worth exploring before looking for a custom PowerCenter XML parser.

1. Genie Code + Lakebridge Switch (Beta)

Databricks documents Informatica-to-Lakeflow pipeline migration using Genie Code, check this.

Upload the Informatica project XML to a Unity Catalog Volume or Workspace, create an empty Lakeflow pipeline, and prompt Genie Code:

Migrate the project at /Volumes/catalog/schema/volume/project

It analyses dependencies, constructs an intermediate representation, generates Python/SQL pipeline code and attempts validation/repair.

2. Lakebridge + BladeBridge

There is similar Community post recommending Bladebridge for PowerCenter XML Mapping, session and workflow conversion. Check this post.

3. Alternative: informatica-sparker 

An existing Python package designed to parse Informatica PowerCenter XML and generate PySpark code. Worth evaluating for a quick POC.

pip install informatica-sparker

4. Custom XML-to-PySpark workaround

If existing converters don't fully support complex transformations or nested mapplets, a metadata-driven approach using Python's lxml could be another alternative.

from lxml import etree
tree = etree.parse("mapping.xml")

Hope this helps.

Valeria_Koz_DBX
Databricks Employee
Databricks Employee

Hi @Office_Solution,

Just to add to @data_pulse 

  1. Assess first, convert second. Run the Lakebridge Analyzer on your PowerCenter repository export. It inventories mappings, mapplets, sessions and workflows and scores their complexity, so you don't have to document everything by hand.
  2. Pick the converter for the job.
    • BladeBridge (in Lakebridge) is rule-based, so the same input always gives the same output. It parses PowerCenter mapping and workflow XML and generates PySpark or Spark SQL notebooks, including mapplets and workflow parameters. This is the best fit for bulk conversion.
    • Switch (also in Lakebridge, and the engine behind Genie Code's Migrate the project at โ€ฆ flow, currently Beta) is LLM-based. It has a built-in Informatica prompt that covers Source Qualifier, Expression, Router, Joiner, Lookup, Aggregator, Normalizer, Sequence Generator and Update Strategy, and it outputs Lakeflow pipelines. It's useful for the long tail of mappings and for producing pipelines directly.
    • Docs: https://databrickslabs.github.io/lakebridge/ and https://docs.databricks.com/aws/en/ldp/de-agent
  3. Check the results, don't just check that it runs. Use Lakebridge Reconcile to compare source and target outputs for each converted mapping before you cut over.
  4. A word of caution on third-party PyPI converters such as informatica-sparker. It has no linked source repository or named maintainer, so review the code in a sandbox before installing it anywhere near production credentials or repository exports.

Lakebridge is a Databricks Labs project, provided as-is with no support SLA. For large estates, contact your Databricks account team, they should be able to help you to scope the migration.

Martรญnezluas
New Contributor II

Iโ€™d start by testing informatica-sparker, a Python tool that aims to convert PowerCenter XML exports into baseline PySpark scripts. It could save time extracting mappings and translating common transformations, but Iโ€™d test it on a few real workflows first, especially ones with nested mapplets, lookups, and complex expressions. If it doesnโ€™t cover everything, you can build a Python parser using lxml to extract the mapping logic and generate PySpark for the unsupported transformations. Then run the original PowerCenter workflows and the generated Databricks jobs against the same sample data and compare the results. That way, you can automate most of the repetitive conversion work without trusting generated code blindly.

Abhishek20
New Contributor II

We hit this exact same issue moving PowerCenter logic over to Databricks. Hand translating all those nested mapplets and expression transformations into PySpark is a huge drain on time.

We used a tool named Pulse Convert to read our .xml exports and generate the Pyspark scripts automatically. It take care of most of the repetitive mapping logic for us.

If you want to do it in-house your best bet is writing a quick Python script to parse the XML tags and dump the transformation logic into Excel first.