Introduction
Data silos affect businesses across all industries, not just Manufacturing, where IT/OT convergence is a game-changer. To address this, Databricks has built a Unified Analytics Platform that brings together all types of data - structured, semi-structured, unstructured. This platform enables organizations to break down data barriers and gain a single, unified view of their operations
At Databricks, we're passionate about empowering data teams to unlock insights from all their data, regardless of where it resides. In this blog, we'll tackle one of the most complex and intimidating types of data out there: SAP. Yes, we're talking about the behemoth of business software, known for its intricate systems and daunting data structures.
As former SAP insiders, we've lived through the challenges of extracting data from SAP systems. We've developed a deep understanding of the complexities and nuances of SAP data extraction, gained through extensive hands-on experience and expertise.
That's why we're excited to share our expertise on the recent changes to SAP's data extraction policy and how it impacts third-party extraction tools. And don't worry, we won't make you learn “SAPanese” (read SAP jargon)! Our goal is to break down the technical barriers and make SAP data extraction accessible to everyone.
Understanding the ODP Framework and SAP Notes Restrictions
SAP has recently implemented stricter controls on data extraction from its applications, particularly affecting third-party extraction tools. This shift has been a topic of significant discussion among customers and partners.
- ODP Framework and its key role in extracting SAP data
- Operational Data Provisioning (ODP) is a framework widely used for ETL processes in SAP environments, including the ERP (be it legacy ECC or the latest S/4HANA), and the SAP Data Warehouse (a.k.a. Business Warehouse - BW). ODP allows access to various data sources and replication tools.
- The ODP framework enables data transfers between SAP systems and SAP as well as non-SAP data targets while preserving business context. The framework involves an "ODP Provider" (the source SAP system, like the Core Data Services or CDS Views) and an "ODP Consumer" (the target application). The ODP provider can act as a data source for OData services, as illustrated in the below schema, allowing REST-based integrations with external consumers.
- ODP is increasingly recognized as the central infrastructure for data extraction and replication in SAP environments. Its ability to efficiently manage delta processes, and supports various consumer systems.
- SAP’s data extractors like SAP Datasphere Replication Flow and Data Services use the ODP Framework via Remote Function Calls (RFC), which allow native access to the ERP Application Layer components.
SAP Layered Stack Architecture: A High-Level View of Database, Application, and Extraction Components (source: « AWS for SAP » Blog Post)
- Restriction on ODP-based Extraction
- A support note (3255746) issued in October 2022 initially stated that third-party applications using the native ODP RFC method were unsupported. This was updated on February 2nd, 2024, changing the term "unsupported" to "unpermitted" and indicating that SAP would implement technical measures to restrict and audit the use of RFC modules for data extraction.
- The July 11th, 2024 update of the same SAP note (3255746) clearly advises the use of SAP Datasphere for "realizing data replication scenarios to move data from various SAP sources (such as SAP S/4HANA, SAP BW, SAP ECC sources, etc.) into third-party applications and tools".
- ABAP Add-ons Certification Changes
- ABAP (Advanced Business Application Programming) is a high-level programming language developed by SAP for building business applications. It is primarily used for developing SAP ERP applications, as well as for customizing and extending SAP systems.
- SAP has announced it will discontinue certifying classic ABAP Add-Ons (Tier 3) from independent partners, including extraction tools. This change affects proprietary tools that previously used SAP certification as an endorsement. However, partners can still provide and support these add-ons; only the certification process is being phased out.
Implications for Customers
- SAP ODP support note impact on popular non-SAP extraction tools
|
System |
Connector |
Is Impacted |
Comments/Notes |
|
Azure |
SAP Table Connector (legacy) |
No |
Does not use ODP so not impacted by the SAP Note. |
|
Azure |
ECC Connector |
No |
Does not use ODP. |
|
Azure |
BW Open Hub Connector |
No |
Does not use ODP. |
|
Azure |
BW MDX Connector |
No |
Does not use ODP. |
|
Azure |
HANA Copy Activity Connector |
No |
Does not use ODP. |
|
Azure |
SAP CDC Connector |
Yes |
Uses ODP RFC API, now unpermitted for third-party tools. |
|
Azure |
ADF OData CDC Connector |
No |
Workaround introduced in March 2024, uses ODP OData API, less performant. |
|
Fivetran |
HVR (LDP) |
No |
Uses database log replication. For HANA database, log replication is not supported by SAP (Note 2971304). |
|
Fivetran |
SAP ERP on HANA Connector |
No |
ABAP add-on, does not use ODP. |
|
Fivetran |
ODP OData Connector |
No |
Uses ODP OData API. Released in preview, is compliant. |
|
Qlik |
Qlik Replicate SAP HANA |
No |
Uses database log replication. For HANA database, log-based replication is not supported by SAP (Note 2971304) and trigger-based replication requires premium HANA licence for ERP. |
|
Qlik |
Qlik Replicate ODP Connector |
Yes |
Uses ODP RFC API, for CDS views/BW Objects/SAP Extractors/HANA Views/SLT, now unpermitted for third-party tools. |
|
Qlik |
Qlik Replicate ODP OData Connector |
No |
Uses ODP OData API. Planned for Q4 2024, will be compliant. |
Impacted connectors are concerned by the SAP ODP Support Note because they leverage the ODP RFC API, which poses a high risk for customers using it since it became unpermitted by SAP. The above list may not be exhaustive regarding all the SAP connectors, especially the ones that are not impacted by the SAP Note. We wanted to include the main SAP connectors for these vendors. For the most up-to-date and comprehensive information, it's advisable to consult directly with specific vendors.
- Operational Concerns
SAP's Datasphere Replication Flow is now the SAP recommended tool for data extraction, but it has limitations, including costs associated with data egress and a cap on concurrent replication threads as specified in the Replication Flow Blog Series Part 4 - Sizing. This limitation has raised concerns for high-volume customers needing to replicate numerous source objects simultaneously.
Regarding an OData-based connection, some customers have reported that the overall performance of OData and the additional setup required on the source side act as significant barriers to adoption.
- Future Considerations with Databricks
The existing partnership between SAP Datasphere and Databricks continues to drive joint exploration of integration enhancements based on customer requests. This collaboration aims to enable seamless data convergence, allowing businesses to leverage SAP's structured data alongside Databricks' analytics and AI capabilities, thereby unlocking greater value from their combined data assets.
Key Takeaways
- SAP recommended tool: SAP Datasphere Replication Flow is now the SAP recommended tool for data extraction.
- ODP Support Note Implications: The SAP ODP support note prohibits third-party tools from using the ODP RFC API, posing significant challenges for non-SAP extraction tools reliant on this API. However, tools that utilize the ODP OData API or non-ODP interfaces remain unaffected by these restrictions.
- Vendor Adaptations: Vendors such as Microsoft (Azure), Fivetran, and Qlik are actively developing workarounds or new connectors to comply with the SAP support note, ensuring continued functionality for their users.
- Performance Considerations: ODP OData-based connectors are compliant with the ODP support note, though are noted to be less performant compared to the ODP RFC interface.
- Future Enhancements: SAP and third-party vendors are actively enhancing their data extraction and integration capabilities, with SAP focusing on improving Datasphere Replication Flow while vendors like Microsoft, Fivetran, and Qlik develop compliant solutions. As the landscape evolves, organizations can expect a wider range of options for efficiently extracting and integrating SAP data, with improvements in performance, scalability, and compliance across both SAP-native and third-party solutions.
Conclusion
SAP's recent policy changes regarding data extraction have significant implications for customers and third-party vendors. By understanding the implications of the SAP ODP support note, organizations can assess the risks and opportunities associated with their non-SAP extraction tools and plan accordingly. Organizations relying on third-party extraction tools must adapt to these new restrictions and consider SAP's own solutions for compliance and support. As organizations navigate these changes, Databricks stands ready to facilitate seamless integration with SAP systems and its Unified Analytics Platform by providing best practices through its team of experts.
Ultimately, as both SAP and Databricks continue to innovate, businesses can look forward to more efficient and effective ways to leverage their data across platforms, breaking down silos and unlocking valuable insights for informed decision-making.
Call to Action
Feel free to reach out to your Databricks account team to engage with our SAP team to discuss the complexities of your SAP applications data integration and explore customer use cases for the SAP Lakehouse powered by Databricks.
1 Comment
