Context

We are in the process of extracting data between SAP BDC Datasphere and Databricks (Brownfield Implementation).

• SAP Datasphere is hosted in AWS (eu10)
• Databricks is hosted in Azure (West Europe)
• The BDC Connect System is located in the same region as SAP Datasphere (eu10).

So far, we’ve successfully:

• Set up the BDC Connect connection from SAP Datasphere to Databricks.
• Shared data from SAP to Databricks, which is now available in our catalog.
• Read data from these delta shares in Databricks notebooks.

✅Extracting and reading SAP data in Databricks works fine (even though it is slow, which would be a question for another discussion).

The Problem

When trying to exchange data from Databricks back to SAP, things break down. We followed the official Databricks Delta Sharing Instructions, and everything worked until the last part of Step 6:

"Use Delta Sharing to share and receive data between Databricks and SAP BDC"

Specifically, the link: "Grant SAP Business Data Cloud (BDC) recipients access to Delta Sharing data shares."

Here's what we did:

• Created the share.
• Shared it with the BDC Connect connection we had set up earlier.
• Used notebook to try to describe the share in Core Schema Notation so SAP BDC users could read it.

Notebook snippet (with variables filled in):

bdc_connect_client = BdcConnectClient(DatabricksClient(dbutils, "bdc-connect-test"))

share_name = "test_share"

open_resource_discovery_information = {
    "@openResourceDiscoveryV1": {
        "title": "Title to Share",
        "shortDescription": "Share Short Description",
        "description": "Share Description"
    }
}

catalog_response = bdc_connect_client.create_or_update_share(
    share_name,
    open_resource_discovery_information
)

Error returned:

ValueError: Error when trying to obtain 'access_token' from JSON response: {'error_code': 'INTERNAL_ERROR', 'message': 'Failed to retrieve partner token: INTERNAL: INTERNAL_ERROR: Unable to get user object using principal context with error invalid_client. Please try again or contact support if the issue persists.', 'details': [{'@type': 'type.googleapis.com/google.rpc.RequestInfo', 'request_id': 'c81582e9-e3f0-9d59-8ee3-7c3bbb463b92', 'serving_data': ''}]}

What we've tried so far