Delta Sharing from Databricks to SAP BDC fails with invalid_client error

[4Twannie]

Context

We are in the process of extracting data between SAP BDC Datasphere and Databricks (Brownfield Implementation).

• SAP Datasphere is hosted in AWS (eu10)
• Databricks is hosted in Azure (West Europe)
• The BDC Connect System is located in the same region as SAP Datasphere (eu10).

So far, we’ve successfully:

• Set up the BDC Connect connection from SAP Datasphere to Databricks.
• Shared data from SAP to Databricks, which is now available in our catalog.
• Read data from these delta shares in Databricks notebooks.

✅Extracting and reading SAP data in Databricks works fine (even though it is slow, which would be a question for another discussion).

The Problem

When trying to exchange data from Databricks back to SAP, things break down. We followed the official Databricks Delta Sharing Instructions, and everything worked until the last part of Step 6:

"Use Delta Sharing to share and receive data between Databricks and SAP BDC"

Specifically, the link: "Grant SAP Business Data Cloud (BDC) recipients access to Delta Sharing data shares."

Here's what we did:

• Created the share.
• Shared it with the BDC Connect connection we had set up earlier.
• Used notebook to try to describe the share in Core Schema Notation so SAP BDC users could read it.

Notebook snippet (with variables filled in):

bdc_connect_client = BdcConnectClient(DatabricksClient(dbutils, "bdc-connect-test"))

share_name = "test_share"

open_resource_discovery_information = {
    "@openResourceDiscoveryV1": {
        "title": "Title to Share",
        "shortDescription": "Share Short Description",
        "description": "Share Description"
    }
}

catalog_response = bdc_connect_client.create_or_update_share(
    share_name,
    open_resource_discovery_information
)

Error returned:

ValueError: Error when trying to obtain 'access_token' from JSON response: {'error_code': 'INTERNAL_ERROR', 'message': 'Failed to retrieve partner token: INTERNAL: INTERNAL_ERROR: Unable to get user object using principal context with error invalid_client. Please try again or contact support if the issue persists.', 'details': [{'@type': 'type.googleapis.com/google.rpc.RequestInfo', 'request_id': 'c81582e9-e3f0-9d59-8ee3-7c3bbb463b92', 'serving_data': ''}]}

What we've tried so far

• Changed the compute setup: tested both serverless (across multiple environments) and all-purpose compute using Databricks Runtime 17.3 LTS (Apache Spark 4.0.0, Scala 2.13). Side note: across these requests, we also tested multiple versions of the BDC Connect client library (from 1.1.4 down to 1.1.1) on the different computes.
• Updated the description of the share information.
• Re-ran the notebook with different values.

None of these attempts resolved the issue.

Question

Has anyone successfully set up Delta Sharing from Databricks back into SAP BDC Datasphere?

• Is there a specific configuration required for the Core Schema Notation step?
  
• Could this be an authentication/BDC Connect principal context issue?
• Any known workarounds for the invalid_client error when retrieving the token?

Any guidance, experiences, or best practices would be greatly appreciated!

[Abeshek]

This is a common challenge in enterprise SAP Datasphere and Databricks integrations, particularly in brownfield, cross-cloud setups. We’ve seen multiple cases where sharing between SAP and Databricks works as expected, while the reverse path introduces additional complexity related to identity, principal context, and partner trust that is not always evident from the documentation.

In similar SAP BDC–Databricks implementations, the issue has usually been less about Delta Sharing mechanics and more about how authentication and BDC Connect context are established across clouds and runtimes. Once that layer is aligned, the downstream steps tend to fall into place.

If helpful, we’ve supported teams through comparable bidirectional sharing scenarios and can share what has worked in practice.

[anshu_roy]

Thanks for sharing all the details and troubleshooting steps – that’s very helpful context. ​

In the current notebook, open_resource_discovery_information is defined as a Python dict. Before calling create_or_update_share, this structure must be serialized to a JSON string so the BDC Connect client can correctly parse the payload and obtain the access token. Otherwise, the request body is not in the expected format, which surfaces as an invalid_client / “Failed to retrieve partner token” error on the token retrieval path.​

Concretely, you can adjust your code like this:

import json
from bdc_connect_sdk.auth import BdcConnectClient, DatabricksClient

bdc_connect_client = BdcConnectClient(DatabricksClient(dbutils, "bdc-connect-test"))

share_name = "test_share"

open_resource_discovery_information = {
"@openResourceDiscoveryV1": {
"title": "Title to Share",
"shortDescription": "Share Short Description",
"description": "Share Description"
}
}

# Serialize ORD metadata to JSON so the token and payload can be parsed correctly
ord_json = json.dumps(open_resource_discovery_information)

catalog_response = bdc_connect_client.create_or_update_share(
share_name,
ord_json
)

This ensures the ORD metadata follows the expected JSON format used by the BDC Connect SDK and should allow the access token to be parsed correctly in your environment.​

[4Twannie]

Thank you for the support, but after changing the code and putting in the correct variables it is still returning the same error.

ValueError: Error when trying to obtain 'access_token' from JSON response: {'error_code': 'INTERNAL_ERROR', 'message': 'Failed to retrieve partner token: INTERNAL: INTERNAL_ERROR: Unable to get user object using principal context with error invalid_client. Please try again or contact support if the issue persists.', 'details': [{'@type': 'type.googleapis.com/google.rpc.RequestInfo', 'request_id': 'cea24364-363b-47cd-90d9-cfb933be4a42', 'serving_data': ''}]}
File <command-5894062492658599>, line 19
     16 # Serialize ORD metadata to JSON so the token and payload can be parsed correctly
     17 ord_json = json.dumps(open_resource_discovery_information)
---> 19 catalog_response = bdc_connect_client.create_or_update_share(
     20 share_name,
     21 ord_json
     22 )
File <command-5894062492658599>, line 19
     16 # Serialize ORD metadata to JSON so the token and payload can be parsed correctly
     17 ord_json = json.dumps(open_resource_discovery_information)
---> 19 catalog_response = bdc_connect_client.create_or_update_share(
     20 share_name,
     21 ord_json
     22 )
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-5973f1ae-85db-47f6-ab11-a54e8ade95c3/lib/python3.12/site-packages/bdc_connect_sdk/auth/bdc_connect_client.py:51, in BdcConnectClient.create_or_update_share(self, share_name, body)
     50 def create_or_update_share(self, share_name: str, body: str | dict[str, Any]) -> PutShareResponse | None:
---> 51     self._prepare_client_for_request(share_name)
     53     body = _cast_body_string_to_dict(body)
     55     body = self.partner_client.build_create_or_update_share_request_body(body)
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-5973f1ae-85db-47f6-ab11-a54e8ade95c3/lib/python3.12/site-packages/bdc_connect_sdk/auth/bdc_connect_client.py:85, in BdcConnectClient._prepare_client_for_request(self, share_name)
     82     cert_pem, key_pem = CertificateManager().generate_self_signed_certificate()
     83     self.cert_info = CertificateInformation(cert_pem, key_pem)
---> 85 access_token = self.partner_client.get_access_token(self.cert_info, share_name)
     86 bdc_connect_endpoint = self.partner_client.get_bdc_connect_endpoint()
     87 tenant = self.partner_client.get_bdc_connect_tenant()
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-5973f1ae-85db-47f6-ab11-a54e8ade95c3/lib/python3.12/site-packages/bdc_connect_sdk/auth/databricks_client.py:35, in DatabricksClient.get_access_token(self, cert_info, share_name)
     33 def get_access_token(self, cert_info: CertificateInformation, share_name: str) -> str:
     34     if self.is_brownfield_environment:
---> 35         return self._get_access_token_for_bdc_connect(cert_info, share_name)
     37     return self._get_access_token_for_databricks_connect(cert_info, share_name)
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-5973f1ae-85db-47f6-ab11-a54e8ade95c3/lib/python3.12/site-packages/bdc_connect_sdk/auth/databricks_client.py:117, in DatabricksClient._get_access_token_for_bdc_connect(self, cert_info, share_name)
    114     raise ValueError(f"Response is not valid JSON. Status code: {response.status_code}, Content: {content_preview}") from e
    116 if "access_token" not in data:
--> 117     raise ValueError(f"Error when trying to obtain 'access_token' from JSON response: {data}")
    119 access_token = data.get("access_token")
    121 self._store_access_token_information(access_token)

So it looks like it is a problem with the client_id that is sourced from the environment. Now I have tried to go through this by going into the bdc_connect_client, using the following code:

from bdc_connect_sdk.auth import BdcConnectClient
from bdc_connect_sdk.auth import DatabricksClient

databricks_client = DatabricksClient(dbutils, "bdc-connect-test")
bdc_connect_client = BdcConnectClient(databricks_client)

bdc_connect_client.__dict__["partner_client"].__dict__

It is returning the following:

{'bdc_connect_access_token_information': {'client_id': '',
                                          'share_location': '',
                                          'share_url': ''},
 'bdc_connect_endpoint': '',
 'bdc_connect_tenant': '',
 'databricks_api_token': '[REDACTED]',
 'databricks_workspace_url': 'https://westeurope.azuredatabricks.net',
 'dbutils': Package 'dbutils'. For more information, type 'dbutils.help()' in a cell.,
 'is_brownfield_environment': True,
 'recipient_name': 'bdc-connect-test'}

In this we can see that it is identifying the environment correctly as brownfield and the location is also correctly found. Client_id here is likely just a placeholder, so nothing to worry about yet. 

Then after that when looking into the recipient, with this code:

%sql
DESCRIBE RECIPIENT `bdc-connect-test`;

Which errors with: [DELTA_SHARING_INVALID_RECIPIENT_AUTH] Illegal authentication type UNKNOWN for recipient bdc-connect-test. SQLSTATE: 28000

It looks like the setup of the connection between the two platforms has been incorrectly configured, can someone confirm this? If this is correctly configured, I'll have to look into a different direction regarding the cause of the problem.

[anshu_roy]

The error DELTA_SHARING_INVALID_RECIPIENT_AUTH refers to an invalid authorization specification when accessing Delta Sharing resources. This maps to SQLSTATE code 28000 ("invalid authorization specification") and typically occurs when the recipient's authentication with the Delta Sharing service has failed or is misconfigured.

It could be because of - 

• Recipient has not activated their credentials—status may show as "Pending" in the UI until activation is completed.
• There may be an invalid recipient configuration or a missing/incorrect recipient token.

I'd suggest to re‑validate BDC Connect client and “recipient-name” and work the below checks  -

• In SAP BDC, open the Databricks connection (BDC Connect for Databricks) and confirm:
  • The connection status is Connected and uses the same Databricks workspace you are calling from.​
  • The recipient name shown in BDC exactly matches the string you pass into DatabricksClient(dbutils, "<recipient-name>") (case‑sensitive, no extra spaces)
• ​In Databricks, verify that 
  • You are running the notebook in the workspace that is linked to that BDC connection. 

  • The cluster/warehouse has access to the Unity Catalog and shares you intend to publish.

If the error persists, please open a support ticket with SAP team, who can look into the logs and address the authentication issue.