Sunday
yesterday
I couldn't appreciate it more, thank you for your help, below are additional information
Runtime version: 18 LTS
Access mode: Standard
[UC_UNCAUGHT_CLIENT_EXCEPTION] Encountered an unexpected HTTP error while communicating with the Databricks Unity Catalog backend. Error Code: UNEXPECTED_HTTP_ERROR. Message: HTTP request failed with status: HTTP/1.1 403 Forbidden, original HTTP response body: Unauthorized network access to workspace: 4444444444444 SQLSTATE: KCUC4
File <command-8700090828854269>, line 2
----> 2 spark.sql("show catalogs")
File /databricks/spark/python/pyspark/sql/connect/session.py:920, in SparkSession.sql(self, sqlQuery, args, **kwargs)
917 _views.append(SubqueryAlias(df._plan, name))
919 cmd = SQL(sqlQuery, _args, _named_args, _views)
--> 920 data, properties, ei = self.client.execute_command(cmd.command(self._client))
921 if "sql_command_result" in properties:
922 df = DataFrame(CachedRelation(properties["sql_command_result"]), self)
File /databricks/spark/python/pyspark/sql/connect/client/core.py:1618, in SparkConnectClient.execute_command(self, command, observations, extra_request_metadata)
1616 req.user_context.user_id = self._user_id
1617 self._set_command_in_plan(req.plan, command)
-> 1618 data, _, metrics, observed_metrics, properties = self._execute_and_fetch(
1619 req, observations or {}, extra_request_metadata
1620 )
1621 # Create a query execution object.
1622 ei = ExecutionInfo(metrics, observed_metrics)
File /databricks/spark/python/pyspark/sql/connect/client/core.py:2188, in SparkConnectClient._execute_and_fetch(self, req, observations, extra_request_metadata, self_destruct)
2185 properties: Dict[str, Any] = {}
2187 with Progress(handlers=self._progress_handlers, operation_id=req.operation_id) as progress:
-> 2188 for response in self._execute_and_fetch_as_iterator(
2189 req, observations, extra_request_metadata or [], progress=progress
2190 ):
2191 if isinstance(response, StructType):
2192 schema = response
File /databricks/spark/python/pyspark/sql/connect/client/core.py:2164, in SparkConnectClient._execute_and_fetch_as_iterator(self, req, observations, extra_request_metadata, progress)
2162 raise kb
2163 except Exception as error:
-> 2164 self._handle_error(error)
File /databricks/spark/python/pyspark/sql/connect/client/core.py:2537, in SparkConnectClient._handle_error(self, error)
2535 self.thread_local.inside_error_handling = True
2536 if isinstance(error, grpc.RpcError):
-> 2537 self._handle_rpc_error(error)
2538 raise error
2539 finally:
File /databricks/spark/python/pyspark/sql/connect/client/core.py:2615, in SparkConnectClient._handle_rpc_error(self, rpc_error)
2611 logger.debug(f"Received ErrorInfo: {info}")
2613 self._handle_rpc_error_with_error_info(info, status.message, status_code) # EDGE
-> 2615 raise convert_exception(
2616 info,
2617 status.message,
2618 self._fetch_enriched_error(info),
2619 self._display_server_stack_trace(),
2620 status_code,
2621 ) from None
2623 raise SparkConnectGrpcException(
2624 message=status.message,
2625 sql_state=ErrorCode.CLIENT_UNEXPECTED_MISSING_SQL_STATE, # EDGE
2626 grpc_status_code=status_code,
2627 ) from None
2628 else:
Monday
Additionally, I have used SRA to deploy with custom networking option
yesterday
The HTTP 200 from the workspace REST API call confirms the basic Classic compute → workspace PrivateLink path is working.
One additional thing to check is the regional hostname used by UC, since UC can use that path directly rather than the workspace URL.
%sh
nslookup ncalifornia.cloud.databricks.comCheck if that also resolves to private IPs. Databricks documents that if Private DNS is not enabled on the workspace VPC endpoint, the regional hostname can resolve publicly even though the workspace URL resolves correctly.
yesterday
The nslookup resolves to a public IP. Is this an issue? it seems AWS did not create a private route for it.
%sh
nslookup ncalifornia.cloud.databricks.com
Server: 10.10.0.2
Address: 10.10.0.2#53
Non-authoritative answer:
ncalifornia.cloud.databricks.com canonical name = public-ingress-a84df8d06aede39e.elb.us-west-2.amazonaws.com.
Name: public-ingress-a84df8d06aede39e.elb.us-west-2.amazonaws.com
Address: 44.234.192.46
Name: public-ingress-a84df8d06aede39e.elb.us-west-2.amazonaws.com
Address: 44.234.192.47
Name: public-ingress-a84df8d06aede39e.elb.us-west-2.amazonaws.com
Address: 44.234.192.45yesterday
Thank you @ivanvyd, @data_pulse , and @balajij8. The issue was resolved after I added wild card private zone to Route 53,
yesterday
Great, that confirms this was a private DNS resolution issue. Thanks for posting the fix.