cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Unauthorized network access to workspace

nsingh_tl
New Contributor
We're setting up a new workspace on AWS commercial cloud (customer-managed VPC, back-end PrivateLink) and are blocked from using Unity Catalog due to an account-level network policy restriction.
 
Issue:
Notebook calls to Unity Catalog fail with:
Error Code: UNEXPECTED_HTTP_ERROR
Message: HTTP request failed with status: HTTP/1.1 403 Forbidden, original HTTP response body: Unauthorized network access to workspace
20 REPLIES 20

I couldn't appreciate it more, thank you for your help, below are additional information

Runtime version: 18 LTS
Access mode: Standard 

[UC_UNCAUGHT_CLIENT_EXCEPTION] Encountered an unexpected HTTP error while communicating with the Databricks Unity Catalog backend. Error Code: UNEXPECTED_HTTP_ERROR. Message: HTTP request failed with status: HTTP/1.1 403 Forbidden, original HTTP response body: Unauthorized network access to workspace: 4444444444444 SQLSTATE: KCUC4
File <command-8700090828854269>, line 2
----> 2 spark.sql("show catalogs")
File /databricks/spark/python/pyspark/sql/connect/session.py:920, in SparkSession.sql(self, sqlQuery, args, **kwargs)
    917         _views.append(SubqueryAlias(df._plan, name))
    919 cmd = SQL(sqlQuery, _args, _named_args, _views)
--> 920 data, properties, ei = self.client.execute_command(cmd.command(self._client))
    921 if "sql_command_result" in properties:
    922     df = DataFrame(CachedRelation(properties["sql_command_result"]), self)
File /databricks/spark/python/pyspark/sql/connect/client/core.py:1618, in SparkConnectClient.execute_command(self, command, observations, extra_request_metadata)
   1616     req.user_context.user_id = self._user_id
   1617 self._set_command_in_plan(req.plan, command)
-> 1618 data, _, metrics, observed_metrics, properties = self._execute_and_fetch(
   1619     req, observations or {}, extra_request_metadata
   1620 )
   1621 # Create a query execution object.
   1622 ei = ExecutionInfo(metrics, observed_metrics)
File /databricks/spark/python/pyspark/sql/connect/client/core.py:2188, in SparkConnectClient._execute_and_fetch(self, req, observations, extra_request_metadata, self_destruct)
   2185 properties: Dict[str, Any] = {}
   2187 with Progress(handlers=self._progress_handlers, operation_id=req.operation_id) as progress:
-> 2188     for response in self._execute_and_fetch_as_iterator(
   2189         req, observations, extra_request_metadata or [], progress=progress
   2190     ):
   2191         if isinstance(response, StructType):
   2192             schema = response
File /databricks/spark/python/pyspark/sql/connect/client/core.py:2164, in SparkConnectClient._execute_and_fetch_as_iterator(self, req, observations, extra_request_metadata, progress)
   2162     raise kb
   2163 except Exception as error:
-> 2164     self._handle_error(error)
File /databricks/spark/python/pyspark/sql/connect/client/core.py:2537, in SparkConnectClient._handle_error(self, error)
   2535     self.thread_local.inside_error_handling = True
   2536     if isinstance(error, grpc.RpcError):
-> 2537         self._handle_rpc_error(error)
   2538     raise error
   2539 finally:
File /databricks/spark/python/pyspark/sql/connect/client/core.py:2615, in SparkConnectClient._handle_rpc_error(self, rpc_error)
   2611             logger.debug(f"Received ErrorInfo: {info}")
   2613             self._handle_rpc_error_with_error_info(info, status.message, status_code)  # EDGE
-> 2615             raise convert_exception(
   2616                 info,
   2617                 status.message,
   2618                 self._fetch_enriched_error(info),
   2619                 self._display_server_stack_trace(),
   2620                 status_code,
   2621             ) from None
   2623     raise SparkConnectGrpcException(
   2624         message=status.message,
   2625         sql_state=ErrorCode.CLIENT_UNEXPECTED_MISSING_SQL_STATE,  # EDGE
   2626         grpc_status_code=status_code,
   2627     ) from None
   2628 else:

 

nsingh_tl
New Contributor

Additionally, I have used SRA to deploy with custom networking option

data_pulse
New Contributor II

@nsingh_tl 

The HTTP 200 from the workspace REST API call confirms the basic Classic compute → workspace PrivateLink path is working.

One additional thing to check is the regional hostname used by UC, since UC can use that path directly rather than the workspace URL.

%sh
nslookup ncalifornia.cloud.databricks.com

Check if that also resolves to private IPs. Databricks documents that if Private DNS is not enabled on the workspace VPC endpoint, the regional hostname can resolve publicly even though the workspace URL resolves correctly.

The nslookup resolves to a public IP. Is this an issue? it seems AWS did not create a private route for it.

%sh
nslookup ncalifornia.cloud.databricks.com

Server:		10.10.0.2
Address:	10.10.0.2#53

Non-authoritative answer:
ncalifornia.cloud.databricks.com	canonical name = public-ingress-a84df8d06aede39e.elb.us-west-2.amazonaws.com.
Name:	public-ingress-a84df8d06aede39e.elb.us-west-2.amazonaws.com
Address: 44.234.192.46
Name:	public-ingress-a84df8d06aede39e.elb.us-west-2.amazonaws.com
Address: 44.234.192.47
Name:	public-ingress-a84df8d06aede39e.elb.us-west-2.amazonaws.com
Address: 44.234.192.45
 

nsingh_tl
New Contributor

Thank you @ivanvyd@data_pulse , and @balajij8. The issue was resolved after I added wild card private zone to Route 53, 

@nsingh_tl 

Great, that confirms this was a private DNS resolution issue. Thanks for posting the fix.