cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Databricks write to iceberg managed table with pyiceberg

tdata
Visitor

Hello,

Im trying to write to databricks managed icerberg table using pyiceberg inside a spark_python_task (Serverless compute).

Im facing an error when writing :

 

Error writing to Iceberg table: When reading information for key '' in bucket '' : AWS Error NETWORK_CONNECTION during HeadObject operation: curlCode: 35, SSL connect error; Details: Recv failure: Connection reset by peer.
 
Im running this code :

catalog_config = {
        "type": "rest",
        "uri": (
            f"https://{workspace_url}/api/2.1/unity-catalog/iceberg-rest"
        ),
        "warehouse": catalog_name,
        "token": databricks_pat,
    }

catalog = load_catalog("catalog_name", **catalog_config)
table = catalog.load_table(table_identifier)
table.append(df)
 
I want to know if there is a special uri to use because im inside databricks compute, or if its just not possible.
This code run find outside databricks.
 
Thank you 
0 REPLIES 0