Databricks write to iceberg managed table with pyiceberg

tdata
New Contributor

Hello,

Im trying to write to databricks managed icerberg table using pyiceberg inside a spark_python_task (Serverless compute).

Im facing an error when writing :

 

Error writing to Iceberg table: When reading information for key '' in bucket '' : AWS Error NETWORK_CONNECTION during HeadObject operation: curlCode: 35, SSL connect error; Details: Recv failure: Connection reset by peer.
 
Im running this code :

catalog_config = {
        "type": "rest",
        "uri": (
            f"https://{workspace_url}/api/2.1/unity-catalog/iceberg-rest"
        ),
        "warehouse": catalog_name,
        "token": databricks_pat,
    }

catalog = load_catalog("catalog_name", **catalog_config)
table = catalog.load_table(table_identifier)
table.append(df)
 
I want to know if there is a special uri to use because im inside databricks compute, or if its just not possible.
This code run find outside databricks.
 
Thank you