curl: (26) Failed to open/read local data from file/application in DBFS
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-08-2023 05:25 AM - edited 12-08-2023 05:35 AM
Hi all,
I am trying to upload a parquet file from S3 to dbfs with airflow bash operator curl command using Databricks python Rest API's as shown below
databricks_load_task = BashOperator(
task_id="upload_to_databricks",
bash_command = """
curl --location --request POST {{task_instance.xcom_pull(task_ids='get_creds', key='DATABRICKS_HOST')}}/api/2.0/dbfs/put \
--header "Authorization: Bearer {{task_instance.xcom_pull(task_ids='get_creds', key='DATABRICKS_TOKEN')}}" \
--form contents="@s3://bucket/test/file.parquet"\
--form path="{{task_instance.xcom_pull(task_ids='get_creds', key='UPLOAD_PATH')}}" \
--form overwrite="true"
"""
)
Parquet files stores the dataframe result. I am unable to upload the file as it gives me the below error
curl: (26) Failed to open/read local data from file/application
I tried to replace the content from s3 path to text(--form contents="test text") this works for me. Please help me with this.
#dbfs
Labels:
- Labels:
-
Workflows
0 REPLIES 0

