How do you Upload TXT and CSV files into Shared Workspace in Databricks?

[Shay]

I try to upload the needed files under the right directory of the project to work.

The files are zipped first as that is an accepted format.

I have a Python project which requires the TXT and CSV format files as they are called and used via .py files to execute the functions from the notebook.

The error I am met with is Import failed with error: Could not deserialize: Unknown language: "txt" culture

Any idea regarding txt and csv upload capabilities in the Databricks Workspace?

This would be greatly appreciated.

Kind Regards,

Shay

[-werners-]

@Shay Alam​, can you share the code with which you read the files? Apparently python interprets the file format as a language, so it seems like some options are not filled in correctly.

[Shay]

Hi @Werner Stinckens​, so the UI is used to upload the files in a zip format, by going to the right part of the directory of the Python project and then selecting import, no particular code is used to upload them, or do you mean the python project code?

Thank You

[-werners-]

@Shay Alam​, I mean the python project code.

I assume the upload itself was successful

[Shay]

@Werner Stinckens​ Hi, sorry for the confusion, the upload itself is the issue, the files and project work fine inside the Repo section, but when uploading the project files into the shared workspace, the .py files and notebook are there, but the txt or csv files do not upload into the shared workspace.

[-werners-]

@Shay Alam​ 

Ok I see 🙂

Files with .zip extension must only contain folders, HTML files, or source files (Python, Scala, SQL, R). So txt and csv do not work. You can store those in the FileStore or external storage (blob, ...)

[Shay]

Ok, understood and appreciate this, is there any other way to upload them into the shared workspace or is is just that these are the only formats allowed?

Additionally if I upload those files to FileStore or elsewhere I assume the project will need to be changed to look for them outside the directory?

I was trying to keep these files within the structure of the project for just general convenience of having the project in one place with necessary files and directories inside.

Thank you 🙂

[-werners- · Accepted solution]

These are the only formats allowed AFAIK.

You do have to change you project indeed. But as long as you do not move the files around that should only be a small change.

The reason it is like that is because the source data (aka 'the files') can be used in several projects, the project is not the owner of the data.

What you could try is to package everything in a wheel or something similar.

I do a similar thing with configuration files in a jar.

[Shay]

Thank you very much for your insight on this, will try a workaround