topic Re: How do you Upload TXT and CSV files into Shared Workspace in Databricks? in Data Engineering

How do you Upload TXT and CSV files into Shared Workspace in Databricks?

Shay — Tue, 07 Jun 2022 07:47:04 GMT

I try to upload the needed files under the right directory of the project to work.

The files are zipped first as that is an accepted format.

I have a Python project which requires the TXT and CSV format files as they are called and used via .py files to execute the functions from the notebook.

The error I am met with is Import failed with error: Could not deserialize: Unknown language: "txt" culture

Any idea regarding txt and csv upload capabilities in the Databricks Workspace?

This would be greatly appreciated.

Kind Regards,

Shay

Re: How do you Upload TXT and CSV files into Shared Workspace in Databricks?

-werners- — Tue, 07 Jun 2022 08:04:50 GMT

@Shay Alam, can you share the code with which you read the files? Apparently python interprets the file format as a language, so it seems like some options are not filled in correctly.

Re: How do you Upload TXT and CSV files into Shared Workspace in Databricks?

Shay — Tue, 07 Jun 2022 08:22:54 GMT

Hi @Werner Stinckens, so the UI is used to upload the files in a zip format, by going to the right part of the directory of the Python project and then selecting import, no particular code is used to upload them, or do you mean the python project code?

Thank You

Re: How do you Upload TXT and CSV files into Shared Workspace in Databricks?

-werners- — Tue, 07 Jun 2022 09:13:27 GMT

@Shay Alam, I mean the python project code.

I assume the upload itself was successful

Re: How do you Upload TXT and CSV files into Shared Workspace in Databricks?

Shay — Tue, 07 Jun 2022 10:30:15 GMT

@Werner Stinckens Hi, sorry for the confusion, the upload itself is the issue, the files and project work fine inside the Repo section, but when uploading the project files into the shared workspace, the .py files and notebook are there, but the txt or csv files do not upload into the shared workspace.

Re: How do you Upload TXT and CSV files into Shared Workspace in Databricks?

-werners- — Tue, 07 Jun 2022 10:33:32 GMT

@Shay Alam

Ok I see 🙂

Files with .zip extension must only contain folders, HTML files, or source files (Python, Scala, SQL, R). So txt and csv do not work. You can store those in the FileStore or external storage (blob, ...)

Re: How do you Upload TXT and CSV files into Shared Workspace in Databricks?

Shay — Tue, 07 Jun 2022 10:39:34 GMT

Ok, understood and appreciate this, is there any other way to upload them into the shared workspace or is is just that these are the only formats allowed?

Additionally if I upload those files to FileStore or elsewhere I assume the project will need to be changed to look for them outside the directory?

I was trying to keep these files within the structure of the project for just general convenience of having the project in one place with necessary files and directories inside.

Thank you 🙂

Re: How do you Upload TXT and CSV files into Shared Workspace in Databricks?

-werners- — Tue, 07 Jun 2022 10:49:31 GMT

These are the only formats allowed AFAIK.

You do have to change you project indeed. But as long as you do not move the files around that should only be a small change.

The reason it is like that is because the source data (aka 'the files') can be used in several projects, the project is not the owner of the data.

What you could try is to package everything in a wheel or something similar.

I do a similar thing with configuration files in a jar.

Re: How do you Upload TXT and CSV files into Shared Workspace in Databricks?

Shay — Wed, 08 Jun 2022 12:38:26 GMT

Thank you very much for your insight on this, will try a workaround