Can't create a new table from uploaded file.

[Corwyn]

I've just started using the Community Edition through the AWS marketplace and I'm trying to set up tables to share with a customer.  I've managed to create 3 of the tables, but when uploading a small version of the fourth file, I'm having problems.

The original file that I want to upload is 14 GB, so for testing purposes I cut it down to the first 1000 rows (and then 100 rows when that didn't work).

It shows me a preview for 50 rows and 539 columns, as expected, and there aren't any error messages that I can find on the page.  I've tried auto-detection of column types and also turning it off, with no change.

However, I can't create the table:

Any ideas of how I can figure out what errors I need to fix?  Thank you!

-Corwyn

[BigRoux]

Common Root Causes of Table Upload Failure

1. File Size Limitations

   • The file upload UI in Databricks only supports a total upload size of under 2 GB, regardless of the number or size of individual files. Attempting to upload or create tables from files larger than this will not succeed.

2. Column Count or Data Complexity

   • While there is no explicit column limit documented in the upload documentation, issues with very wide tables (e.g., several hundred columns) can cause schema inference or display problems in the UI, especially for files with 500+ columns.

3. Preview vs. Table Creation

   • Even if the file previews correctly (e.g., showing 50 rows as expected), there may still be underlying schema inference or parsing issues that only surface when you attempt to "create" the table. The preview step does not perform full validation.

4. Format and File Integrity

   • The file format must be one of the supported types: CSV, TSV, JSON, Avro, Parquet, or plain text, and not compressed (e.g., neither .zip nor .tar files). Any violation can silently fail table creation.
   • Consistency is required in header rows when uploading multiple files; mixed headers can cause silent data loss or schema confusion.

5. Permission and Compute Requirements

   • You must have an active compute resource attached (a running cluster or SQL warehouse) and proper permissions to write to the destination schema. If a cluster is not active or you lack rights, table creation will not proceed (sometimes with little/no error detail).

6. Workspace or Platform Limitations

   • Workspace admins can disable the file upload page entirely. Since Community Edition has limited features (and sometimes divergent UI), some functionality may be restricted or buggy compared to full Databricks workspaces.
   • Community Edition is known to have fewer debugging and logging tools available for users compared to paid tiers.

In short, Community Editions is really designed for learning on very small datasets.

Hope this helps, Lou.

[Corwyn · Accepted solution]

Thank you, Lou.

By loading manually, I found the error that wasn't being displayed in the UI. Once I took care of this, everything loaded just fine.