cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Create New Table, Infer Schema gives error: Invalid column type {colSchemaType}

lauracoursera
New Contributor II

I'm doing a the course 'Distributed Computing with Spark SQL' on Coursera, and need to create a table by uploading a csv file. That seems to work at first, but as soon as I check the box for 'Infer schema' for the preview table, I get the following message:

The following errors prevented the table from rendering:

  • Invalid column type {colSchemaType} at column Call_Date
  • Invalid column type {colSchemaType} at column Watch_Date

 

The csv file I used, can be found by following this link: https://s3-us-west-2.amazonaws.com/davis-dsv1071/data/fire-calls/fire-calls-truncated-comma-no-space...

I would appreciate it very much if you can help me!

 

 

4 REPLIES 4

filipniziol
Contributor

Hi @lauracoursera ,

could you share the how you uploaded the file to Databricks to get the error.

I have downloaded the file, went to Data Ingestion and clicked Create or modify table from file upload:

filipniziol_1-1726053465368.png

The preview worked:

filipniziol_2-1726053526238.png

If it does not work for you, show the content of Advanced attributes:

filipniziol_3-1726053572041.png

Also, what offering of Databricks are you using: Azure Datbricks, AWS Databricks or Databricks Community Edition?

 

Hi @filipniziol,
Thanks for looking with me! I use Community Edition. So I get there by clicking 'Catalog', 'Create Table', and then uploading the file. Then 'Create Table with UI', I choose a cluster and click 'Preview Table'. As soon as I check 'Infer schema', I get this:

Scherm­afbeelding 2024-09-11 om 13.32.14.png

filipniziol
Contributor

Hi @lauracoursera ,

In Databricks Community Edition I am getting the same error as you:

filipniziol_0-1726081709161.png

The Community Edition is very limited - the UI is not updated to the newest version, it has old runtimes, missing features etc.

My recommendation is to register a free Microsoft Azure account. 
The new customers get 200 USD credit to spend within the first 30 days.It is more than enough to go through your course. Additionally, you will be working with the current version of Databricks, having access to the current UI and be able to test the most recent features. 

Check this link:
https://azure.microsoft.com/en-us/pricing/purchase-options/azure-account



Ok! Good to know I'm not doing anything wrong. Thanks, I'm going to look into it! 🙂

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group