โ06-12-2023 11:58 AM
I have a number of csv files that I am working to ingest using autoloader. There is an ID field that I want to require to be a STRING, but using SchemaHints is not working and is instead setting as an INT.
The first few csv files have just integer values for the ID field, i.e. '123'4, '9876', '10371', etc.
A later csv file has some fields with non-numeric values '1234c', '9d87', etc.
This is why the column needs to be set as a STRING, but despite specifying this in schemahints, autoloader is still defining it as an INT.
I have tried doing a full refresh on the table, with all of these files available together in the source directory.
What else can I do to enforce the schema to be a string? Thank you.
โ06-13-2023 06:38 AM
@Jennette Shepardโ Is this what you are looking for?
Basically you define a schema yourself. There are lots of examples to be found online on how to do that.
โ06-13-2023 06:38 AM
@Jennette Shepardโ Is this what you are looking for?
Basically you define a schema yourself. There are lots of examples to be found online on how to do that.
โ06-13-2023 08:19 PM
Hi @Jennette Shepardโ
Thank you for posting your question in our community! We are happy to assist you.
To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?
This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!
โ06-14-2023 12:37 AM
@Jennette Shepardโ
If specifying the data type using SchemaHints is not working as expected, you can try the following steps to enforce the ID field as a string:
Remember to double-check the documentation and syntax specific to your autoloader tool for precise guidance on how to enforce the schema for CSV files.
โ06-14-2023 11:36 PM
Hi @Jennette Shepardโ
We haven't heard from you since the last response from @Suteja Kanuriโ . Kindly share the information with us, and in return, we will provide you with the necessary solution.
Thanks and Regards
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group