@Vasanth Kumar​ a clarification to my original answer: you need to provide schema to the CREATE TABLE statement, it doesn't work (except in some edge cases) without a schema.

The fact that you have new data incoming changes things. If your data were arriving in a single directory, it would be super trivial to load it using the Databricks Auto Loader. You just point Auto Loader to an ABFS path and it continuously loads data (either via direct file listing, or through a cloud queue).

Let me dig into this scenario a bit more and come back to you ...