Hubert-Dudek
Databricks MVP
  • Create, as you said table registered in metastore, but for that, you need to define the schema
  • write stream directly into that table .writeStream.table(table_name)

To get the schema, just read your CSV as not stream and take it from dataframe

csvFile = spark.read.csv("yourfile")
csvFile .createOrReplaceTempView("csvFile ")
schema_json = spark.sql("SELECT * FROM csvFile ").schema.json()
ddl = spark.sparkContext._jvm.org.apache.spark.sql.types.DataType.fromJson(schema_json).toDDL()
print(ddl)


My blog: https://databrickster.medium.com/