Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-18-2022 06:29 AM
- Create, as you said table registered in metastore, but for that, you need to define the schema
- write stream directly into that table .writeStream.table(table_name)
To get the schema, just read your CSV as not stream and take it from dataframe
csvFile = spark.read.csv("yourfile")
csvFile .createOrReplaceTempView("csvFile ")
schema_json = spark.sql("SELECT * FROM csvFile ").schema.json()
ddl = spark.sparkContext._jvm.org.apache.spark.sql.types.DataType.fromJson(schema_json).toDDL()
print(ddl)
My blog: https://databrickster.medium.com/