I am trying to load a delta table from a dataframe. But its giving me an error.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-13-2019 08:14 AM
Code :
from pyspark.sql.functions import *
acDF = spark.read.format('csv').options(header='true', inferschema='true').load("/mnt/rahulmnt/Insurance_Info1.csv");
acDF.write.option("overwriteSchema", "true").format("delta").mode("overwrite").save("/delta/InsuranceInput")
Error :
AnalysisException: 'Incompatible format detected.\n\nYou are trying to write to
/delta/InsuranceInputusing Databricks Delta, but there is no\ntransaction log present. Check the upstream job to make sure that it is writing\nusing format("delta") and that you are trying to write to the table base path.\n\nTo disable this check, SET spark.databricks.delta.formatCheck.enabled=false\nTo learn more about Delta, see https://docs.azuredatabricks.net/delta/index.html\n;'