SparkR or sparklyr not showing history

Sagas — Mon, 29 Apr 2024 10:48:32 GMT

Hi,

for some reason Azure Databricks doesn't show History if the data is saved with SparkR (2 in the figure below) or Sparklyr (3), but it does show it with Data Ingestion (0) or with PySpark (1). Is this a known bug or am I doing something wrong? Is it possible to save data with R while getting the UserId and Username?

PySpark code:

test_data = [[4,"d",30],[5,"e",70]] df = spark.createDataFrame(test_data,['a','b','c']) df.write.mode('append').format("delta").saveAsTable("catalog.schema.table")

SparkR df and save to table:

Sparklyr df and save to table:

Re: SparkR or sparklyr not showing history

Sagas — Mon, 29 Apr 2024 14:00:33 GMT

Thank you for your response! I'll consider capturing UserId for each row. But do you mean that also sparklyr can't automatically capture the userId for the Delta table history?

topic SparkR or sparklyr not showing history in Data Engineering

SparkR or sparklyr not showing history

Re: SparkR or sparklyr not showing history