cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
sanjay
by • Valued Contributor II
  • 5731 Views
  • 8 replies
  • 0 kudos

error after updating delta table com.databricks.sql.transaction.tahoe.DeltaUnsupportedOperationException: Detected a data update

Hi,I have pipeline running. I have updated one file in delta table which is already processed. Now I am getting errorcom.databricks.sql.transaction.tahoe.DeltaUnsupportedOperationException: Detected a data update. This is currently not supported. If ...

Data Engineering
Reply
  • 5731 Views
  • 8 replies
  • 0 kudos
View Replies
Latest Reply
Sanjeev_Chauhan
New Contributor II
  • 0 kudos

Hi Sanjay, You can try adding .option("overwriteSchema", "true")

  • 0 kudos
7 More Replies
sanjay
by • Valued Contributor II
  • 1318 Views
  • 4 replies
  • 1 kudos

Resolved! How can I get date when autoloader processes the file

Hi,I am running autoloader which is running continuously and checks for new file every 1 minute. I need to store when file was received/processed but its giving me date when autoloader started. Here is my code.df = (spark   .readStream   .format("clo...

Data Engineering
Reply
  • 1318 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
Lakshay
Esteemed Contributor
  • 1 kudos

Hi @Sanjay Jain​ , You can use the File Metadata column functionality to collect that information.Ref doc:- https://docs.databricks.com/ingestion/file-metadata-column.html

  • 1 kudos
3 More Replies
Labels