Hi.
I have a workflow in which I write few rows into a Delta Table with auto-generated IDs. Then, I need to retrieve them back just after they're written into the table to collect those generated IDs, so I read the table and I use two columns (one is a client_id and the other is a timestamp) of the "input" DataFrame we inserted as join key.
The problem is that sometimes it's retrieving them, but not always. I did some smoke tests in an interactive notebook and I was never able to recreate what's happening in the workflow.
Here there is a screenshot of the Spark UI in which you can see the operations (saveAsTable and the two subsequent collects) and their submit time.
I did a lot of various tests in both the workflow and the interactive notebook, but still no relevant result. I thought it was something related to the timestamp and/or the timezone, but everything looked good to me. I've also tried to convert them to unix_timestamps and still the issue occurs.
Please, could you help me?
Thank you.
Best Regards.
