Databricks Community

Do you want to add dates for the historical data as well?

That's a valid point about minimal read queries! However, while immediate storage reduction might not be necessary, consistent data integrity and potential future reporting needs might still warrant occasional optimize and vacuuming, even with extern...

Thank you, @SP_6721 , for your input!Could you please share an example snippet demonstrating how to handle batch processing, similar to what we typically do in a relational database?

I like the idea of using Feature Store with Delta Sharing, but I’m a bit worried about its limits like no partition filtering and no streaming support. These could cause problems with performance and scaling in real situations.

Hi @NathanC0926 Ingesting Excel files with streaming tables requires a combination of Databricks Autoloader(for file discovery and exactly-once processing) and a custom UDF for Excel parsing.Here's the native approachKey Features of This Solution1. E...

Hi @RameshRetnasamy first off thank you so much for taking the time to reply to my question. In my case they were indeed created via Notebooks, but I'll re-evaluate on my end as I might've missed something. If the issue persists, I'll re-assert the q...

Hi @nikhilj0421  still the same issue, i could see its working till spark 15.4 LTS above 15.4LTS whatever version i useits not woring i am not able to create any cluster.

Hi @hao-uit, do you see any kind of load on the driver and event logs? Also, what libraries you have installed on your cluster? 

Hi @Dev ,Generally, What happens spark reader APIs point to the DBFS by default. And, to read the file from User workspace, we need to append 'file:/' in the prefix.Thanks