Hi @soumend7115!
I will assume you are talking about managed tables in Unity Catalog here, if thats not the case, let me know.
We can segregate this in two steps:
You can use a DELETE FROM SQL statement to remove rows that match your condition. For e...
Hi @Nagarathna, a couple of questions here...
What error is the job returning after the 2 hours execution?What is the data source we are talking about? Are you trying to load data from a database, another warehouse, a bunch of csv/json/excel files?Ho...
@sahil07, the fact that you could read a csv file using spark.read.csv() is because you're using the spark native API to access the dbfs, which works just fine. The PDF reading was not possible because PyPDF2 does not use the spark API, but python st...
@sahil07, It seems that with your current setup, you can't read from DBFS using vanilla Python. I've ran some tests and managed to reproduce the error and solve it by copying the file in DBFS to the local file system of the driver node using dbutils....
