topic AssertionError: assertion failed: Unable to delete the record but I am able to select it though in Data Engineering

AssertionError: assertion failed: Unable to delete the record but I am able to select it though

User16826994223 — Fri, 25 Jun 2021 16:45:03 GMT

Is there any reason this command works well:

%sql 
SELECT * FROM  datanase.table WHERE  salary > 1000

returning 2 rows, while the below:

%sql 
delete FROM  datanase.table WHERE  salary > 1000

Error

Error in SQL statement: AssertionError: assertion failed: No plan for DeleteFromTable

Re: AssertionError: assertion failed: Unable to delete the record but I am able to select it though

User16826994223 — Fri, 25 Jun 2021 16:45:49 GMT

DELETE FROM (and similarly UPDAT. aren't supported on the Parquet files - right now on Databricks, it's supported for Delta format. You can convert your parquet files into delta using CONVERT TO DELTA, and then this command will work for you.

Re: AssertionError: assertion failed: Unable to delete the record but I am able to select it though

kjoth — Wed, 13 Oct 2021 07:02:00 GMT

Have multiple questions on converting to delta.

When I tried to covert as DELTA, then I cant access the parquet files using spark.read.parquet
If converted to delta, then i ran a delete command for a filter example> where id = 1001. After the query execution, this data for id 1001 will be deleted from the underlying files or it will present, but not returned when running query