I created a table1 with some data. Then I truncated it to load new dataset. When I do select * from table . I get row count 0. But when I am trying to copy into using following command. I get error saying target table must be delta table: COPY INTO...
are there any other parameters to consider running OPTIMIZE depending cloud vendor?
The optimize is not dependent on the cloud provider whatsoever. Optimize will produce the same results regardless of the underlying storage. It is idempotent, meaning if it is run twice on the same dataset the the second execution has no effect.
I am trying to find a way to list all files, and related file sizes, in all folders and all sub folders. I guess these are called blobs, in the Databricks world. Anyway, I can easily list all files, and related file sizes, in one single folder, but ...
from azure.storage.blob import BlockBlobService block_blob_service = BlockBlobService(account_name='your_acct_name', account_key='your_acct_key') mylist = [] generator = block_blob_service.list_blobs('rawdata') for blob in generator: mylist.append(...
