topic Re: External table issue in Data Engineering

External table issue

RohitKulkarni — Wed, 28 Sep 2022 15:33:42 GMT

Hello Team,

I am using df.write command and the table is getting created.

If you refer the below screenshot the table got created in Tables folder in dedicated sql pool. But i required in the External Tables folder.

Regards

Re: External table issue

-werners- — Thu, 29 Sep 2022 07:46:28 GMT

if you actually write into Synapse, it is not an external table. the data resides on synapse.

If you want to have an external table, write the data on your data lake in parquet/delta lake format and then create an external table on that location in synapse.

Re: External table issue

RohitKulkarni — Thu, 29 Sep 2022 07:54:52 GMT

Please can you explain me with examples .

Re: External table issue

-werners- — Thu, 29 Sep 2022 08:04:49 GMT

https://learn.microsoft.com/en-us/azure/synapse-analytics/sql/develop-tables-external-tables?tabs=hadoop

External tables are not actual tables but a 'view' on top of data which resides somewhere else, NOT in the database.

If you use df.write.format(...sqldw) you are actually gonna write into the database itself which makes it a common table.

If you would use df.write.parquet(<path>) you could create an external table on top of the data, see the link I provided.

Re: External table issue

RohitKulkarni — Thu, 29 Sep 2022 08:52:32 GMT

Thanks for sharing the link.But this can be tested in synapse analytics but not in databricks

Re: External table issue

-werners- — Thu, 29 Sep 2022 08:54:02 GMT

yes, databricks can send data to synapse but table management is done in Synapse, not databricks