AmanSehgal
Honored Contributor III

You can write your ETL logic in notebooks, run the notebook over a cluster and write the data to a location where your S3 bucket is mounted.

Next, you can register that table with Hive MetaStore and access the same table in Databricks SQL.

To see the table, go to Data tab and select your schema/database to see registered tables.

Two ways to do this:

Option 1:

df.write.option("path",<s3-path-of-table>).saveAsTable(tableName)

Option 2

%python
df.write.save(<s3-path-of-table>)
 
%sql
CREATE TABLE <table-name>
USING DELTA
LOCATION <s3-path-of-table>

:

View solution in original post