https://community.databricks.com/t5/data-engineering/vimeo-com/m-p/33672#M24621 <P><B>Auto Loader</B> provides Python and Scala methods to ingest new data from a folder location into a Delta Lake table by using directory listing or file notifications.&nbsp;Here's a <A href="https://vimeo.com/643704503?embedded=true&amp;source=video_title&amp;owner=40921445." alt="https://vimeo.com/643704503?embedded=true&amp;source=video_title&amp;owner=40921445." target="_blank">quick video (7:00)</A> on how to use Auto Loader for <A href="https://databricks.com/try-databricks?_ga=2.267305741.2022881793.1638748477-1565564318.1621642043&amp;_gac=1.251483636.1638463719.Cj0KCQiA-qGNBhD3ARIsAO_o7yklJ-pR-pOZnU66U1871wkeyIQBFmP14IH3EzJTlBcBSJuTnsT5H3YaAoQoEALw_wcB" alt="https://databricks.com/try-databricks?_ga=2.267305741.2022881793.1638748477-1565564318.1621642043&amp;_gac=1.251483636.1638463719.Cj0KCQiA-qGNBhD3ARIsAO_o7yklJ-pR-pOZnU66U1871wkeyIQBFmP14IH3EzJTlBcBSJuTnsT5H3YaAoQoEALw_wcB" target="_blank">Databricks on AWS</A> with Databricks Runtime (DBR) 8.3 or higher.</P><P></P><P>To follow along with the video, import this notebook into your workspace:</P><P><A href="https://files.training.databricks.com/lms/aws-onboarding-incremental-etl/incremental-etl.dbc" alt="https://files.training.databricks.com/lms/aws-onboarding-incremental-etl/incremental-etl.dbc" target="_blank">https://files.training.databricks.com/lms/aws-onboarding-incremental-etl/incremental-etl.dbc</A></P><P></P><P>For more on Auto Loader, visit the <A href="https://docs.databricks.com/spark/latest/structured-streaming/auto-loader.html" alt="https://docs.databricks.com/spark/latest/structured-streaming/auto-loader.html" target="_blank">docs</A>. Question and ideas? Leave a comment below!</P> Mon, 06 Dec 2021 20:24:26 GMT MadelynM 2021-12-06T20:24:26Z https://community.databricks.com/t5/data-engineering/vimeo-com/m-p/33672#M24621 <P><B>Auto Loader</B> provides Python and Scala methods to ingest new data from a folder location into a Delta Lake table by using directory listing or file notifications.&nbsp;Here's a <A href="https://vimeo.com/643704503?embedded=true&amp;source=video_title&amp;owner=40921445." alt="https://vimeo.com/643704503?embedded=true&amp;source=video_title&amp;owner=40921445." target="_blank">quick video (7:00)</A> on how to use Auto Loader for <A href="https://databricks.com/try-databricks?_ga=2.267305741.2022881793.1638748477-1565564318.1621642043&amp;_gac=1.251483636.1638463719.Cj0KCQiA-qGNBhD3ARIsAO_o7yklJ-pR-pOZnU66U1871wkeyIQBFmP14IH3EzJTlBcBSJuTnsT5H3YaAoQoEALw_wcB" alt="https://databricks.com/try-databricks?_ga=2.267305741.2022881793.1638748477-1565564318.1621642043&amp;_gac=1.251483636.1638463719.Cj0KCQiA-qGNBhD3ARIsAO_o7yklJ-pR-pOZnU66U1871wkeyIQBFmP14IH3EzJTlBcBSJuTnsT5H3YaAoQoEALw_wcB" target="_blank">Databricks on AWS</A> with Databricks Runtime (DBR) 8.3 or higher.</P><P></P><P>To follow along with the video, import this notebook into your workspace:</P><P><A href="https://files.training.databricks.com/lms/aws-onboarding-incremental-etl/incremental-etl.dbc" alt="https://files.training.databricks.com/lms/aws-onboarding-incremental-etl/incremental-etl.dbc" target="_blank">https://files.training.databricks.com/lms/aws-onboarding-incremental-etl/incremental-etl.dbc</A></P><P></P><P>For more on Auto Loader, visit the <A href="https://docs.databricks.com/spark/latest/structured-streaming/auto-loader.html" alt="https://docs.databricks.com/spark/latest/structured-streaming/auto-loader.html" target="_blank">docs</A>. Question and ideas? Leave a comment below!</P> Mon, 06 Dec 2021 20:24:26 GMT https://community.databricks.com/t5/data-engineering/vimeo-com/m-p/33672#M24621 MadelynM 2021-12-06T20:24:26Z