How can I extract data from different sources and transform it into a fresh, reliable data pipeline?

User16835756816 — Mon, 28 Nov 2022 20:04:54 GMT

Tip: These steps are built out for AWS accounts and workspaces that are using Delta Lake. If you would like to learn more watch this video and reach out to your Databricks sales representative for more information.

Step 1: Create your own notebook or use an existing notebook

Watch this quick video

Step 2: Ensure your data is ingested into the lakehouse

Step 3: Efficiently read and write your data

Step 4: Automate your notebook with a job

Watch this quick video

Step 5: Run your job interactively or on a schedule

Step 6: Further optimize your data pipeline with delta write optimize & multi-task jobs

If you'd like to learn more, sign up for the Databricks Data Engineering Databricks Academy course and get your Data Engineering Certification.

Need more help? Get hands-on with building data pipelines by attending the Data Engineering Activation Day on December 6, 2022, at 9 AM PT. Whether you are just getting started ingesting data or you are ready to automate your data pipeline, our Databricks experts will walk you through each step of the way with ample time for your questions.

In this session, you’ll see demos and learn:

- How to quickly create your own notebook

- All ways to ingest your data

- Best practices to read and write your data

- Automated ways to run your commands with Jobs

Notebooks will be provided so you can follow along live or review the best practices at your own pace.

Re: How can I extract data from different sources and transform it into a fresh, reliable data pipeline?

Aviral-Bhardwaj — Fri, 02 Dec 2022 09:44:02 GMT

thanks man

Re: How can I extract data from different sources and transform it into a fresh, reliable data pipeline?

Own — Fri, 02 Dec 2022 12:09:36 GMT

If you are using Azure Databricks prefer ADF Pipelines for ETL.

Re: How can I extract data from different sources and transform it into a fresh, reliable data pipeline?

Aj2 — Mon, 05 Dec 2022 03:47:03 GMT

Thanks for this.

Re: How can I extract data from different sources and transform it into a fresh, reliable data pipeline?

Ajay-Pandey — Mon, 05 Dec 2022 07:02:29 GMT

Thanks @Nithya Thangaraj

topic Re: How can I extract data from different sources and transform it into a fresh, reliable data pipeline? in Data Engineering

How can I extract data from different sources and transform it into a fresh, reliable data pipeline?

Re: How can I extract data from different sources and transform it into a fresh, reliable data pipeline?

Re: How can I extract data from different sources and transform it into a fresh, reliable data pipeline?

Re: How can I extract data from different sources and transform it into a fresh, reliable data pipeline?

Re: How can I extract data from different sources and transform it into a fresh, reliable data pipeline?