cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How can I extract data from different sources and transform it into a fresh, reliable data pipeline?

User16835756816
Valued Contributor

Tip: These steps are built out for AWS accounts and workspaces that are using Delta Lake. If you would like to learn more watch this video and reach out to your Databricks sales representative for more information.

Step 1: Create your own notebook or use an existing notebook

Step 2: Ensure your data is ingested into the lakehouse

Step 3: Efficiently read and write your data

Step 4: Automate your notebook with a job

Step 5: Run your job interactively or on a schedule

Step 6: Further optimize your data pipeline with delta write optimize & multi-task jobs

If you'd like to learn more, sign up for the Databricks Data Engineering Databricks Academy course and get your Data Engineering Certification.

Need more help? Get hands-on with building data pipelines by attending the Data Engineering Activation Day on December 6, 2022, at 9 AM PT. Whether you are just getting started ingesting data or you are ready to automate your data pipeline, our Databricks experts will walk you through each step of the way with ample time for your questions.

In this session, you’ll see demos and learn:

- How to quickly create your own notebook

- All ways to ingest your data

- Best practices to read and write your data

- Automated ways to run your commands with Jobs

Notebooks will be provided so you can follow along live or review the best practices at your own pace. 

4 REPLIES 4

Aviral-Bhardwaj
Esteemed Contributor III

thanks man

AviralBhardwaj

Own
Contributor

If you are using Azure Databricks prefer ADF Pipelines for ETL.​

Aj2
New Contributor III

Thanks for this.

Ajay-Pandey
Esteemed Contributor III

Thanks @Nithya Thangaraj​ 

Ajay Kumar Pandey

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group