topic Re: Data preparation in Databricks in Data Engineering

Data preparation in Databricks

Priyag1 — Mon, 26 Jun 2023 17:26:32 GMT

Data preparation in Databricks

Good data is important to ensure accurate and useful results. To get good data following tasks must be done

Cleaning and formatting data - Handling missing values or outliers, ensuring data is in the correct format, and removing unneeded columns.
Preprocessing data- Numerical transformations, aggregating data, encoding text or image data, and creating new features.
Combining data.- Joining tables or merging datasets.

Data preparation resources

Re: docs.databricks.com

Anonymous — Sat, 17 Jun 2023 09:40:50 GMT

Hi @Priyadarshini G

Great to meet you, and thanks for your question!

Let's see if your peers in the community have an answer to your question. Thanks.

Re: docs.databricks.com

bharats — Fri, 23 Jun 2023 05:35:18 GMT

Useful Information. Hope u do more summarized posts on these concepts

Re: Data preparation in Databricks

Sandro — Wed, 28 Jun 2023 21:59:25 GMT

Great introduction, for some cases, I would add some other dimensions of data quality, such as completeness of data and referential integrity validation.

Re: Data preparation in Databricks

dplante — Sun, 16 Jul 2023 00:16:34 GMT

Data governance and data lineage are other things to call out.

Here's a cheat sheet that is also useful -> Data Preparation Cheatsheet