DevOps Essentials for Data Engineering: Lab does not match Videos

[Daniel_B]

Hi,

The Lab for DevOps Essentials for Data Engineering does not match the training videos.

The videos and demos talk about health data while the lab uses taxi data.

No additional details is required for this to be understand as something that would benefit for immediate rectification.

Dan

[Daniel_B]

If you are reading this and are from the Databricks team and can make an improvement let me know by replying to this comment. Thanks.

[Daniel_B]

This is the output of file /Workspace/Users/labuser16784881_1789317069@vocareum.com/devops-essentials-for-data-engineering-en_us-2.2.1/DevOps Essentials for Data Engineering/Course Notebooks/M02 - CI/03 Demo - Modularizing PySpark Code - REQUIRED

It all about health. Then look at the functions from /Workspace/Users/labuser16784881_1789317069@vocareum.com/devops-essentials-for-data-engineering-en_us-2.2.1/DevOps Essentials for Data Engineering/Course Notebooks/M02 - CI/04 Lab - Modularize PySpark Code which I have pasted after the output. The functions are about NY Taxi trips. It's all a mess and distracts from learning time.

From 03 Demo - Modularizing PySpark Code - REQUIRED

%run ../Includes/Classroom-Setup-03-REQUIRED

Created the volume: labuser16784881_1789317069.default.health. Created the volume: labuser16784881_1789317069_1_dev.default.health. Created the volume: labuser16784881_1789317069_2_stage.default.health. Created the volume: labuser16784881_1789317069_3_prod.default.health. ----Creating the CSV files in the necessary volumes for the course---- ---DELETE FILES IN THE HEALTH VOLUME WITHIN THE MAIN, DEV, STAGE, PROD CATALOGS---- Searching for files in /Volumes/labuser16784881_1789317069_3_prod/default/health volume to delete prior to creating files... No files found in /Volumes/labuser16784881_1789317069_3_prod/default/health.

From 04 Lab - Modularize PySpark Code

# Import necessary libraries

from pyspark.sql import functions as F

# Load the data and create a new column named trip_distance_km

new_taxi = (spark

.read

.table("samples.nyctaxi.trips")

.withColumn("trip_distance_km", F.round(F.col("trip_distance") * 1.60934, 2))

)

## Upper case all columns

new_taxi = new_taxi.select([F.col(col).alias(col.upper()) for col in new_taxi.columns])

## Save the table to the your catalog

(new_taxi

.write

.mode('overwrite')

.saveAsTable(f'{DA.catalog_name}.default.nyc_lab_solution_table')

)

## View the final table

display(spark.table(f'{DA.catalog_name}.default.nyc_lab_solution_table'))

[Daniel_B]

Okay - ignore this thread. The 03 Demo is actually a Lab even through it's called a Demo. The 04 Lab is a Lab as well.