Daniel_B
Contributor

Sunday

This is the output of file /Workspace/Users/labuser16784881_1789317069@vocareum.com/devops-essentials-for-data-engineering-en_us-2.2.1/DevOps Essentials for Data Engineering/Course Notebooks/M02 - CI/03 Demo - Modularizing PySpark Code - REQUIRED

It all about health. Then look at the functions from /Workspace/Users/labuser16784881_1789317069@vocareum.com/devops-essentials-for-data-engineering-en_us-2.2.1/DevOps Essentials for Data Engineering/Course Notebooks/M02 - CI/04 Lab - Modularize PySpark Code which I have pasted after the output. The functions are about NY Taxi trips. It's all a mess and distracts from learning time.

 
From 03 Demo - Modularizing PySpark Code - REQUIRED
 
%run ../Includes/Classroom-Setup-03-REQUIRED
 
Created the volume: labuser16784881_1789317069.default.health. Created the volume: labuser16784881_1789317069_1_dev.default.health. Created the volume: labuser16784881_1789317069_2_stage.default.health. Created the volume: labuser16784881_1789317069_3_prod.default.health. ----Creating the CSV files in the necessary volumes for the course---- ---DELETE FILES IN THE HEALTH VOLUME WITHIN THE MAIN, DEV, STAGE, PROD CATALOGS---- Searching for files in /Volumes/labuser16784881_1789317069_3_prod/default/health volume to delete prior to creating files... No files found in /Volumes/labuser16784881_1789317069_3_prod/default/health.
From 04 Lab - Modularize PySpark Code
 
# Import necessary libraries
from pyspark.sql import functions as F

# Load the data and create a new column named trip_distance_km
new_taxi = (spark
.read
.table("samples.nyctaxi.trips")
.withColumn("trip_distance_km", F.round(F.col("trip_distance") * 1.60934, 2))
)


## Upper case all columns
new_taxi = new_taxi.select([F.col(col).alias(col.upper()) for col in new_taxi.columns])


## Save the table to the your catalog
(new_taxi
.write
.mode('overwrite')
.saveAsTable(f'{DA.catalog_name}.default.nyc_lab_solution_table')
)

## View the final table
display(spark.table(f'{DA.catalog_name}.default.nyc_lab_solution_table'))
 
 
 
 
 
0 Kudos