- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-12-2024 10:59 PM
Hi @jiteshraut20 ,
If you are using CSV file please use the below code to convert it into delta.
spark.read .option("header", "true") .option("ignoreLeadingWhiteSpace", true) .option("ignoreTrailingWhiteSpace", true) .csv("/path/to/config.csv") .coalesce(1) .write.format("delta") .save("/myPath/overwatch/configs/prod_config")
Please validate the data which is present in "/myPath/overwatch/configs/prod_config" and use the below code to run the pipeline
import com.databricks.labs.overwatch.MultiWorkspaceDeployment
val configTable = "/myPath/overwatch/configs/prod_config" // Path to the config table
val tempLocation = "/tmp/overwatch/templocation"
MultiWorkspaceDeployment(configTable, tempLocation).deploy(1)