cancel
Turn on suggestions
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
nolanreilly
by • New Contributor II
  • 981 Views
  • 1 replies
  • 1 kudos

Impossible to read a custom pipeline? (Scala)

I have created a custom transformer to be used in a ml pipeline. I was able to write the pipeline to storage by extending the transformer class with DefaultParamsWritable. Reading the pipeline back in however, does not seem possible in Scala. I have...

Data Engineering
Reply
  • 981 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
WarrenO
New Contributor III
  • 1 kudos

Hi, did you ever find a solution for this?

  • 1 kudos
Nik
by • New Contributor III
  • 15413 Views
  • 19 replies
  • 0 kudos

write from a Dataframe to a CSV file, CSV file is blank

Hi i am reading from a text file from a blob val sparkDF = spark.read.format(file_type) .option("header", "true") .option("inferSchema", "true") .option("delimiter", file_delimiter) .load(wasbs_string + "/" + PR_FileName) Then i test my Datafra...

Data Engineering
Reply
  • 15413 Views
  • 19 replies
  • 0 kudos
View Replies
Latest Reply
nl09
New Contributor II
  • 0 kudos

Create temp folder inside output folder. Copy file part-00000* with the file name to output folder. Delete the temp folder. Python code snippet to do the same. fpath=output+'/'+'temp' def file_exists(path): try: dbutils.fs.ls(path) return...

  • 0 kudos
18 More Replies
Labels
li.common.scroll-to.top