cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Christine
by • Contributor II
  • 7817 Views
  • 9 replies
  • 5 kudos

Resolved! pyspark dataframe empties after it has been saved to delta lake.

Hi, I am facing a problem that I hope to get some help to understand. I have created a function that is supposed to check if the input data already exist in a saved delta table and if not, it should create some calculations and append the new data to...

Data Engineering
Reply
  • 7817 Views
  • 9 replies
  • 5 kudos
View Replies
Latest Reply
SharathE
New Contributor III
  • 5 kudos

Hi,im also having similar issue ..does creating temp view and reading it again after saving to a table works?? /

  • 5 kudos
8 More Replies
shamly
by • New Contributor III
  • 5274 Views
  • 9 replies
  • 2 kudos

Resolved! need to remove doubledagger delimiter from a csv using databricks

My csv data looks like this‡‡companyId‡‡,‡‡empId‡‡,‡‡regionId‡‡,‡‡companyVersion‡‡,‡‡Question‡‡I tried this codedff = spark.read.option("header", "true").option("inferSchema", "true").option("delimiter", "‡,").csv(f"/mnt/data/path/datafile.csv")But I...

Data Engineering
Reply
  • 5274 Views
  • 9 replies
  • 2 kudos
View Replies
Latest Reply
UmaMahesh1
Honored Contributor III
  • 2 kudos

Hi @shamly pt​ I took a bit another approach since I guess no one would be sure of the the encoding of the data you showed. Sample data I took :‡‡companyId‡‡,‡‡empId‡‡,‡‡regionId‡‡,‡‡companyVersion‡‡,‡‡Question‡‡‡‡1‡‡,‡‡121212‡‡,‡‡R‡‡,‡‡1.0A‡‡,‡‡NA‡‡...

  • 2 kudos
8 More Replies
vanessafvg
by • New Contributor III
  • 3330 Views
  • 4 replies
  • 5 kudos

Resolved! How do I remove a filter from the visualisations pane in databricks sql, 2 i am able to remove but for some reason the Year filter doesn't have a delete option? The documentation isn't very comprehensive.

I want to remove one of these filters but there doesn't seem to be a way to do it.

image
Data Engineering
Reply
  • 3330 Views
  • 4 replies
  • 5 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 5 kudos

We're always here, even for newbie errors @Vanessa Van Gelder​ !Thanks for posting, and thanks @Hubert Dudek​ for always being so helpful.

  • 5 kudos
3 More Replies
Labels