cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
explorer
by • New Contributor III
  • 3557 Views
  • 4 replies
  • 1 kudos

Resolved! Deleting records manually in databricks streaming table.

Hi Team , Let me know if there is any ways I can delete records manually from databricks streaming table without corrupting table and data.Can we delete the few records (based on some condition) manually in databricks streaming table (having checkpoi...

Data Engineering
Reply
  • 3557 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
JunYang
Contributor
  • 1 kudos

  If you use the applyChanges method in DLT for Change Data Capture (CDC), you can delete records manually without affecting the consistency of the table, as applyChanges respects manual deletions. You must configure your DLT pipeline to respect manu...

  • 1 kudos
3 More Replies
mahesh_vardhan_
by • New Contributor
  • 3398 Views
  • 2 replies
  • 2 kudos

Resolved! How do I use numpy case when condition in pyspark.pandas?

I do have some legacy pandas codes which I want to migrate to spark to leaverage parellelization in Databricks. I see datadricks has launched a wrapper package on top of pandas which uses pandas nomenclature but use spark engine in the backend.I comf...

Data Engineering
Reply
  • 3398 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @mahesh vardhan gandhi​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from ...

  • 2 kudos
1 More Replies
sreedata
by • New Contributor III
  • 3566 Views
  • 4 replies
  • 7 kudos

Resolved! Getting status of "If Condition" Activity into a variable

"If Condition" has lot of activities that can succeeded or fail. If any activity fails then whole "If Condition" fails. I have to get the status of the "If Condition" activity (pass or fail) so that i can use it for processing in the next notebook t...

Data Engineering
Reply
  • 3566 Views
  • 4 replies
  • 7 kudos
View Replies
Latest Reply
UmaMahesh1
Honored Contributor III
  • 7 kudos

In your ADF Pipeline activity, set two different pipeline activities from the If condition activity based on success or failure (the green and red arrows). Then inside each pipeline activity, you can add a set variable, get variable and your adb note...

  • 7 kudos
3 More Replies
Mado
by • Valued Contributor II
  • 6964 Views
  • 6 replies
  • 2 kudos

Resolved! How to see if condition is True / False for all rows in a DataFrame?

Assume that I have a Spark DataFrame, and I want to see if records satisfy a condition.Example dataset:# Prepare Data data = [('A', 1), \ ('A', 2), \ ('B', 3) ]   # Create DataFrame columns= ['col_1', 'col_2'] df = spark.createDataF...

image image
Data Engineering
Reply
  • 6964 Views
  • 6 replies
  • 2 kudos
View Replies
Latest Reply
Ajay-Pandey
Esteemed Contributor III
  • 2 kudos

Hi you can use display() or show() function that will provide you expected results.

  • 2 kudos
5 More Replies
Labels