cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
swatish0395
by • New Contributor III
  • 3242 Views
  • 3 replies
  • 4 kudos

Resolved! how to create a scala jar using db notebook and save it in a file path inside databricks

I have scala function as below, i am unable to understand how to write a scala jar with the same, please find below code i have used Enforcing Column-Level Encryption - Databrick %scala import com.macasaet.fernet.{Key, StringValidator, Token}import o...

Data Engineering
Reply
  • 3242 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
swatish0395
New Contributor III
  • 4 kudos

I had to finally create the jar using teh intellij and sbt iconfiguration on the same env. and then installed the jar in the cluster it worked

  • 4 kudos
2 More Replies
Joey
by • New Contributor II
  • 11725 Views
  • 3 replies
  • 0 kudos

How to fix the error on INVALID_PARAMETER_VALUE when using mlflow for tracking a yolo model training?

I'm new to databricks, and I'm trying to train yolo model and use mlflow to track the parameters and log the models. I keep getting this error related to the requirements.txt file path: INVALID_PARAMETER_VALUE: Invalid value '/Shared/YOLOv8/requireme...

Data Engineering
Reply
  • 11725 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Joey
New Contributor II
  • 0 kudos

Thanks for the reply, @Suteja Kanuri​ . I tried the proposed solution. This time got this message:Invalid artifact path: '/Shared/YOLOv8'. Names may be treated as files in certain cases, and must not resolve to other names when treated as such. This ...

  • 0 kudos
2 More Replies
uzairm
by • New Contributor III
  • 5613 Views
  • 2 replies
  • 1 kudos

My whole code is running on driver node, I want my code to run on worker nodes so that the memory of driver node is not exhausted. Please tell me improvement is my codes. My spark crashes frequently when the pulled data from s3 is huge.

I am running process which has 4 steps.Querying s3 file paths from dynamo DB based on certain parameters given by user. (function to do so provided by client, just have to import). Returns a list of filesCheck if those file paths have already been qu...

Data Engineering
Reply
  • 5613 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Vartika
Databricks Employee
  • 1 kudos

Hi @uzair mustafa​ Thank you for posting your question in our community! We are happy to assist you.Does @Suteja Kanuri​'s answer help? If it does, would you be happy to mark it as best?This will help other community members who may have similar ques...

  • 1 kudos
1 More Replies
Ank
by • New Contributor II
  • 1206 Views
  • 1 replies
  • 2 kudos

Why am I getting a FileNotFoundError after providing the file path?

I used copy file path to get the file path of the notebook I am trying to run from another notebook.file_path = "/Users/ankur.lohiya@workday.com/PAS/Training/Ingest/TrainingQueries-Cloned.py/"ddi = DatabricksDataIngestion(file_path=file_path,        ...

Data Engineering
Reply
  • 1206 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
Vidula
Honored Contributor
  • 2 kudos

Hello @Ankur Lohiya​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...

  • 2 kudos
Labels