cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
alejandrofm
by • Valued Contributor
  • 3139 Views
  • 3 replies
  • 1 kudos

Can't enable CLI 2.1 on CI

Hi! this is my CI configuration, I added the databricks jobs configure --version=2.1 command but it stills showing this error, any idea of what can I be doing wrong?Error:Resetting Databricks Job with job_id 1036...WARN: Your CLI is configured to use...

Data Engineering
Reply
  • 3139 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
karthik-kandiko
New Contributor II
  • 1 kudos

I got to solve this by downgrading the Databricks runtime to 13.3 and had the below commands for optimization and it worked well in my case.spark.conf.set("spark.sql.shuffle.partitions", "200")spark.conf.set("spark.sql.execution.arrow.pyspark.enabled...

  • 1 kudos
2 More Replies
Oliver_Angelil
by • Valued Contributor II
  • 1962 Views
  • 2 replies
  • 2 kudos

Automated CI code checks using workflows when PR is raised

I'm familiar with Github Actions workflows to automate code checks whenever a PR is raised to a specified branch. For example for Python code, very useful is if unit tests (e.g. pytest), syntax (flake8), and code formatting (black formatter), type h...

Data Engineering
Reply
  • 1962 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Priyag1
Honored Contributor II
  • 2 kudos

In a typical software development workflow (e.g. Github flow), a feature branch is created based on the master branch for feature development. A notebook can be synced to the feature branch via Github integration. Or a notebook can be exported from D...

  • 2 kudos
1 More Replies
maranBH
by • New Contributor III
  • 2338 Views
  • 3 replies
  • 1 kudos

Resolved! Trained model artifact, CI/CD and Databricks without MLFlow.

Hi all,We are constructing our CI/CD pipelines with the Repos feature following this guide:https://databricks.com/blog/2021/09/20/part-1-implementing-ci-cd-on-databricks-using-databricks-notebooks-and-azure-devops.htmlI'm trying to implement my pipes...

Data Engineering
Reply
  • 2338 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
sean_owen
Databricks Employee
  • 1 kudos

So you are managing your models with MLflow, and want to include them in a git repository?You can do that in a CI/CD process; it would run the mlflow CLI to copy the model you want (e.g. model:/my_model/production) to a git checkout and then commit i...

  • 1 kudos
2 More Replies
Labels