Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-18-2021 01:00 PM
Here's the embedded links list!
Jobs scheduling and orchestration
- Built-in job scheduling: https://docs.databricks.com/jobs.html#schedule-a-job
- Periodic scheduling of the jobs
- Execute notebook / jar / Python script / Spark-submit
- Multitask Jobs
- Execute notebook / jar / Python script / Spark-submit
- Contrib module in Airflow
- Execute notebook / jar / Python script
Development interface resources
- Notebooks: https://docs.databricks.com/notebooks/index.html#notebooks
- More flexibility and control than the self-service git integration (Workspace CLI)
- Databricks REST API
- Databricks CLI - interface to the REST API
- Databricks Terraform Provider - create reproducible environments
- Databricks Connect - executing code on the Databricks cluster(s) from the local
- R Studio: https://docs.databricks.com/spark/latest/sparkr/rstudio.html
Testing Code
- Notebook driven: https://databricks.com/blog/2020/01/16/automate-deployment-and-testing-with-databricks-notebook-mlfl...
- CICD automated testing: https://docs.databricks.com/dev-tools/ci-cd.html
- Nutter library (Microsoft): https://github.com/microsoft/nutter
- spark-testing-base (Scala & Python support)
- spark-fast-tests (Scala, Spark 2 & 3)
- chispa (Python version of spark-fast-tests)
- pytest-spark (Python, native integration with pytest)
- Code samples for all libraries in one place
Source code repository resources
- Git integration: https://docs.databricks.com/notebooks/github-version-control.html#enable-and-disable-git-versioning
- Databricks CLI:
Code promotion resources