How are you running the notebooks in production?

[Anonymous]

@Deepak Kini​ :

Databricks provides multiple ways to run notebooks in production, depending on the use case and requirements. Here are some common ways to run notebooks in production:

1. Scheduled Jobs: This is the most common way to run notebooks in production. You can schedule a notebook to run at a specific time or interval using the Databricks Jobs feature. You can configure the notebook input parameters, cluster, and other settings in the Job configuration, and the notebook will run automatically based on the schedule. You can monitor the job status, view logs, and configure alerts for job failures.
2. REST API: You can use the Databricks REST API to run a notebook programmatically. This is useful if you need to integrate notebook execution with other systems or workflows. You can specify the notebook path, input parameters, and other settings in the API request, and the notebook will run on a cluster. You can retrieve the output of the notebook as well.
3. Databricks CLI: You can use the Databricks command-line interface (CLI) to run a notebook from a script or a terminal. This is useful for ad-hoc or interactive tasks, or for running notebooks from a workflow management system. You can specify the notebook path, input parameters, and other settings in the CLI command, and the notebook will run on a cluster. You can retrieve the output of the notebook as well.
4. Notebook Workflows: You can use the Databricks Notebook Workflows feature to create complex workflows that involve multiple notebooks and other tasks. You can define the dependencies and execution order of the notebooks, and Databricks will automatically manage the execution. This is useful for building end-to-end data pipelines or workflows.
5. Delta Live Tables: You can use Delta Live Tables to run continuous queries on streaming or batch data in real-time. You can define a notebook that contains the query logic, and Delta Live Tables will execute the notebook on a cluster and continuously update the query results as new data arrives.

These are some of the common ways to run notebooks in production in Databricks.

[Anonymous]

Hi @Deepak Kini​ 

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!