Can I run jobs continuously without interruption on a permanent free (Community Edition) account?

[FastFoodBro]

Chào mọi người,

Tôi hiện đang sử dụng Databricks Community Edition (tài khoản miễn phí) và muốn hỏi về việc chạy các tác vụ theo lịch trình/liên tục trên đó.

Cụ thể:
1. Liệu có thể chạy một tác vụ liên tục (ví dụ: tác vụ xử lý dữ liệu trực tuyến hoặc tác vụ định kỳ theo lịch trình) trên phiên bản Cộng đồng miễn phí mà không bị gián đoạn không?
2. Có bất kỳ hạn chế nào về thời gian hoạt động của cụm máy chủ, tự động chấm dứt hoặc lập lịch tác vụ có thể ngăn cản việc chạy các tác vụ dài hạn hoặc 24/7 trên gói dịch vụ này không?
3. Nếu không hỗ trợ chạy liên tục, có giải pháp thay thế nào được khuyến nghị không, hay tôi cần nâng cấp lên gói trả phí để đạt được điều này?

Tôi muốn hiểu rõ các hạn chế trước khi thiết kế quy trình làm việc của mình, vì vậy bất kỳ sự giải thích nào từ cộng đồng hoặc nhóm Databricks đều sẽ được đánh giá cao.

Cảm ơn!

[FastFoodBro]

Hi everyone,

I'm currently using the Databricks Community Edition (free account) and I'd like to ask about running scheduled/continuous jobs on it.

Specifically:
1. Is it possible to run a job continuously (e.g., a streaming job or a recurring scheduled job) on the free Community Edition without interruption?
2. Are there any limitations on cluster uptime, auto-termination, or job scheduling that would prevent long-running or 24/7 jobs on this tier?
3. If continuous execution isn't supported, is there a recommended workaround, or would I need to upgrade to a paid plan to achieve this?

I want to understand the constraints before designing my workflow, so any clarification from the community or Databricks team would be greatly appreciated.

Thank you!

[data_pulse]

@FastFoodBro 
Good question. One important clarification: the old Databricks Community Edition has been replaced by Databricks Free Edition.

Free Edition does support Lakeflow Jobs and scheduling, but I would not design it for guaranteed 24/7 execution. Free Edition uses serverless compute and is subject to fair usage quotas. Databricks states that if the quota is exceeded, compute can be shut down for the rest of the day and Free Edition has no SLA or guaranteed reliability. It also currently allows a maximum of 5 concurrent job tasks per account.

Can find reference to all free edition limitations here

For recurring processing, a scheduled job is fine. For eg: with Structured Streaming on serverless, Databricks recommends Trigger.AvailableNow():

(spark.readStream
    .format("cloudFiles")
    .option("cloudFiles.format", "json")
    .load(source_path)
    .writeStream
    .trigger(availableNow=True)
    .option("checkpointLocation", checkpoint_path)
    .toTable("catalog.schema.target"))

You can then schedule that job every few minutes/hourly/daily depending on the requirement. AvailableNow processes everything that has arrived since the previous checkpoint and then stops. More details here

One caveat with serverless Jobs is that normal continuously running Structured Streaming triggers such as trigger(processingTime="10 seconds") are not supported. Serverless Jobs support bounded triggers such as AvailableNow. 

Summary: On Free Edition, the practical workaround is to use a scheduled job with AvailableNow, so each run processes newly arrived data and then stops. This works well for near-real-time or periodic ingestion, but it is not the same as keeping a streaming process alive 24/7.

If you need a reliably always-on workload, such as a continuously running streaming pipeline, then Free Edition is not the right tier. You would need a paid Databricks workspace and use an appropriate option such as Lakeflow Declarative Pipelines in Continuous mode or other supported compute.

[balajij8]

@FastFoodBro 

You can use Databricks Free account for personal use only (personal use, learning, experimentation - not for production or commercial use). You can upgrade to a paid plan to access full platform features as you will face interruptions in free account. More details here

[FastFoodBro]

How long can a job actually stay running? I'm using it for learning purposes, and I'm wondering how long a job that keeps a notebook running/active can be sustained for

[data_pulse]

Single serverless job run: up to 7 days maximum.
Free Edition: may stop earlier if you hit usage quota.

For learning better to design the notebook so it can restart safely, using checkpoints/state, rather than depending on one notebook staying alive indefinitely.

[FastFoodBro]

When a single job run finishes its 7-day maximum, can it be restarted to run for another 7 days, or will it get blocked and require an upgrade to a paid plan?

[data_pulse]

The 7 day limit is per serverless job run. Once a run reaches that limit, Databricks terminates it and does not automatically retry it, but the docs does not indicate that job is permanently blocked or requires an upgrade. You can start a new run, which gets its own runtime window within the fair usage quota limits.

Hope this helps.

[FastFoodBro]

Thanks, that's helpful! One follow-up question: how can I actually check or monitor my remaining fair usage quota?

Specifically:
1. Is there any dashboard, page, or API in the Free Edition workspace where I can see how much of my quota (compute hours, DBUs, etc.) I've used and how much is left?
2. What metric does the fair usage policy actually track — is it based on total compute runtime, number of concurrent jobs, DBU consumption, or something else?
3. Is there any warning or notification before the quota is exceeded and compute gets shut down, or does it just stop without notice?

I'd like to plan my job runs better to avoid unexpectedly hitting the limit. Thank you!

[FastFoodBro]

Thanks for the clarification! Could you share a concrete example of how to properly set up a notebook/job so it can restart safely using checkpoints/state?

Specifically:
1. What's the recommended pattern to structure a scheduled job so that when it stops (after the 7-day limit or due to quota), it can automatically pick up right where it left off on the next run?
2. Should I use Trigger.AvailableNow() with a checkpoint location for this, or is there a better approach for Free Edition?
3. Is there a way to schedule the job to auto-restart periodically (e.g., every few hours/daily) so it effectively maintains "continuous" processing without manual intervention?

A sample notebook structure or best-practice pattern for learning purposes would be very helpful. Thank you!

[data_pulse]

For learning, I would keep the notebook restartable than keeping it running continuously. Follow the below steps.

• Create Catalog, Schema, Volume by running it in a notebook first like:

Create a catalog
CREATE CATALOG IF NOT EXISTS learning_catalog;

Create a schema
CREATE SCHEMA IF NOT EXISTS learning_catalog.streaming_demo;

Create a managed volume
CREATE VOLUME IF NOT EXISTS learning_catalog.streaming_demo.streaming_files;

• Add this into a new notebook: streaming_example to use the above configured ones.

source_path = (
    "/Volumes/learning_catalog/streaming_demo/"
    "streaming_files/incoming"
)

checkpoint_path = (
    "/Volumes/learning_catalog/streaming_demo/"
    "streaming_files/checkpoints/orders"
)

target_table = "learning_catalog.streaming_demo.orders_bronze"

• Add this further into same notebook streaming_example for restartable streaming

(
    spark.readStream
        .format("cloudFiles")
        .option("cloudFiles.format", "json")
        .load(source_path)
        .writeStream
        .trigger(availableNow=True)
        .option("checkpointLocation", checkpoint_path)
        .toTable(target_table)
)

•  Then create a Lakeflow Job for that notebook streaming_example and schedule it every hour / few hours / daily. Follow this for reference to set up a job.

The above is to process json files from a volume location (So ensure you add some sample files or mechanism to periodically add new Json files to that volume location).

Each run processes everything that arrived since the previous checkpoint and then exits. On the next run, Structured Streaming reads the same checkpoint and continues from where it left off instead of starting again.