cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Trigger a Databricks job using sql query

Neo_Reeves
Visitor

Trying to run a SQL call function to trigger a Databricks job and am stuck with an error stating:
"The schema <schema_name>.system cannot be found. Verify the spelling and correctness of the schema and catalog. If you did not qualify the name with a catalog, verify the current_schema() output, or qualify the name with the correct catalog. To tolerate the error on drop use DROP SCHEMA IF EXISTS. SQLSTATE: 42704".

 
The query that I am trying to run:
 
%sql
CALL test_catalog.system.run_job("load_table")

 
2 REPLIES 2

ManojkMohan
Honored Contributor II

@Neo_Reeves 

The error is due to the non-existence or incorrect reference to the "system" schema under your catalog. Check the correct catalog and schema names where the stored procedure run_job resides, and update the CALL statement accordingly.

CALL test_catalog.system.run_job("load_table")

Verify that the schema "system" truly exists in the catalog test_catalog

ManojkMohan_0-1764150591502.png

If "run_job" is a stored procedure or SQL function, be sure it exists under the correct catalog and schema name.

if you intended to execute a stored procedure or run a job, confirm that it is created in a schema you have access to and specify that schema instead of "system"

Coffee77
Contributor III

When you say "Databricks Job", do you mean a custom SQL code you created in an stored procedure or function? Saying this because if you're really trying to run a real Databricks Job (lakeflow), it would be much better to establish another strategy different from that initial one. I mean, create firstly the Databricks Job and then, add type of tasks better for you scenario (https://docs.databricks.com/aws/en/jobs/configure-task#types-of-tasks )


Lifelong Learner Cloud & Data Solution Architect | https://www.youtube.com/@CafeConData