When using the JobsAPI you need to specify dbutils.notebook.exit("returnValue") to pass the results once the notebook finished it's job (https://docs.databricks.com/notebooks/notebook-workflows.html#notebook-workflows-exit).

Then you can get notebook_output.result through the API (https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsRunsGetOutput)