Databricks Community

dylan19 · 3 weeks ago

How can you control whether a file is a Notebook or a Query? Long story short - can you do anything to make Databricks recognise a .dbquery.ipynb file as a query and open it in SQL Warehouse?

I created a file in my git folder as a query. All good. Then deployed it into another workspace using a DAB. However, when I open the query file in the new workspace, it opens as a notebook. However, if I open through a git folder in new workspace, it does open in SQL Warehouse. I can't seem to get the DAB-deployed location to open as a query in SQL Warehouse though, even if selected SQL Editor as the context, etc.

The kicker is I need to deploy into a 3rd workspace where (separation of responsibilities) the Databricks user profile won't have access to git...
Is it as dumb as "you need to copy all the content from workbook into query?"

The main difference is the (new) SQL Warehouse view will show the output of many queries, whereas a notebook only shows the last.

Khaja_Zaffer · 3 weeks ago

Hello @dylan19

Good day!!

Thats very frustrating! well just to check

Did you change the .dbquery.ipynb file to a .sql file in your Git repository. For example, extract the SQL content from the .ipynb file (a JSON format) and saved it as plain text in a .sql file? ( I am not sure on this, just sharing my ideas may be others also confirm on this) I am also curious to know what could be solution for this issue.

Just checking?

Is your DAB config is like this?

resources:
  jobs:
    my_sql_job:
      name: RunSQLQuery
      tasks:
        - task_key: run_sql
          sql_task:
            warehouse_id: "<your-sql-warehouse-id>"
            sql_file:
              source: workspace
              path: "/Workspace/Path/To/Your/file.sql"

dylan19 · 3 weeks ago

Hi @Khaja_Zaffer ,

No, I did not change the file extension at all. Given that this was working as a query from the git folder in the UI, presumed that the extension was ok.

You might be onto something with the DAB config though. I can't see the specific mechanism that is deploying the file, but the folder was crated from the default-python template. It looks suspiciously like that may be relevant? The file is in src sub-tree, so is being deployed by location. There is no task configured for this file.

The issue is that when the file is opened in the UI from the .bundle location, it is opening as a notebook rather than a query.

CI/CD workflow is: develop in dev Databricks UI and push to git. Deploy DAB bundle from DevOps to dev, test and prod. This works fine for notebooks 🙂