Dbx installation for local development on Vscode
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ11-09-2022 10:05 PM
Hi Folks,
Since databricks is now asking to use DBX instead of databricks-connect ,We are trying to set up our local environment following the guide.
dbx by Databricks Labs - Azure Databricks | Microsoft Learn
Have create conf/deployment.yml and dbx/project.json and added to root folder of my repo.
Deployment file looks like below
build:
python: "pip"
environments:
default:
workflows:
- name: "dbx-xxxx-job"
spark_python_task:
python_file: "C:\framework\library\py\platformdata\tests\test_dbutils.py"
And I am getting below error while running
dbx execute workflowname -- cluster name =""
Any suggestion here?
โ
โ C:\Python\lib\site-packages\yaml\scanner.py:1149 in scan_flow_scalar โ
โ โ
โ 1146 โ โ start_mark = self.get_mark() โ
โ 1147 โ โ quote = self.peek() โ
โ 1148 โ โ self.forward() โ
โ โฑ 1149 โ โ chunks.extend(self.scan_flow_scalar_non_spaces(double, start_mark)) โ
โ 1150 โ โ while self.peek() != quote: โ
โ 1151 โ โ โ chunks.extend(self.scan_flow_scalar_spaces(double, start_mark)) โ
โ 1152 โ โ โ chunks.extend(self.scan_flow_scalar_non_spaces(double, start_mark)) โ
โ โ
โ C:\Python\lib\site-packages\yaml\scanner.py:1223 in scan_flow_scalar_non_spaces โ
โ โ
โ 1220 โ โ โ โ โ self.scan_line_break() โ
โ 1221 โ โ โ โ โ chunks.extend(self.scan_flow_scalar_breaks(double, start_mark)) โ
โ 1222 โ โ โ โ else: โ
โ โฑ 1223 โ โ โ โ โ raise ScannerError("while scanning a double-quoted scalar", start_ma โ
โ 1224 โ โ โ โ โ โ โ "found unknown escape character %r" % ch, self.get_mark()) โ
โ 1225 โ โ โ else: โ
โ 1226 โ โ โ โ return chunks
- Labels:
-
Local Development
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ11-18-2022 07:01 AM
hi @Ritu Kumariโ
Is this the full error message ?
BTW, at least the python file path is not compliant: File references - dbx
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ11-21-2022 07:26 AM
fixed this issue. but I am getting another issue while syncing local repo with Workspace in Databricks UI.
When I run command
dbx sync repo -d workspace name --source.
Command runs fine . I can see that dbfs but not under workspace in Databricks page.
Any suggestion here
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ11-21-2022 09:10 AM
hmm... `dbx sync repo -d [repo_name]` works well from my side
coudl you please check:
- use the latest dbx version, currently 0.8.7
- it should be under Repos menu, not Workspace
- if your files are not delcared in the gitignore, check the help of `dbx sync repo --help`, there're multiple exclusion settings
- default behavior of `dbx sync repo` keeps watching the files changes, you can modifiy a file, and check if you see in the output, sth similar to my example that changed the file `__ini__.py`

