topic Override ruff linter settings for notebook cells in Data Engineering

Override ruff linter settings for notebook cells

rgooch_cfa — Tue, 17 Dec 2024 19:02:03 GMT

How can I override the ruff linter settings for my notebooks?

I have various projects/git folders in my workspace, and oftentimes, they represent different teams and thus different sets of code formatting patterns. I would like to override the default ruff settings on a project-by-project basis using `pyproject.toml` files in the project directories, similar to how I can do this locally. However, when these files are added, the ruff linter still ignores my specific settings.

I have tried my own ideas + looked for documentation but I can't find the info. Thanks for your help!

Re: Override ruff linter settings for notebook cells

Louis_Frolio — Tue, 17 Dec 2024 20:46:53 GMT

Steps to Ensure Ruff Reads `pyproject.toml`

1. Place `pyproject.toml` in the Project Root:
- Ruff automatically detects configuration files like `pyproject.toml`, `ruff.toml`, or `.ruff.toml` when they are located at the root of the project directory. Ensure your `pyproject.toml` is in the correct directory for each project.

2. Define Ruff Settings in `pyproject.toml`:
- Use the `[tool.ruff]` section to define your desired settings. For example:
```toml
[tool.ruff]
line-length = 120
select = ["E", "F", "W"]
ignore = ["E501"]
```
- This ensures Ruff uses these specific rules and ignores others.

3. Verify Configuration Detection:
- Run Ruff manually using the command:
```bash
ruff check . --config pyproject.toml
```
- This explicitly tells Ruff to use your configuration file. If you see a message like `Using pyproject.toml at /path/to/project`, it confirms that Ruff is reading the file.

4. Handle Workspace or Global Overrides:
- In environments like Databricks or VSCode, global or workspace-level configurations may override project-specific settings. For instance:
- In VSCode, check if `ruff.format.args` or similar settings are overriding your `pyproject.toml`. You can unset these or explicitly specify the path to your configuration file.
- In Databricks, ensure no global Ruff configurations are conflicting.

5. Pass Configuration Explicitly (if Necessary):
- If automatic detection fails, you can pass the configuration file explicitly in commands or pre-commit hooks:
```bash
ruff check . --config=/path/to/pyproject.toml
```
- For pre-commit hooks, update the arguments to include:
```yaml
args: ["--config=pyproject.toml"]
``'

6. Check for Nested Configurations:
- If you have multiple configuration files (e.g., one at a parent directory and another at a subdirectory), Ruff might prioritize one over the other. Ensure only the intended file exists for each project.

Re: Override ruff linter settings for notebook cells

rgooch_cfa — Tue, 17 Dec 2024 21:07:43 GMT

Thanks for following up. I was following this documentation and the custom settings weren't picked up in my Databricks notebooks in the project directory. I've placed the `pyproject.toml` in the project root, but the ruff tooltip still appears (see screenshot)

Re: Override ruff linter settings for notebook cells

rgooch_cfa — Wed, 18 Dec 2024 21:03:54 GMT

So my takeaway after moving the `pyproject.toml` file location throughout my workspace is that it's impossible to apply custom settings to the ruff formatter in Databricks notebooks.

Please confirm that it is impossible to use custom settings, or if not, I would love guidance on how to make this work.

Thanks

Re: Override ruff linter settings for notebook cells

Louis_Frolio — Wed, 18 Dec 2024 22:05:43 GMT

If your `pyproject.toml` file is not being picked up by Ruff in your Databricks notebooks, there are a few potential reasons and solutions to address the issue:

Common Causes and Solutions

1. Ruff Version Compatibility:
- Ensure you are using a recent version of Ruff (preferably 0.5.1 or later). Earlier versions had issues with reliably detecting `pyproject.toml` configurations automatically.

2. Explicitly Specify the Configuration File:
- If Ruff is not automatically detecting your `pyproject.toml`, you can explicitly pass the configuration file path using the `--config` flag when running Ruff. For example:
```bash ruff check . --config=path/to/pyproject.toml
```
- This ensures that Ruff uses the specified configuration file regardless of its automatic detection logic.

3. Correct Placement of `pyproject.toml`:
- Verify that the `pyproject.toml` file is located in the root directory of your project. Ruff treats the directory containing the configuration file as the "project root" and applies rules accordingly.

4. Databricks-Specific Considerations:
- Databricks environments may not automatically apply Ruff configurations to notebooks. If you are running Ruff as part of a CI/CD pipeline or through a script, ensure that the command explicitly references the configuration file.
- For interactive notebook usage, consider running Ruff commands manually or integrating them into your workflow with explicit configuration paths.

5. Alternative Configuration Files:
- If `pyproject.toml` is not working as expected, try using a `ruff.toml` or `.ruff.toml` file instead. These files do not require the `[tool.ruff]` prefix and may simplify configuration.

6. Restarting Services:
- If you are using a language server (e.g., Ruff's LSP or an IDE integration), restart the server after adding or modifying your `pyproject.toml`. Some services cache settings and need to be refreshed to pick up changes[4][9].

7. Debugging Configuration Issues:
- Use the following command to verify which configuration file Ruff is using:
```bash ruff config path/to/file
```
- This command lists all active settings and can help identify whether your custom rules are being applied.

Example Configuration in `pyproject.toml`
Ensure your `pyproject.toml` includes a valid `[tool.ruff]` section, like this:
```toml
[tool.ruff]
select = ["E", "F", "W"]
ignore = ["E501"]
per-file-ignores = {
"__init__.py" = ["F401"],
"tests/*.py" = ["D"]
}
```

Additional Notes for Databricks
- If you are working with Jupyter Notebooks in Databricks, ensure that you have enabled Ruff's notebook linting support (available in version 0.6.0 and higher). You can configure this via the `exclude` or `per-file-ignores` options for `.ipynb` files.

Re: Override ruff linter settings for notebook cells

GalenSwint — Tue, 22 Apr 2025 04:50:19 GMT

How does databricks define a "project" and does it work with ruff.toml files or does it have to be pyproject.toml? I can't seem to get it to adopt settings either, and Ctrl+Shift+F for formatting doesn't seem to do anything.

Trying to run ruff manually seems to result in errors but i nevertheless get little ruff popups so it's *somewhere*

Re: Override ruff linter settings for notebook cells

Louis_Frolio — Tue, 22 Apr 2025 11:26:10 GMT

Databricks defines a “project” as a structured collection of assets (jobs, pipelines, notebooks) organized using Databricks Asset Bundles, which use infrastructure-as-code templates to standardize deployments. Hope this helps. Lou.