Databricks Community

If your `pyproject.toml` file is not being picked up by Ruff in your Databricks notebooks, there are a few potential reasons and solutions to address the issue: Common Causes and Solutions 1. Ruff Version Compatibility:- Ensure you are using a recent...

To revalidate this, are you using Serverless SQL Warehouse or Serverless Compute which are different? Can you share an screenshot of the compute being used to make sure we are aligned?

Found the reason!  It's the runtime, it doesn't work on Databricks Runtime Version 15.4 LTS, but started to work after changing to 16.0.   Maybe this is something supported from the latest version?

I think in case you're deliberately installing databricks-connect, then you need to handle the local spark session creation.My issue is that I'm using databricks-dlt package which installs databricks-connect as a dependency. In the latest package ver...

As per serverless compute limitations I can see the following:  Task libraries are not supported for notebook tasks. Use notebook-scoped libraries instead. See Notebook-scoped Python libraries.

Did you also try cloning the cluster or using other cluster for the testing? The metastore down is normally a Hive Metastore issue, should not be impacting here, but you could check for more details on the error on the log4j under Driver logs.

I understand, yes it seems that this is currently not possible, only option will be to export your dataset as a csv file and import it in Databricks.

Hi @rai00, Ensure that the mock user me@example.com has the necessary privileges at both the catalog and schema levels. The user needs specific privileges such as USE_SCHEMA and CREATE_VOLUME   Use the WorkspaceClient to check the effective privilege...

@Alberto_Umana Thank you for the quick reply. But how are we to use the above, this looks like structured streaming with CDF mode.But currently our tables being in Unity catalog, finding the start version and end version is taking huge time as the ta...