Databricks Community

pavlosskev · 08-01-2024

This is not a question, this is just the solution to a problem we encountered in case someone from the community finds it useful.Recently we encountered an issue, where our users' jobs started failing out of nowhere on the following command, with the...

pavlosskev · 10-25-2024

Fermin_vicente's answer is pretty good already. Below is how you can do something similar with conftest.py# conftest.py import pytest from unittest.mock import MagicMock from pyspark.sql import SparkSession @pytest.fixture(scope="session") def dbuti...

pavlosskev · 08-15-2024

I don't clearly understand your full problem. But I do know the following regarding UDFs:1. PySpark UDFs are extremely slow, because it needs to deserialize the Java Object (DataFrame), transform it with the Python UDF, then Serialize it back. This h...

pavlosskev · 08-02-2024

If the validation is fine on your colleague's laptop and not on yours, my first assumption would be that it's a version issue. Do you have the same Databricks CLI version as your colleagues? You can check with databricks --version Also according to...

Databricks Community

User Stats

User Activity

Databricks java.util.NoSuchElementException: None.get Error on "SHOW TABLES IN" command

Re: Python mocking dbutils in unittests

Re: How to solve udf performance issue with databricks sql function?

Re: Complex type variable in Databricks.yml not working