Hi All,I took the course: Get Started With Data Engineering from below course link https://www.databricks.com/learn/training/getting-started-with-data-engineering#data-videoBut, after completing the Quiz, I am not able to download Certificate. The a...
I am setting up pytest for my repo. I have my functions in separate python files and run pytest from one notebook. For each testing file, I have to create a new Spark session as follows:@pytest.fixture(scope="session")def spark(): spark = ( SparkSe...
I was able to do it by placing the Spark session fixture in the conftest.py file in the root directory.
I encountered numerous challenges during my exam, starting with issues related to system compatibility and difficulties with my microphone and other settings. Despite attempting to contact support multiple times, it was not easy to get assistance.Aft...
We have a team that works in Azure Databricks on notebooks.We are not allowed to push any data to Github per corporate policy.Instead of everyone having to always remember to clear their notebook outputs prior to commit and push, is there a way this ...
I’m not able to have any workspace view or cluster
Hey , could you be more precise with your question ?
I am doing a test run. I am uploading files to a volume and then using autoloader to ingesgt files and creating a table. I am getting this error message:-----------------------------------------------------------com.databricks.sql.cloudfiles.errors....
Hey, i think you are mixing DLT syntaxt with pyspark syntax:In DLT you should use:CREATE OR REFRESH STREAMING TABLE <table-name> AS SELECT * FROM STREAM read_files( '<path-to-source-data>', format => '<file-format>' )or in Python@dlt....
Hi,I have a SQL Warehouse, serverless mode, set to shut down after 5 minutes. Using the databricks web IDE, this works as expected. However, if I connect Power BI, import data to PBI and then leave the application open, the SQL Warehouse does not s...
Repeating this test today, the SQL Warehouse shut down properly. Thanks for your helpful reply.
Team,I need to migrate only specific notebooks which has changes committed to be pulled from one repo to another repoEnvironment/Repo Setup:Master -> Dev -> Feature Branch -> Developer commits the code in Feature Branch -> Dev has the changes from D...
I encountered an issue while using the Metabase JDBC driver to connect to Databricks SQL Warehouse: I noticed that the SQL Warehouse cluster is always running and never stops automatically. Every few seconds, a SELECT 1 query log appears, which I sus...
Will try to remove "preferredTestQuery" or add "idleConnectionTestPeriod" to avoid keep send select 1 query.
Hello,I am a professor of IT in the Price College of Business at The University of Oklahoma. In the past, I have used the Databricks Community Edition to demonstrate the principles of building and maintaining a data warehouse. This spring semester ...
Hello,When I call this function,mlflow.models.utils.add_libraries_to_model(MODEL_URI)It register a new model into the Model Registry. Is it possible to do the same but without registering a new model?Thanks,
I ended up publishing the library to AWS CodeArtifact repository. Now, how can I tell MLFlow to use AWS CodeArtifact private repository instead of PyPi?
Hello Community Members, We are thrilled to announce a fresh look for our Community platform! Get ready for an enhanced user experience with our brand-new UI changes. Our team has worked diligently to bring you a more intuitive and visually appealing...
Hi all,Using databricks-connect 11.3.19, I get an "java.lang.ClassCastException" when attempting to timetravel. The exact same statement works fine when executed in the databricks GUI directly. Any ideas on what's going on? Is this a known limitation...
0I have a Spark application (using Java library) which needs to replicate data from one blob storage to another. I have created a readStream() within it which is listening continuously to a Kafka topic for incoming events. The corresponding writeStre...
The problem was indeed with the way ClassLoader was being set in the ForkJoinPool (common Pool used) thread. Spark in SparkClassUtils uses Thread.currentThread().getContextClassLoader which might behave differently in another thread.To solve it I cre...
Hello Everyone, Is there a way we can execute the delete query from azure notebook on azure synapse database.I tried using read api method with option "query" but getting error like jdbc connector not able to handle code.Can any suggest how we can de...
