I am building a bronze table with CDF-enables in these steps:Initially, Reading json file from landing zone and write to table locationdf = spark.readStream.format("cloudFiles") \
.option("cloudFiles.schemaLocation", <schema_loc>) \
.option("clou...
HiOn December 28th ,I attempted the Databricks Certified Machine Learning Professional exam for 1st time , unfortunately I ended up by failing grade. My passing grade was 70%, and I received 68.33%.I am planning to reattempt the exam, Could you kindl...
Getting below error Context: Using Databricks shared interactive cluster for scheduled run multiple parallel jobs at the same time after every 5 mins. When I check Ganglia, driver node's memory reaches almost max and then restart of driver happens an...
I am using the databricks JDBC driver (https://databricks.com/spark/jdbc-drivers-download) to connect to Azure databricks.The connection needs to be routed through a HTTP proxy. I found parameters that can be configured for using the HTTP proxy:By pa...
Hi @Jonas Minning​ , actually I am also having the same issue and when i looked into the driver related documentation I found that the driver currently only supports SOCKS proxies and I believe this is the reason why we are getting this error. So, I ...
Implementing a data vault model in databricks can be challenging, but it can significantly improve the manageability of your data, particularly in heavily regulated industries such as banking. While it may involve significant data duplication, duplic...
Hi , We are capturing the exception if an error occurs using try except. But we want the job status to be failed once we got the exception. Whats the best way to do that. We are using pyspark.
Instead of exiting the notebook which make the task/job success, Exception objects needs to be raised again from Exception block to fail the job.try: <you code>except Exception as err: <your block of exception handling> raise err
Hi there, Yes, you could do federated queries from DB SQL Editor. This is an experimental feature, though. UC is actually not supported. You can read more here:https://docs.databricks.com/query-federation/index.htmlPS: check out #DAIS2023 talks
I create a cluster or a pool with the runtime version 12.2 LTS or even with the latest 13.0According to the documentation the worker should use an image. with Ubuntu 20.04 https://docs.databricks.com/release-notes/runtime/12.2.html#system-environment...
Hi,I watched the videos for Lakehouse Fundamentals. However, when I click on "Take the quiz" it opens another tab that says I don't have permission to access the page.I've done all the necessary signing up.Could someone please help. Thanks!
I'm familiar with Github Actions workflows to automate code checks whenever a PR is raised to a specified branch. For example for Python code, very useful is if unit tests (e.g. pytest), syntax (flake8), and code formatting (black formatter), type h...
In a typical software development workflow (e.g. Github flow), a feature branch is created based on the master branch for feature development. A notebook can be synced to the feature branch via Github integration. Or a notebook can be exported from D...
Hello Everyone,I am new to Databricks, so I am at the learning stage. It would be very helpful if someone helps in resolving the issue or I can say helped me to fix my code.I have built the query that fetches the data based on CASE, in Case I have a ...
I'm new to databricks, and I'm trying to train yolo model and use mlflow to track the parameters and log the models. I keep getting this error related to the requirements.txt file path: INVALID_PARAMETER_VALUE: Invalid value '/Shared/YOLOv8/requireme...
Thanks for the reply, @Suteja Kanuri​ . I tried the proposed solution. This time got this message:Invalid artifact path: '/Shared/YOLOv8'. Names may be treated as files in certain cases, and must not resolve to other names when treated as such. This ...
Have you ever wondered how to automate your #databricks jobs and workflows without using the UI? If you want to manage your Databricks resources as code, you should check out Terraform.Here is a simple example of creating a job that runs a notebook o...
