- 7730 Views
- 7 replies
- 2 kudos
Error while establishing JDBC connection to Azure databricks via HTTP proxy
I am using the databricks JDBC driver (https://databricks.com/spark/jdbc-drivers-download) to connect to Azure databricks.The connection needs to be routed through a HTTP proxy. I found parameters that can be configured for using the HTTP proxy:By pa...
- 7730 Views
- 7 replies
- 2 kudos
- 2 kudos
Hi @Jonas Minning​ , actually I am also having the same issue and when i looked into the driver related documentation I found that the driver currently only supports SOCKS proxies and I believe this is the reason why we are getting this error. So, I ...
- 2 kudos
- 2885 Views
- 3 replies
- 3 kudos
Error while running the data engineering course notebook : "DE 2.2 - Providing Options for External Sources"
Can somebody help me fixing this problem? I am running this notebook on databricks community edition
- 2885 Views
- 3 replies
- 3 kudos
- 3 kudos
df = spark.read.format('parquet').load(path = datasource_path) df = df.select("*").toPandas() df.to_sql('users', conn, if_exists='replace', index = False)
- 3 kudos
- 2343 Views
- 2 replies
- 8 kudos
Implementing a data vault model in databricks can be challenging, but it can significantly improve the manageability of your data, particularly in hea...
Implementing a data vault model in databricks can be challenging, but it can significantly improve the manageability of your data, particularly in heavily regulated industries such as banking. While it may involve significant data duplication, duplic...
- 2343 Views
- 2 replies
- 8 kudos
- 1218 Views
- 1 replies
- 11 kudos
How to get reward points ?
How to get reward points ?
- 1218 Views
- 1 replies
- 11 kudos
- 11 kudos
You must share informative posts and be active in the community.Thank you
- 11 kudos
- 3765 Views
- 5 replies
- 0 kudos
connect rds from databricks sql editor
Is it possible to connect and execute query directly from rds in sql editor without using unity catelog
- 3765 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi there, Yes, you could do federated queries from DB SQL Editor. This is an experimental feature, though. UC is actually not supported. You can read more here:https://docs.databricks.com/query-federation/index.htmlPS: check out #DAIS2023 talks
- 0 kudos
- 1760 Views
- 2 replies
- 3 kudos
The workers in the cluster uses old end-of-life Ubuntu 18.04.
I create a cluster or a pool with the runtime version 12.2 LTS or even with the latest 13.0According to the documentation the worker should use an image. with Ubuntu 20.04 https://docs.databricks.com/release-notes/runtime/12.2.html#system-environment...
- 1760 Views
- 2 replies
- 3 kudos
- 3108 Views
- 3 replies
- 3 kudos
Resolved! Unable to take Lakehouse Fundamentals Quiz
Hi,I watched the videos for Lakehouse Fundamentals. However, when I click on "Take the quiz" it opens another tab that says I don't have permission to access the page.I've done all the necessary signing up.Could someone please help. Thanks!
- 3108 Views
- 3 replies
- 3 kudos
- 1487 Views
- 2 replies
- 2 kudos
Automated CI code checks using workflows when PR is raised
I'm familiar with Github Actions workflows to automate code checks whenever a PR is raised to a specified branch. For example for Python code, very useful is if unit tests (e.g. pytest), syntax (flake8), and code formatting (black formatter), type h...
- 1487 Views
- 2 replies
- 2 kudos
- 2 kudos
In a typical software development workflow (e.g. Github flow), a feature branch is created based on the master branch for feature development. A notebook can be synced to the feature branch via Github integration. Or a notebook can be exported from D...
- 2 kudos
- 2290 Views
- 2 replies
- 1 kudos
Trying to build subquery in Databricks notebook, similar to SQL in a data frame with the Top(1)
Hello Everyone,I am new to Databricks, so I am at the learning stage. It would be very helpful if someone helps in resolving the issue or I can say helped me to fix my code.I have built the query that fetches the data based on CASE, in Case I have a ...
- 2290 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello Rishabh,Thank you for your suggestion, we tried to limit 1 but the output values are coming the same for all the dates. which is not correct.
- 1 kudos
- 10537 Views
- 3 replies
- 0 kudos
How to fix the error on INVALID_PARAMETER_VALUE when using mlflow for tracking a yolo model training?
I'm new to databricks, and I'm trying to train yolo model and use mlflow to track the parameters and log the models. I keep getting this error related to the requirements.txt file path: INVALID_PARAMETER_VALUE: Invalid value '/Shared/YOLOv8/requireme...
- 10537 Views
- 3 replies
- 0 kudos
- 0 kudos
Thanks for the reply, @Suteja Kanuri​ . I tried the proposed solution. This time got this message:Invalid artifact path: '/Shared/YOLOv8'. Names may be treated as files in certain cases, and must not resolve to other names when treated as such. This ...
- 0 kudos
- 1081 Views
- 2 replies
- 6 kudos
Have you ever wondered how to automate your #databricks jobs and workflows without using the UI? If you want to manage your Databricks resources as co...
Have you ever wondered how to automate your #databricks jobs and workflows without using the UI? If you want to manage your Databricks resources as code, you should check out Terraform.Here is a simple example of creating a job that runs a notebook o...
- 1081 Views
- 2 replies
- 6 kudos
- 6 kudos
or use ADF Still waiting for actual added value on using Databricks Workflow over ADF.
- 6 kudos
- 1677 Views
- 3 replies
- 2 kudos
In the spirit of the Holiday season, share us a picture of reward/s you received from Databricks Community Rewards Store below! Â
In the spirit of the Holiday season, share us a picture of reward/s you received from Databricks Community Rewards Store below!
- 1677 Views
- 3 replies
- 2 kudos
- 8368 Views
- 2 replies
- 1 kudos
How to read data from Azure Log Analitycs ?
Hi guys,I need to read data from Azure Log Analitycs Workspace directaly, have any idea ?thank you
- 8368 Views
- 2 replies
- 1 kudos
- 4228 Views
- 6 replies
- 0 kudos
Resolved! change data feed on delta live tables
I have a delta live table where I am reading cdc data and merging this data in silver using apply changes. In silver can I find out what all data has changed since the last run similar to change data feed table_changes?
- 4228 Views
- 6 replies
- 0 kudos
- 0 kudos
I also have a requirment where i write to a live table (materialized view) and have cdf enabled i want to see the changes but here to i see overwrites happening after dlt pipeline runs
- 0 kudos
- 2689 Views
- 3 replies
- 2 kudos
Resolved! Data Science & Engineering Dashboard Refresh Issue Using Databricks
Hi everyone,I create a Data Science & Engineering notebook in databricks to display some visualizations and also set up a schedule for the notebook to run every hour. I can see that the scheduled run is successful every hour, but the dashboard I crea...
- 2689 Views
- 3 replies
- 2 kudos
- 2 kudos
To schedule a dashboard to refresh at a specified interval, schedule the notebook that generates the dashboard graphs.PS: Check #DAIS2023 talks
- 2 kudos