- 3515 Views
- 3 replies
- 3 kudos
Error while running the data engineering course notebook : "DE 2.2 - Providing Options for External Sources"
Can somebody help me fixing this problem? I am running this notebook on databricks community edition
- 3515 Views
- 3 replies
- 3 kudos
- 3 kudos
df = spark.read.format('parquet').load(path = datasource_path) df = df.select("*").toPandas() df.to_sql('users', conn, if_exists='replace', index = False)
- 3 kudos
- 2985 Views
- 2 replies
- 8 kudos
Implementing a data vault model in databricks can be challenging, but it can significantly improve the manageability of your data, particularly in hea...
Implementing a data vault model in databricks can be challenging, but it can significantly improve the manageability of your data, particularly in heavily regulated industries such as banking. While it may involve significant data duplication, duplic...
- 2985 Views
- 2 replies
- 8 kudos
- 1601 Views
- 1 replies
- 11 kudos
How to get reward points ?
How to get reward points ?
- 1601 Views
- 1 replies
- 11 kudos
- 11 kudos
You must share informative posts and be active in the community.Thank you
- 11 kudos
- 4423 Views
- 5 replies
- 0 kudos
connect rds from databricks sql editor
Is it possible to connect and execute query directly from rds in sql editor without using unity catelog
- 4423 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi there, Yes, you could do federated queries from DB SQL Editor. This is an experimental feature, though. UC is actually not supported. You can read more here:https://docs.databricks.com/query-federation/index.htmlPS: check out #DAIS2023 talks
- 0 kudos
- 2065 Views
- 2 replies
- 3 kudos
The workers in the cluster uses old end-of-life Ubuntu 18.04.
I create a cluster or a pool with the runtime version 12.2 LTS or even with the latest 13.0According to the documentation the worker should use an image. with Ubuntu 20.04 https://docs.databricks.com/release-notes/runtime/12.2.html#system-environment...
- 2065 Views
- 2 replies
- 3 kudos
- 3574 Views
- 3 replies
- 3 kudos
Resolved! Unable to take Lakehouse Fundamentals Quiz
Hi,I watched the videos for Lakehouse Fundamentals. However, when I click on "Take the quiz" it opens another tab that says I don't have permission to access the page.I've done all the necessary signing up.Could someone please help. Thanks!
- 3574 Views
- 3 replies
- 3 kudos
- 1868 Views
- 2 replies
- 2 kudos
Automated CI code checks using workflows when PR is raised
I'm familiar with Github Actions workflows to automate code checks whenever a PR is raised to a specified branch. For example for Python code, very useful is if unit tests (e.g. pytest), syntax (flake8), and code formatting (black formatter), type h...
- 1868 Views
- 2 replies
- 2 kudos
- 2 kudos
In a typical software development workflow (e.g. Github flow), a feature branch is created based on the master branch for feature development. A notebook can be synced to the feature branch via Github integration. Or a notebook can be exported from D...
- 2 kudos
- 2874 Views
- 2 replies
- 1 kudos
Trying to build subquery in Databricks notebook, similar to SQL in a data frame with the Top(1)
Hello Everyone,I am new to Databricks, so I am at the learning stage. It would be very helpful if someone helps in resolving the issue or I can say helped me to fix my code.I have built the query that fetches the data based on CASE, in Case I have a ...
- 2874 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello Rishabh,Thank you for your suggestion, we tried to limit 1 but the output values are coming the same for all the dates. which is not correct.
- 1 kudos
- 12691 Views
- 3 replies
- 0 kudos
How to fix the error on INVALID_PARAMETER_VALUE when using mlflow for tracking a yolo model training?
I'm new to databricks, and I'm trying to train yolo model and use mlflow to track the parameters and log the models. I keep getting this error related to the requirements.txt file path: INVALID_PARAMETER_VALUE: Invalid value '/Shared/YOLOv8/requireme...
- 12691 Views
- 3 replies
- 0 kudos
- 0 kudos
Thanks for the reply, @Suteja Kanuri​ . I tried the proposed solution. This time got this message:Invalid artifact path: '/Shared/YOLOv8'. Names may be treated as files in certain cases, and must not resolve to other names when treated as such. This ...
- 0 kudos
- 1294 Views
- 2 replies
- 6 kudos
Have you ever wondered how to automate your #databricks jobs and workflows without using the UI? If you want to manage your Databricks resources as co...
Have you ever wondered how to automate your #databricks jobs and workflows without using the UI? If you want to manage your Databricks resources as code, you should check out Terraform.Here is a simple example of creating a job that runs a notebook o...
- 1294 Views
- 2 replies
- 6 kudos
- 6 kudos
or use ADF Still waiting for actual added value on using Databricks Workflow over ADF.
- 6 kudos
- 2022 Views
- 3 replies
- 2 kudos
In the spirit of the Holiday season, share us a picture of reward/s you received from Databricks Community Rewards Store below! Â
In the spirit of the Holiday season, share us a picture of reward/s you received from Databricks Community Rewards Store below!
- 2022 Views
- 3 replies
- 2 kudos
- 9399 Views
- 2 replies
- 1 kudos
How to read data from Azure Log Analitycs ?
Hi guys,I need to read data from Azure Log Analitycs Workspace directaly, have any idea ?thank you
- 9399 Views
- 2 replies
- 1 kudos
- 5129 Views
- 6 replies
- 0 kudos
Resolved! change data feed on delta live tables
I have a delta live table where I am reading cdc data and merging this data in silver using apply changes. In silver can I find out what all data has changed since the last run similar to change data feed table_changes?
- 5129 Views
- 6 replies
- 0 kudos
- 0 kudos
I also have a requirment where i write to a live table (materialized view) and have cdf enabled i want to see the changes but here to i see overwrites happening after dlt pipeline runs
- 0 kudos
- 3217 Views
- 3 replies
- 2 kudos
Resolved! Data Science & Engineering Dashboard Refresh Issue Using Databricks
Hi everyone,I create a Data Science & Engineering notebook in databricks to display some visualizations and also set up a schedule for the notebook to run every hour. I can see that the scheduled run is successful every hour, but the dashboard I crea...
- 3217 Views
- 3 replies
- 2 kudos
- 2 kudos
To schedule a dashboard to refresh at a specified interval, schedule the notebook that generates the dashboard graphs.PS: Check #DAIS2023 talks
- 2 kudos
- 1937 Views
- 2 replies
- 1 kudos
Location of files previously uploaded on DBFS
I have uploaded a csv data file and used it in a spark job three months back. I am now running the same spark job with a new cluster created. Program is running properly. I want to know where I can see the previously uploaded csv data file.
- 1937 Views
- 2 replies
- 1 kudos
- 1 kudos
@Pranay Gupta​ you can see that in dbfs root directory, based on path you provided in job. please check .please go to data explorer and select below option that i shown in screen shot
- 1 kudos