- 1234 Views
- 1 replies
- 0 kudos
Issue with VSCode Extension and Databricks Cluster Using Docker Image
I've encountered a significant issue while using the VSCode extension for Databricks, particularly when working with a cluster configured with a Docker image. Here's a detailed description of the problem:Problem DescriptionWhen attempting to upload a...
- 1234 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Retired_mod thanks for a such quick response.Actually, I am using the Dockerfile from the Databricks runtime example here: https://github.com/databricks/containers/blob/master/ubuntu/minimal/Dockerfile . The configuration with the VSCode extensio...
- 0 kudos
- 1091 Views
- 2 replies
- 0 kudos
RDD Issue in Unity Catalog
################################################ New Code as per UC ################################################ def parse_json(df, *cols, clean=True res = df for i in cols: if clean: res = ( res.wit...
- 1091 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @sambgp , Thank you for reaching out to our community! We're here to help you. To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedback...
- 0 kudos
- 1097 Views
- 4 replies
- 2 kudos
Databricks All Delta Tables Data Read
If we want to read all the data of the databricks tables at single time how can we able to do it.
- 1097 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @Krishna2110 ,Here it is, it should work now tables = spark.sql("SHOW TABLES IN ewt_edp_prod.crm_raw").collect() for row in tables: table_name = f"ewt_edp_prod.{row[0]}.{row[1]}" try: df = spark.table(table_name) count = df...
- 2 kudos
- 3148 Views
- 2 replies
- 2 kudos
Resolved! Convert Date String to Date type Returning null
Hi,I am using Databricks SQL and I am converting a integer field which is of format ('20240719' or 'yyyyMMdd'),now I am able to convert it to date type using to_date(forecastedHorizonPeriodDate,'yyyyMMdd')I then tried to change the format of this dat...
- 3148 Views
- 2 replies
- 2 kudos
- 3778 Views
- 1 replies
- 0 kudos
Login issue
Hi,Getting below error message.We were not able to find a Community Edition workspace with this email. Please login to non-community-edition workspaces you may have access to.
- 3778 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @virendra1212 ,Take a look at below thread. Maybe some of the suggestions will help in your case.Community Edition Login Issues Below is a list of ... - Page 4 - Databricks Community - 26926You signed up before May 14, 2022 -Please confirm that th...
- 0 kudos
- 602 Views
- 0 replies
- 0 kudos
Shuffle Partitions
What if I have lot of empty shuffled partitions due to data skewness Secondly , if the shuffle partition size is 128 MB and if the size of the key's partition is 700 MB
- 602 Views
- 0 replies
- 0 kudos
- 1113 Views
- 3 replies
- 4 kudos
The way to visualize logging of Spark Job graphically
Is there any way to visualize logging such as execution runtime or memory usage of Spark Job graphically just like the image below by utilizing Databricks Partner Connect for free?Also, I'd appreciate to know any other ways to visualize the logging o...
- 1113 Views
- 3 replies
- 4 kudos
- 3860 Views
- 1 replies
- 0 kudos
How to Add value to Comment Column in Databricks Catalog View for Foreign table (Azure SQL)
Hi.Struggling to add description by script to the comment column within the catalog view in Databricks, particularly for foreign/external tables sourced from Azure SQL.I have no issue doing that for delta tables. Also for information schema columns f...
- 3860 Views
- 1 replies
- 0 kudos
- 0 kudos
@Retired_mod Sorry, I did read this way back but forgot to reply.Thanks for information.And all those steps I found as well and for "3. Programmatic Approach (For Azure SQL)" if you are adding to sql any description that way I don't think it will app...
- 0 kudos
- 1669 Views
- 2 replies
- 0 kudos
Resolved! DLT Online Table with VNnet Enable on Blob Storage Get 403 Issue
I am trying to create an online table in a Unity catalog. However, I get a GET, 403 error. DataPlaneException: Failed to start the DLT service on cluster . Please check the stack trace below or driver logs for more details. com.databricks.pipelines....
- 1669 Views
- 2 replies
- 0 kudos
- 0 kudos
I figured it out. It was because of the Network Connectivity Configurations. I did not have one setup with a private endpoint connection to the ADLS Gen2. I followed the instructions here: https://learn.microsoft.com/en-us/azure/databricks/security/...
- 0 kudos
- 671 Views
- 2 replies
- 0 kudos
Issues around float datatypes
Float data from file, is never getting copied to the temp_table created though the schema matches.As a workaround, using CREATE TABLE [USING] which is able to insert the file data to a temp_table. Is this a known issue? COPY INTO issue is explained w...
- 671 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @inagar , Thank you for reaching out to our community! We're here to help you. To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedback...
- 0 kudos
- 1469 Views
- 2 replies
- 2 kudos
Learning via Free Trial Azure
Hi I am following a Databricks course on Udemy and the course instructed to access Databricks via the free trial of Azure. Once I created my account on Azure and load Databricks, I try to create a cluster but this never succeeds. It takes an extremel...
- 1469 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @clock4eva , Thank you for reaching out to our community! We're here to help you. To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedb...
- 2 kudos
- 2068 Views
- 2 replies
- 1 kudos
How to Optimize Delta Lake Performance for Large-Scale Data Ingestion?
Hi everyone,I'm currently working on a project that involves large-scale data ingestion into Delta Lake on Databricks. While the ingestion process is functioning, I've noticed performance bottlenecks, especially with increasing data volumes. Could yo...
- 2068 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Syleena23 , Thank you for reaching out to our community! We're here to help you. To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedb...
- 1 kudos
- 973 Views
- 2 replies
- 0 kudos
Monitoring Databricks
Good Day All, Did any one done Databricks Monitoring integration with any other 3rd party applications like Grafana or ELK to get infrastructure monitoring like cpu_utilization ,memory ,job monitor and I can able to write spark code to get cpu_utili...
- 973 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @SainnathReddy_M , Thank you for reaching out to our community! We're here to help you. To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your...
- 0 kudos
- 880 Views
- 1 replies
- 0 kudos
UnsupportedOperationException: com.databricks.backend.daemon.data.client.DBFSV1.resolvePathOnPhysica
dbutils.fs.mv('dbfs:/FileStore/tables/Employee-2.csv','dbfs:/FileStore/backup/Employee-5.csv',recurse=True)--is giving errorUnsupportedOperationException: com.databricks.backend.daemon.data.client.DBFSV1.resolvePathOnPhysicalStorage(path: Path)File <...
- 880 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @v_sravan_sai , Thank you for reaching out to our community! We're here to help you. To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your fe...
- 0 kudos
- 1262 Views
- 2 replies
- 1 kudos
How can I Resolve QB Desktop Update Error 15225?
I'm encountering QB Desktop update error 15225. What could be causing this issue, and how can I resolve it? It's disrupting my workflow, and I need a quick fix.
- 1262 Views
- 2 replies
- 1 kudos
- 1 kudos
hi @Dorothy80Galvin To resolve Desktop Update Error 15225, follow these steps. First, verify your Internet Explorer settings by ensuring it is set as the default browser and that SSL settings are enabled. Next, add trusted sites by navigating to Inte...
- 1 kudos
