- 5957 Views
- 3 replies
- 2 kudos
How can I deduplicate data from my stream?
Hi,I'm new to databricks and I'm trying to use stream for my incremental data. This data has duplicates which can be solved using a window function. Can you check where my code goes wrong?1-------#Using Auto Loader to read new files schema = df1.sche...
- 5957 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @zll_0091 ,Change the output mode to update. Other than that, your code looks fine, but I would rename variable microdf to windowSpec, because now it's little confusing.
- 2 kudos
- 5081 Views
- 3 replies
- 0 kudos
Resolved! DLT Compute: "Ephemeral" Job Compute vs. All-purpose compute 2.0 ... WHY?
Hi there, this is a follow-up from a discussion I started last monthSolved: Re: DLT Compute: "Ephemeral" Job Compute vs. All-p... - Databricks Community - 71661Based on what was discussed, I understand that it's not possible to use "All Purpose Clust...
- 5081 Views
- 3 replies
- 0 kudos
- 0 kudos
@ChristianRRL regarding on why DLT doesn't allow you to use all-purpose clusters: 1. The DLT runtime is derived from the shared compute DBR, it's not the same runtime and has different features than the common all-purpose runtime. A DLT pipeline is n...
- 0 kudos
- 7409 Views
- 1 replies
- 0 kudos
Easy GIF Animator 7.4.8 Crack + License Key 2024
Easy GIF Animator 7.4.8 Crack + License Key 2024Easy GIF Animator Crack is popular ever for animation creation and editing purpose there are sufficient modules that works in moderate session and also upgrade the further styles which available here to...
- 7409 Views
- 1 replies
- 0 kudos
- 0 kudos
Download Easy GIF Animator
- 0 kudos
- 4590 Views
- 3 replies
- 0 kudos
Error Creating Primary Key Constraint in DLT
Hello There!Greetings!!I am getting the following error when trying to Create a DLT table in my Gold Layer..com.databricks.sql.managedcatalog.PrimaryKeyColumnsNullableException: Cannot create the primary key `x_key` because its child column(s) `x_key...
- 4590 Views
- 3 replies
- 0 kudos
- 0 kudos
Thank you @szymon_dybczak for the response.As you can see I have already defined the not null constraint in my definition for the primary key x_keyCONSTRAINT pk_key_not_null EXPECT (x_key IS NOT NULL) But still I am getting the same error.Also I chec...
- 0 kudos
- 6742 Views
- 2 replies
- 1 kudos
Oracle datawarehous Replacement With Databricks Using DeltaLake
I am new to Spark and DataBricks and exploring these to understand to replace Oracle DataWarehouse by DataBricks(deltalake) and to use Spark to improve the ELT/ETL performance of existing DW.Now, I have done some lookups in databricks blogs, spark do...
- 6742 Views
- 2 replies
- 1 kudos
- 1209 Views
- 0 replies
- 1 kudos
registration and certification of databricks certification course regarding
I have a doubt only working personal are register for certification or student who studying data bricks course can register for exam because in some steps of registration ask for company and company email.
- 1209 Views
- 0 replies
- 1 kudos
- 31956 Views
- 6 replies
- 0 kudos
Error "Root storage credential for metastore does not exist"While creating the Databricks Volume in
Hi, I tried to create the databricks volume in unity catalog but it threw this error:Root storage credential for metastore XXXXXX does not exist. Please contact your Databricks representative or consider updating the metastore with a valid storage cr...
- 31956 Views
- 6 replies
- 0 kudos
- 0 kudos
I had a similar issue, that was resolved by assigning storage credential to the metastore using REST API call.- make sure that you are metastore or account admin- make sure that the storage credential is correctly configured, its access corrector ide...
- 0 kudos
- 1592 Views
- 0 replies
- 0 kudos
DBFS problem
Hi All, When trying to get to my DBFS in Databricks comunity edition I get the next errorI have already disabled and Enabled de DBFS file browser but it did not help
- 1592 Views
- 0 replies
- 0 kudos
- 3910 Views
- 1 replies
- 0 kudos
Issue with VSCode Extension and Databricks Cluster Using Docker Image
I've encountered a significant issue while using the VSCode extension for Databricks, particularly when working with a cluster configured with a Docker image. Here's a detailed description of the problem:Problem DescriptionWhen attempting to upload a...
- 3910 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Retired_mod thanks for a such quick response.Actually, I am using the Dockerfile from the Databricks runtime example here: https://github.com/databricks/containers/blob/master/ubuntu/minimal/Dockerfile . The configuration with the VSCode extensio...
- 0 kudos
- 3138 Views
- 2 replies
- 0 kudos
RDD Issue in Unity Catalog
################################################ New Code as per UC ################################################ def parse_json(df, *cols, clean=True res = df for i in cols: if clean: res = ( res.wit...
- 3138 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @sambgp , Thank you for reaching out to our community! We're here to help you. To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedback...
- 0 kudos
- 4377 Views
- 4 replies
- 2 kudos
Databricks All Delta Tables Data Read
If we want to read all the data of the databricks tables at single time how can we able to do it.
- 4377 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @Krishna2110 ,Here it is, it should work now tables = spark.sql("SHOW TABLES IN ewt_edp_prod.crm_raw").collect() for row in tables: table_name = f"ewt_edp_prod.{row[0]}.{row[1]}" try: df = spark.table(table_name) count = df...
- 2 kudos
- 9618 Views
- 2 replies
- 2 kudos
Resolved! Convert Date String to Date type Returning null
Hi,I am using Databricks SQL and I am converting a integer field which is of format ('20240719' or 'yyyyMMdd'),now I am able to convert it to date type using to_date(forecastedHorizonPeriodDate,'yyyyMMdd')I then tried to change the format of this dat...
- 9618 Views
- 2 replies
- 2 kudos
- 4899 Views
- 1 replies
- 0 kudos
Login issue
Hi,Getting below error message.We were not able to find a Community Edition workspace with this email. Please login to non-community-edition workspaces you may have access to.
- 4899 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @virendra1212 ,Take a look at below thread. Maybe some of the suggestions will help in your case.Community Edition Login Issues Below is a list of ... - Page 4 - Databricks Community - 26926You signed up before May 14, 2022 -Please confirm that th...
- 0 kudos
- 1320 Views
- 1 replies
- 0 kudos
auto statistics cost
hi,is there any cost implications for automatic statistics collection?or databricks is providing it as a feature and didn't cost on my cluster?
- 1320 Views
- 1 replies
- 0 kudos
- 1212 Views
- 0 replies
- 0 kudos
Shuffle Partitions
What if I have lot of empty shuffled partitions due to data skewness Secondly , if the shuffle partition size is 128 MB and if the size of the key's partition is 700 MB
- 1212 Views
- 0 replies
- 0 kudos
