- 758 Views
- 1 replies
- 0 kudos
New Group in Denver
Can we create a new Group in Denver?
- 758 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes Please.
- 0 kudos
- 3442 Views
- 3 replies
- 0 kudos
Virtual Learning Festival Enrollment
Hi everyone,I tried to enroll to Virtual Learning Festival: 9 April - 30 April but upon clicking the Customers & Prospects link for LEARNING PATHWAY 1: ASSOCIATE DATA ENGINEERING I got the error (refer attached image).Thank you in advance for the hel...
- 3442 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Advika,Refer attached image. I thought it was attached in my question. I really new to here.
- 0 kudos
- 512 Views
- 1 replies
- 0 kudos
Expectation in DLT using multiple columns
Is it possible to define an expectation in DLT pipeline using multiple columns?For example, my source has two fields - Division, Material_Number. For division 20, material number starts with 5; for 30 material number starts with 9.Can we have this ...
- 512 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Master_DataBric , Yes its possibleHere is the doc link : - https://docs.databricks.com/aws/en/dlt/expectations?language=Python- https://docs.databricks.com/aws/en/dlt/expectations?language=SQL
- 0 kudos
- 6438 Views
- 2 replies
- 1 kudos
POC Comparison: Databricks vs AWS EMR
Hello,I need some assistance with a comparison between Databricks and AWS EMR. We've been evaluating the Databricks Data Intelligence platform for a client and found it to be significantly more expensive than AWS EMR. I understand the challenge in ma...
- 6438 Views
- 2 replies
- 1 kudos
- 1 kudos
Databricks is highly optimized for Delta, which leverages columnar storage, indexing, and caching for better performance.Instead of directly processing CSV files, convert them to Delta first, then perform aggregations and joins, see if this helps
- 1 kudos
- 590 Views
- 1 replies
- 1 kudos
Is it possible to concatenate two notebooks?
I don't think it's possible but I thought I would check. I need to combine notebooks. While developing I might have code in various notebooks. I read them in with "%run".Then when all looks good I combine many cells into fewer notebooks. Is there any...
- 590 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @397973, Combining multiple notebooks into a single notebook isn't an out-of-the-box feature, but will try to combine %run commands ando output them to see if it works, sort of like: %run "/path/to/notebook1"%run "/path/to/notebook2"
- 1 kudos
- 842 Views
- 2 replies
- 1 kudos
Databricks Lakehouse Monitoring
Hi,I am trying to implement lakehouse monitoring using Inference profile for my inference data that I have, I see that when I create the monitor, two tables get generated profile and drift, I wanted to understand how are these two tables generating a...
- 842 Views
- 2 replies
- 1 kudos
- 1 kudos
When you create a Databricks Lakehouse Monitoring monitor with an Inference profile, the system automatically generates two metric tables: a profile metrics table and a drift metrics table. Here's how this process works: Background Processing When yo...
- 1 kudos
- 1026 Views
- 2 replies
- 0 kudos
Liquid Clustering Key Change Question
If i already have a cluster key1 for existing table, i want to change cluster key to key2 using ALTER TABLE table CLUSTER BY (key2), then run OPTIMIZE table, based on databrick document , existing files will not be rewritten (verified by my test as w...
- 1026 Views
- 2 replies
- 0 kudos
- 0 kudos
@ShivangiB You're correct in your understanding. When you change a clustering key using ALTER TABLE followed by OPTIMIZE, it doesn't automatically recluster existing data. Let me explain why this happens and what options you have.In Delta Lake (which...
- 0 kudos
- 600 Views
- 1 replies
- 0 kudos
Unable to Access S3 from Serverless but Works on Cluster
Hi everyone,I am trying to access data from S3 using an access key and secret. When I run the code through Databricks clusters, it works fine. However, when I try to do the same from a serverless cluster , I am unable to access the data.I have alread...
- 600 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @HarryRichard08! It looks like this post duplicates the one you recently posted. A response has already been provided to the Original post. I recommend continuing the discussion in that thread to keep the conversation focused and organized.
- 0 kudos
- 2371 Views
- 0 replies
- 0 kudos
NiFi on EKS Fails to Connect to Databricks via JDBC – "Connection reset" Error
I'm using Apache Nifi (running on AWS EKS) to connect to Databricks (with compute on EC2) via JDBC. My JDBC URL is as follows: jdbc:databricks://server_hostname:443/default;transportMode=http;ssl=1;httpPath=my_httppath;AuthMech=3;UID=token;PWD=my_tok...
- 2371 Views
- 0 replies
- 0 kudos
- 3717 Views
- 4 replies
- 0 kudos
Resolved! Exam suspended due to sudden power cut
Hi @Cert-Team I hope this message finds you well. I am writing to request a review of my recently suspended exam. I believe that my situation warrants reconsideration, and I would like to provide some context for your understanding.I applied for Da...
- 3717 Views
- 4 replies
- 0 kudos
- 2102 Views
- 2 replies
- 0 kudos
Workspace Assignment Issue via REST API
I’m relying on workspace assignment via REST API to have the account user created in the workspace. This is like the workspace assignment screen at account level or adding existing user screen at workspace level. The reference URL is below.Workspace ...
- 2102 Views
- 2 replies
- 0 kudos
- 0 kudos
It turns out, the problem is the documentation. It says that the permission parameter (that's supplied in) is an array of strings. It really just expects a string, either UNKNOWN, USER, or ADMIN. It would be great if the team could fix the documentat...
- 0 kudos
- 777 Views
- 3 replies
- 0 kudos
Require Information on SQL Analytics DBU Cluster
Hello TeamWe are seeking cost information as we have noticed fluctuations in the daily costs for the "SQL Analytics DBU." We would like to understand the reasons behind the daily cost differences, even though the workload remains consistent.trying to...
- 777 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @gauravmahajan,Most of the cost / DBU used can be retrieved from System tables across your different workspaces in a databricks account. Details related to job compute types and it's associated cost can be fetched from the queries mentioned in the...
- 0 kudos
- 718 Views
- 2 replies
- 0 kudos
Is there a way to install hail on cluster?
Hi all!Been trying to install hail (https://hail.is/) on databricks with no luck so far. Is there an easy way to make it work? So far I could not get further than (providing sparkContext like `hl.init(sc=spark.sparkContext` also did not help):import ...
- 718 Views
- 2 replies
- 0 kudos
- 3843 Views
- 10 replies
- 19 kudos
Resolved! Databricks Demos
I'm looking to build or select a demo in Databricks. Has anyone found any of the particular Databricks demos to deliver a "wow" factor. I am new to Databricks and I'm looking to use one of the staple demos if possible.All the best,BS
- 3843 Views
- 10 replies
- 19 kudos
- 19 kudos
> Has anyone found any of the particular Databricks demos to deliver a "wow" factor.Yes, in fact the last two sprints I did POCs starting with Databricks' AI demos. First, who is your audience--business users, or other technology people? They'll b...
- 19 kudos
- 872 Views
- 2 replies
- 0 kudos
Delta Live Table Pipeline
I have a pipeline that has given me no problems up until today with the following error message:com.databricks.pipelines.common.errors.deployment.DeploymentException: Failed to launch pipeline cluster 0307-134831-tgq587us: Attempt to launch cluster w...
- 872 Views
- 2 replies
- 0 kudos
- 0 kudos
@SB93 The error message you are seeing indicates that the cluster failed to launch because the Spark driver was unresponsive, with possible causes being library conflicts, incorrect metastore configuration, or other configuration issues. Given that t...
- 0 kudos
