cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

vigneshp
by New Contributor
  • 848 Views
  • 1 replies
  • 0 kudos

bitmap_count() function's output is different in databricks compared to snowflake

I have found that the results of the bitmap_count() function output differs significantly between databricks and snowflake.eg: snowflake returns a value of '1' for this code. "select bitmap_count(X'0001056c000000000000') " while  Databricks returns a...

vigneshp_1-1701992518337.png vigneshp_0-1701992493192.png
  • 848 Views
  • 1 replies
  • 0 kudos
Latest Reply
Ayushi_Suthar
Honored Contributor
  • 0 kudos

Hi @vigneshp , Good Day!  In Databricks, bitmap_count function returns the number of bits set in a BINARY string representing a bitmap. This function is typically used to count distinct values in combination with the bitmap_bucket_number() and the bi...

  • 0 kudos
Peter_Jones
by New Contributor III
  • 4732 Views
  • 6 replies
  • 0 kudos

Resolved! Clusters are failing to launch

Hi Guys,I am a complete newbie to data bricks, we are trying to figure out if our data models and ETL can run on it.I have got the failure to launch message. I have read this message as well.https://community.databricks.com/t5/data-engineering/cluste...

PeterJones_0-1708350996925.png
  • 4732 Views
  • 6 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Peter_Jones, Let’s tackle this cluster launch issue step by step. Quota Exceeded Error: The error message indicates that your cluster launch failed due to exceeding the approved quota for standardEDSv4Family Cores in the westeurope location. ...

  • 0 kudos
5 More Replies
farazanwar
by New Contributor II
  • 3161 Views
  • 1 replies
  • 0 kudos

Jira Connector

Is there any native connector in development fir JIRA

  • 3161 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @farazanwar, Jira Software offers a wide range of integrations to enhance its functionality. While there isn’t a native connector specifically developed by the Jira team, there are several apps (also known as add-ons or plugins) available in the A...

  • 0 kudos
Phani1
by Valued Contributor
  • 846 Views
  • 1 replies
  • 1 kudos

Huge data migration from HDFS to Databricks

Hi Team,Could you please help me what is the best way/best practices to copy around 3 TB of data(parquet) from HDFS to Databricks delta format and create external tables on top of it?Regards,Phanindra

  • 846 Views
  • 1 replies
  • 1 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @Phani1, To efficiently copy around 3 TB of Parquet data from HDFS to Databricks Delta format and create external tables, you can follow these best practices: Use the COPY INTO SQL Command: The COPY INTO SQL command allows you to load data fr...

  • 1 kudos
BillGuyTheScien
by New Contributor II
  • 976 Views
  • 2 replies
  • 0 kudos

how do committed-use discounts work?

How do committed-use discounts work for Databricks?  Do I purchase a chunk of DBUs for a flat fee and then draw down on them until exhausted?  Or am I purchasing a % discount to all DBUs I use until the time period ends?In either case, is this reflec...

  • 976 Views
  • 2 replies
  • 0 kudos
Latest Reply
BillGuyTheScien
New Contributor II
  • 0 kudos

Thanks @Kaniz_Fatma that helps!  What is the commitment term?  one month? one year?

  • 0 kudos
1 More Replies
VANNGA
by New Contributor II
  • 3820 Views
  • 3 replies
  • 1 kudos

POC

Hi, I wonder if you could help me on the below please.We tried Databricks Data Intelligence platform for one of our clients and found that its very expensive when compared to AWS EMR. I understand its not apple-apple comparision as one being platform...

  • 3820 Views
  • 3 replies
  • 1 kudos
Latest Reply
VANNGA
New Contributor II
  • 1 kudos

Hi @Kaniz_Fatma Thanks for getting back with so valuable information.SystemFile sizeDurationSystemDurationCommentsComments1EMR225 GB22 minsDatabricks63 minsEMR is cheaper than Databricks by 5 timesThis involves various S3 writes with m5d4xlargeEMR225...

  • 1 kudos
2 More Replies
philipkd
by New Contributor III
  • 3083 Views
  • 4 replies
  • 2 kudos

Resolved! Idle Databricks trial costs me $1/day on AWS

I created a 14-day trial account on Databricks.com and linked it to my AWS. I'm aware that DBUs are free for 14 days, but any AWS charges are my own. I created one workspace, and the CloudFormation was successful. I haven't used it for two days and t...

  • 3083 Views
  • 4 replies
  • 2 kudos
Latest Reply
dataguru
New Contributor II
  • 2 kudos

I also faced the same not sure how to disable or limit the usage. 

  • 2 kudos
3 More Replies
haseeb2001
by New Contributor II
  • 748 Views
  • 2 replies
  • 0 kudos

Feature Store with Spark Pipeline

Hi,I am using a spark pipeline having stages VectoreAssembler, StandardScalor, StringIndexers, VectorAssembler, GbtClassifier. And then logging this pipeline using feature store log_model function as follows:fe = FeatureStoreClient() // I have tried ...

image.png
  • 748 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @haseeb2001, It seems you’re encountering an issue after logging your Spark pipeline model using the fe.log_model function in MLflow. Let’s break down the steps and address the error: Pipeline Stages: You’ve mentioned several stages in your S...

  • 0 kudos
1 More Replies
hpicatto
by New Contributor III
  • 1618 Views
  • 5 replies
  • 2 kudos

Problem updating a one time run Job

I'm creating a series of runs using the /api/2.1/jobs/runs/submit, I wanted to add some tags for more control on the cost and usage, but I notice it's not an option. My first idea was using /api/2.1/jobs/update but it returns that it doesn't have any...

  • 1618 Views
  • 5 replies
  • 2 kudos
Latest Reply
hpicatto
New Contributor III
  • 2 kudos

It could be, but I can still list the job permissions, so it's creating some kind of job... Is there a way of adding from the begining/updating tags into that job?

  • 2 kudos
4 More Replies
Surajv
by New Contributor III
  • 537 Views
  • 2 replies
  • 0 kudos

Can I use databricks principals on databricks connect 12.2?

Hi community,Is it possible to use Databricks service principals for authentication on Databricks connect 12.2 to connect my notebook or code to Databricks compute, rather than using personal access token? I checked the docs and got to know that upgr...

  • 537 Views
  • 2 replies
  • 0 kudos
Latest Reply
Surajv
New Contributor III
  • 0 kudos

Hi @Kaniz_FatmaThanks for your response. I was able to generate the token of the service principal following this doc, later saved it in the <Databricks Token> variable prompted when running databricks-connect configure command in terminal. And was a...

  • 0 kudos
1 More Replies
Phani1
by Valued Contributor
  • 5117 Views
  • 4 replies
  • 1 kudos

UCX Installation

We aim to streamline the UCX installation process by utilizing Databricks CLI and automating the manual input of required details at each question level .Could you please guide us how can we achieve to automate the parameter's while installation? wha...

  • 5117 Views
  • 4 replies
  • 1 kudos
Latest Reply
Phani1
Valued Contributor
  • 1 kudos

Hi Team, We don't see option at UCX command level  passing parameters as json /config file, could you please help me in this case how we can automate the installation.    

  • 1 kudos
3 More Replies
nidhin
by New Contributor
  • 348 Views
  • 1 replies
  • 0 kudos

Error Handling for Web Data Retrieval and Storage in Databricks UNITY Clusters

The following code works well in a normal Databricks cluster, where it passes a null JSON and retrieves content from the web link. However, in a Unity cluster, it produces the following error: 'FileNotFoundError: [Errno 2] No such file or directory: ...

  • 348 Views
  • 1 replies
  • 0 kudos
Latest Reply
Ayushi_Suthar
Honored Contributor
  • 0 kudos

Hi @nidhin , Good Day!  The reason behind the below error while trying to access the external dbfs mount file using "with open" is that you are using a shared access mode cluster. 'FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/mnt/ra...

  • 0 kudos
Phani1
by Valued Contributor
  • 1872 Views
  • 1 replies
  • 0 kudos

UCX Installation without CLI

 Hi Team, Can we install UCX Toolkit into databricks workspace without installing it via Databricks CLI? If its possible then How ?https://github.com/databrickslabs/ucx

  • 1872 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Phani1, The UCX Toolkit is a powerful tool for upgrading your Databricks workspace to Unity Catalog. It helps you migrate various assets within your workspace, including: Legacy Table ACLsEntitlementsAWS instance profilesClusters and cluster p...

  • 0 kudos
Phani1
by Valued Contributor
  • 4217 Views
  • 1 replies
  • 1 kudos

Resolved! SAP Successfator

Hi Team,We are working on a new Data Product onboarding to the current Databricks Lakehouse Platform.The first step is foundation where we should get data from SAP success factors to S3+ Bronze layer and then do the initial setup of Lakehouse+Power B...

  • 4217 Views
  • 1 replies
  • 1 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @Phani1, Retrieving data from SAP SuccessFactors and storing it in your Databricks Lakehouse Platform involves several considerations.    Let’s break it down step by step:   Data Extraction from SAP SuccessFactors: You’ve correctly identified that...

  • 1 kudos
Phani1
by Valued Contributor
  • 1291 Views
  • 1 replies
  • 1 kudos

cost finding and optimization

 Hi Team,Could you please suggest the best way to track the cost of Databricks objects/components? Could you please share any best practices for optimizing costs and conducting detailed cost analysis?Regards,Phanindra

  • 1291 Views
  • 1 replies
  • 1 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @Phani1,  Understand Databricks Units (DBUs): A DBU (Databricks Unit) is the fundamental unit of consumption within the Databricks platform.DBUs are based on the number of nodes and the computational power of VM instance types in your clusters...

  • 1 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels
Top Kudoed Authors