cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Peter_Jones
by New Contributor
  • 26 Views
  • 0 replies
  • 0 kudos

Syntax of UPDATE Command in DataBricks

Hi All,I am testing the sql generated by our ETL software to see if it can run on data bricks SQL which I believe is Delta Tables underneath. This is the statement we are testing. As far as I can tell from the manual the from clause is not supported ...

  • 26 Views
  • 0 replies
  • 0 kudos
277745
by New Contributor
  • 28 Views
  • 0 replies
  • 0 kudos

Pandas_Udod max batch size not working in notebook

Hello I am trying to set max batch size for pandas-udf in Databricks notebook, but in my tests it doesn’t have any effect on size. spark.conf.set("spark.sql.execution.arrow.enabled", "true")spark.conf.set('spark.sql.execution.arrow.maxRecordsPerBatch...

  • 28 Views
  • 0 replies
  • 0 kudos
dollyb
by New Contributor
  • 202 Views
  • 13 replies
  • 1 kudos

Databricks Connect Scala -

Hi,I'm using Databricks Connect to run Scala code from IntelliJ on a Databricks single node cluster.Even with the simplest code, I'm experiencing this error:org.apache.spark.SparkException: grpc_shaded.io.grpc.StatusRuntimeException: INTERNAL: org.ap...

  • 202 Views
  • 13 replies
  • 1 kudos
Latest Reply
dollyb
New Contributor
  • 1 kudos

I just hope Databricks will pay attention to it.

  • 1 kudos
12 More Replies
puravivecapsule
by New Contributor
  • 45 Views
  • 0 replies
  • 0 kudos

Puravive jennifer lopez rice hack

 Puravive  -  Puravive doesn't rely on traditional methods like calorie restriction or appetite suppression.It focuses on maximizing your body's potential to burn calories even at rest by activating BAT cells.Official Websitehttps://www.healthnews360...

  • 45 Views
  • 0 replies
  • 0 kudos
vigneshp
by New Contributor
  • 161 Views
  • 1 replies
  • 0 kudos

bitmap_count() function's output is different in databricks compared to snowflake

I have found that the results of the bitmap_count() function output differs significantly between databricks and snowflake.eg: snowflake returns a value of '1' for this code. "select bitmap_count(X'0001056c000000000000') " while  Databricks returns a...

vigneshp_1-1701992518337.png vigneshp_0-1701992493192.png
  • 161 Views
  • 1 replies
  • 0 kudos
Latest Reply
Ayushi_Suthar
Valued Contributor II
  • 0 kudos

Hi @vigneshp , Good Day!  In Databricks, bitmap_count function returns the number of bits set in a BINARY string representing a bitmap. This function is typically used to count distinct values in combination with the bitmap_bucket_number() and the bi...

  • 0 kudos
Peter_Jones
by New Contributor
  • 185 Views
  • 6 replies
  • 0 kudos

Resolved! Clusters are failing to launch

Hi Guys,I am a complete newbie to data bricks, we are trying to figure out if our data models and ETL can run on it.I have got the failure to launch message. I have read this message as well.https://community.databricks.com/t5/data-engineering/cluste...

PeterJones_0-1708350996925.png
  • 185 Views
  • 6 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @Peter_Jones, Let’s tackle this cluster launch issue step by step. Quota Exceeded Error: The error message indicates that your cluster launch failed due to exceeding the approved quota for standardEDSv4Family Cores in the westeurope location. ...

  • 0 kudos
5 More Replies
farazanwar
by New Contributor II
  • 1431 Views
  • 1 replies
  • 0 kudos

Jira Connector

Is there any native connector in development fir JIRA

  • 1431 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @farazanwar, Jira Software offers a wide range of integrations to enhance its functionality. While there isn’t a native connector specifically developed by the Jira team, there are several apps (also known as add-ons or plugins) available in the A...

  • 0 kudos
Phani1
by Valued Contributor
  • 59 Views
  • 1 replies
  • 1 kudos

Huge data migration from HDFS to Databricks

Hi Team,Could you please help me what is the best way/best practices to copy around 3 TB of data(parquet) from HDFS to Databricks delta format and create external tables on top of it?Regards,Phanindra

  • 59 Views
  • 1 replies
  • 1 kudos
Latest Reply
Kaniz
Community Manager
  • 1 kudos

Hi @Phani1, To efficiently copy around 3 TB of Parquet data from HDFS to Databricks Delta format and create external tables, you can follow these best practices: Use the COPY INTO SQL Command: The COPY INTO SQL command allows you to load data fr...

  • 1 kudos
BillGuyTheScien
by New Contributor II
  • 102 Views
  • 2 replies
  • 0 kudos

how do committed-use discounts work?

How do committed-use discounts work for Databricks?  Do I purchase a chunk of DBUs for a flat fee and then draw down on them until exhausted?  Or am I purchasing a % discount to all DBUs I use until the time period ends?In either case, is this reflec...

  • 102 Views
  • 2 replies
  • 0 kudos
Latest Reply
BillGuyTheScien
New Contributor II
  • 0 kudos

Thanks @Kaniz that helps!  What is the commitment term?  one month? one year?

  • 0 kudos
1 More Replies
VANNGA
by New Contributor II
  • 973 Views
  • 3 replies
  • 0 kudos

POC

Hi, I wonder if you could help me on the below please.We tried Databricks Data Intelligence platform for one of our clients and found that its very expensive when compared to AWS EMR. I understand its not apple-apple comparision as one being platform...

  • 973 Views
  • 3 replies
  • 0 kudos
Latest Reply
VANNGA
New Contributor II
  • 0 kudos

Hi @Kaniz Thanks for getting back with so valuable information.SystemFile sizeDurationSystemDurationCommentsComments1EMR225 GB22 minsDatabricks63 minsEMR is cheaper than Databricks by 5 timesThis involves various S3 writes with m5d4xlargeEMR225 GB45 ...

  • 0 kudos
2 More Replies
philipkd
by New Contributor III
  • 842 Views
  • 4 replies
  • 2 kudos

Resolved! Idle Databricks trial costs me $1/day on AWS

I created a 14-day trial account on Databricks.com and linked it to my AWS. I'm aware that DBUs are free for 14 days, but any AWS charges are my own. I created one workspace, and the CloudFormation was successful. I haven't used it for two days and t...

  • 842 Views
  • 4 replies
  • 2 kudos
Latest Reply
dataguru
New Contributor II
  • 2 kudos

I also faced the same not sure how to disable or limit the usage. 

  • 2 kudos
3 More Replies
haseeb2001
by New Contributor II
  • 223 Views
  • 2 replies
  • 0 kudos

Feature Store with Spark Pipeline

Hi,I am using a spark pipeline having stages VectoreAssembler, StandardScalor, StringIndexers, VectorAssembler, GbtClassifier. And then logging this pipeline using feature store log_model function as follows:fe = FeatureStoreClient() // I have tried ...

image.png
  • 223 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @haseeb2001, It seems you’re encountering an issue after logging your Spark pipeline model using the fe.log_model function in MLflow. Let’s break down the steps and address the error: Pipeline Stages: You’ve mentioned several stages in your S...

  • 0 kudos
1 More Replies
hpicatto
by New Contributor II
  • 264 Views
  • 5 replies
  • 2 kudos

Problem updating a one time run Job

I'm creating a series of runs using the /api/2.1/jobs/runs/submit, I wanted to add some tags for more control on the cost and usage, but I notice it's not an option. My first idea was using /api/2.1/jobs/update but it returns that it doesn't have any...

  • 264 Views
  • 5 replies
  • 2 kudos
Latest Reply
hpicatto
New Contributor II
  • 2 kudos

It could be, but I can still list the job permissions, so it's creating some kind of job... Is there a way of adding from the begining/updating tags into that job?

  • 2 kudos
4 More Replies
TestuserAva
by New Contributor II
  • 428 Views
  • 4 replies
  • 1 kudos

Getting HTML sign I page as api response from databricks api with statuscode 200

Response:<!doctype html><html><head>    <meta charset="utf-8" />    <meta http-equiv="Content-Language" content="en" />    <title>Databricks - Sign In</title>    <meta name="viewport" content="width=960" />    <link rel="icon" type="image/png" href="...

TestuserAva_0-1701165195616.png
  • 428 Views
  • 4 replies
  • 1 kudos
Latest Reply
Abhishek10745
New Contributor II
  • 1 kudos

@TestuserAvaCan you please share your approach to tackle this problem in case if you solved.For me it works using powershell Invoke-WebRequest but not using postman.

  • 1 kudos
3 More Replies
Srjv
by New Contributor III
  • 116 Views
  • 2 replies
  • 0 kudos

Can I use databricks principals on databricks connect 12.2?

Hi community,Is it possible to use Databricks service principals for authentication on Databricks connect 12.2 to connect my notebook or code to Databricks compute, rather than using personal access token? I checked the docs and got to know that upgr...

  • 116 Views
  • 2 replies
  • 0 kudos
Latest Reply
Srjv
New Contributor III
  • 0 kudos

Hi @KanizThanks for your response. I was able to generate the token of the service principal following this doc, later saved it in the <Databricks Token> variable prompted when running databricks-connect configure command in terminal. And was able to...

  • 0 kudos
1 More Replies