cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Flachboard84
by New Contributor II
  • 4003 Views
  • 4 replies
  • 1 kudos

sparkR.session

Why might this be erroring out? My understanding is that SparkR is built into Databricks.Code:library(SparkR, include.only=c('read.parquet', 'collect'))sparkR.session() Error:Error in sparkR.session(): could not find function "sparkR.session"

  • 4003 Views
  • 4 replies
  • 1 kudos
Latest Reply
Flachboard84
New Contributor II
  • 1 kudos

It happens with any code; even something as simple as...x <- 2 + 2

  • 1 kudos
3 More Replies
Sujitha
by Databricks Employee
  • 2199 Views
  • 0 replies
  • 0 kudos

🌟 Welcome Newcomers! 🌟

Hello and welcome to our wonderful Community!Whether you are here by chance or intention, we're thrilled to have you join us. Before you dive into the plethora of discussions and activities happening here, we'd love to get to know you better! ...

  • 2199 Views
  • 0 replies
  • 0 kudos
nan
by New Contributor II
  • 4820 Views
  • 0 replies
  • 0 kudos

TIMEZONE

Can I get some help from Databricks to help me understand how those timestamps being interpreted? Some are really confusing me. I have timestamp coming into AWS Databricks as String type. And the string timestamp is represented in UTC. I ran below qu...

  • 4820 Views
  • 0 replies
  • 0 kudos
Peter_Jones
by New Contributor III
  • 8034 Views
  • 1 replies
  • 0 kudos

Syntax of UPDATE Command in DataBricks

Hi All,I am testing the sql generated by our ETL software to see if it can run on data bricks SQL which I believe is Delta Tables underneath. This is the statement we are testing. As far as I can tell from the manual the from clause is not supported ...

  • 8034 Views
  • 1 replies
  • 0 kudos
277745
by New Contributor
  • 2405 Views
  • 0 replies
  • 0 kudos

Pandas_Udod max batch size not working in notebook

Hello I am trying to set max batch size for pandas-udf in Databricks notebook, but in my tests it doesn’t have any effect on size. spark.conf.set("spark.sql.execution.arrow.enabled", "true")spark.conf.set('spark.sql.execution.arrow.maxRecordsPerBatch...

  • 2405 Views
  • 0 replies
  • 0 kudos
dollyb
by Contributor
  • 6999 Views
  • 13 replies
  • 1 kudos

Databricks Connect Scala -

Hi,I'm using Databricks Connect to run Scala code from IntelliJ on a Databricks single node cluster.Even with the simplest code, I'm experiencing this error:org.apache.spark.SparkException: grpc_shaded.io.grpc.StatusRuntimeException: INTERNAL: org.ap...

  • 6999 Views
  • 13 replies
  • 1 kudos
Latest Reply
dollyb
Contributor
  • 1 kudos

I just hope Databricks will pay attention to it.

  • 1 kudos
12 More Replies
vigneshp
by New Contributor
  • 1360 Views
  • 1 replies
  • 0 kudos

bitmap_count() function's output is different in databricks compared to snowflake

I have found that the results of the bitmap_count() function output differs significantly between databricks and snowflake.eg: snowflake returns a value of '1' for this code. "select bitmap_count(X'0001056c000000000000') " while  Databricks returns a...

vigneshp_1-1701992518337.png vigneshp_0-1701992493192.png
  • 1360 Views
  • 1 replies
  • 0 kudos
Latest Reply
Ayushi_Suthar
Databricks Employee
  • 0 kudos

Hi @vigneshp , Good Day!  In Databricks, bitmap_count function returns the number of bits set in a BINARY string representing a bitmap. This function is typically used to count distinct values in combination with the bitmap_bucket_number() and the bi...

  • 0 kudos
Peter_Jones
by New Contributor III
  • 9985 Views
  • 5 replies
  • 0 kudos

Clusters are failing to launch

Hi Guys,I am a complete newbie to data bricks, we are trying to figure out if our data models and ETL can run on it.I have got the failure to launch message. I have read this message as well.https://community.databricks.com/t5/data-engineering/cluste...

PeterJones_0-1708350996925.png
  • 9985 Views
  • 5 replies
  • 0 kudos
Phani1
by Valued Contributor II
  • 1347 Views
  • 0 replies
  • 0 kudos

Huge data migration from HDFS to Databricks

Hi Team,Could you please help me what is the best way/best practices to copy around 3 TB of data(parquet) from HDFS to Databricks delta format and create external tables on top of it?Regards,Phanindra

  • 1347 Views
  • 0 replies
  • 0 kudos
BillGuyTheScien
by New Contributor II
  • 1660 Views
  • 1 replies
  • 0 kudos

how do committed-use discounts work?

How do committed-use discounts work for Databricks?  Do I purchase a chunk of DBUs for a flat fee and then draw down on them until exhausted?  Or am I purchasing a % discount to all DBUs I use until the time period ends?In either case, is this reflec...

  • 1660 Views
  • 1 replies
  • 0 kudos
Latest Reply
BillGuyTheScien
New Contributor II
  • 0 kudos

Thanks @Retired_mod that helps!  What is the commitment term?  one month? one year?

  • 0 kudos
VANNGA
by New Contributor II
  • 21955 Views
  • 2 replies
  • 0 kudos

POC

Hi, I wonder if you could help me on the below please.We tried Databricks Data Intelligence platform for one of our clients and found that its very expensive when compared to AWS EMR. I understand its not apple-apple comparision as one being platform...

  • 21955 Views
  • 2 replies
  • 0 kudos
Latest Reply
VANNGA
New Contributor II
  • 0 kudos

Hi @Retired_mod Thanks for getting back with so valuable information.SystemFile sizeDurationSystemDurationCommentsComments1EMR225 GB22 minsDatabricks63 minsEMR is cheaper than Databricks by 5 timesThis involves various S3 writes with m5d4xlargeEMR225...

  • 0 kudos
1 More Replies
philipkd
by New Contributor III
  • 11966 Views
  • 3 replies
  • 2 kudos

Resolved! Idle Databricks trial costs me $1/day on AWS

I created a 14-day trial account on Databricks.com and linked it to my AWS. I'm aware that DBUs are free for 14 days, but any AWS charges are my own. I created one workspace, and the CloudFormation was successful. I haven't used it for two days and t...

  • 11966 Views
  • 3 replies
  • 2 kudos
Latest Reply
dataguru
New Contributor II
  • 2 kudos

I also faced the same not sure how to disable or limit the usage. 

  • 2 kudos
2 More Replies
haseeb2001
by New Contributor II
  • 1266 Views
  • 1 replies
  • 0 kudos

Feature Store with Spark Pipeline

Hi,I am using a spark pipeline having stages VectoreAssembler, StandardScalor, StringIndexers, VectorAssembler, GbtClassifier. And then logging this pipeline using feature store log_model function as follows:fe = FeatureStoreClient() // I have tried ...

image.png
  • 1266 Views
  • 1 replies
  • 0 kudos
hpicatto
by New Contributor III
  • 2453 Views
  • 5 replies
  • 2 kudos

Problem updating a one time run Job

I'm creating a series of runs using the /api/2.1/jobs/runs/submit, I wanted to add some tags for more control on the cost and usage, but I notice it's not an option. My first idea was using /api/2.1/jobs/update but it returns that it doesn't have any...

  • 2453 Views
  • 5 replies
  • 2 kudos
Latest Reply
hpicatto
New Contributor III
  • 2 kudos

It could be, but I can still list the job permissions, so it's creating some kind of job... Is there a way of adding from the begining/updating tags into that job?

  • 2 kudos
4 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels
Top Kudoed Authors