cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Phani1
by Valued Contributor II
  • 1200 Views
  • 0 replies
  • 0 kudos

Huge data migration from HDFS to Databricks

Hi Team,Could you please help me what is the best way/best practices to copy around 3 TB of data(parquet) from HDFS to Databricks delta format and create external tables on top of it?Regards,Phanindra

  • 1200 Views
  • 0 replies
  • 0 kudos
BillGuyTheScien
by New Contributor II
  • 1425 Views
  • 1 replies
  • 0 kudos

how do committed-use discounts work?

How do committed-use discounts work for Databricks?  Do I purchase a chunk of DBUs for a flat fee and then draw down on them until exhausted?  Or am I purchasing a % discount to all DBUs I use until the time period ends?In either case, is this reflec...

  • 1425 Views
  • 1 replies
  • 0 kudos
Latest Reply
BillGuyTheScien
New Contributor II
  • 0 kudos

Thanks @Retired_mod that helps!  What is the commitment term?  one month? one year?

  • 0 kudos
VANNGA
by New Contributor II
  • 13564 Views
  • 2 replies
  • 0 kudos

POC

Hi, I wonder if you could help me on the below please.We tried Databricks Data Intelligence platform for one of our clients and found that its very expensive when compared to AWS EMR. I understand its not apple-apple comparision as one being platform...

  • 13564 Views
  • 2 replies
  • 0 kudos
Latest Reply
VANNGA
New Contributor II
  • 0 kudos

Hi @Retired_mod Thanks for getting back with so valuable information.SystemFile sizeDurationSystemDurationCommentsComments1EMR225 GB22 minsDatabricks63 minsEMR is cheaper than Databricks by 5 timesThis involves various S3 writes with m5d4xlargeEMR225...

  • 0 kudos
1 More Replies
philipkd
by New Contributor III
  • 10694 Views
  • 3 replies
  • 2 kudos

Resolved! Idle Databricks trial costs me $1/day on AWS

I created a 14-day trial account on Databricks.com and linked it to my AWS. I'm aware that DBUs are free for 14 days, but any AWS charges are my own. I created one workspace, and the CloudFormation was successful. I haven't used it for two days and t...

  • 10694 Views
  • 3 replies
  • 2 kudos
Latest Reply
dataguru
New Contributor II
  • 2 kudos

I also faced the same not sure how to disable or limit the usage. 

  • 2 kudos
2 More Replies
haseeb2001
by New Contributor II
  • 1097 Views
  • 1 replies
  • 0 kudos

Feature Store with Spark Pipeline

Hi,I am using a spark pipeline having stages VectoreAssembler, StandardScalor, StringIndexers, VectorAssembler, GbtClassifier. And then logging this pipeline using feature store log_model function as follows:fe = FeatureStoreClient() // I have tried ...

image.png
  • 1097 Views
  • 1 replies
  • 0 kudos
Latest Reply
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Hi,I am using a spark pipeline having stages VectoreAssembler, StandardScalor, StringIndexers, VectorAssembler, GbtClassifier. And then logging this pipeline using feature store log_model function as follows:fe = FeatureStoreClient() // I have tried ...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
hpicatto
by New Contributor III
  • 2148 Views
  • 5 replies
  • 2 kudos

Problem updating a one time run Job

I'm creating a series of runs using the /api/2.1/jobs/runs/submit, I wanted to add some tags for more control on the cost and usage, but I notice it's not an option. My first idea was using /api/2.1/jobs/update but it returns that it doesn't have any...

  • 2148 Views
  • 5 replies
  • 2 kudos
Latest Reply
hpicatto
New Contributor III
  • 2 kudos

It could be, but I can still list the job permissions, so it's creating some kind of job... Is there a way of adding from the begining/updating tags into that job?

  • 2 kudos
4 More Replies
Surajv
by New Contributor III
  • 1113 Views
  • 1 replies
  • 0 kudos

Can I use databricks principals on databricks connect 12.2?

Hi community,Is it possible to use Databricks service principals for authentication on Databricks connect 12.2 to connect my notebook or code to Databricks compute, rather than using personal access token? I checked the docs and got to know that upgr...

  • 1113 Views
  • 1 replies
  • 0 kudos
Latest Reply
Surajv
New Contributor III
  • 0 kudos

Hi @Retired_modThanks for your response. I was able to generate the token of the service principal following this doc, later saved it in the <Databricks Token> variable prompted when running databricks-connect configure command in terminal. And was a...

  • 0 kudos
Phani1
by Valued Contributor II
  • 8655 Views
  • 2 replies
  • 1 kudos

UCX Installation

We aim to streamline the UCX installation process by utilizing Databricks CLI and automating the manual input of required details at each question level .Could you please guide us how can we achieve to automate the parameter's while installation? wha...

  • 8655 Views
  • 2 replies
  • 1 kudos
Latest Reply
Phani1
Valued Contributor II
  • 1 kudos

Hi Team, We don't see option at UCX command level  passing parameters as json /config file, could you please help me in this case how we can automate the installation.    

  • 1 kudos
1 More Replies
nidhin
by New Contributor
  • 621 Views
  • 1 replies
  • 0 kudos

Error Handling for Web Data Retrieval and Storage in Databricks UNITY Clusters

The following code works well in a normal Databricks cluster, where it passes a null JSON and retrieves content from the web link. However, in a Unity cluster, it produces the following error: 'FileNotFoundError: [Errno 2] No such file or directory: ...

  • 621 Views
  • 1 replies
  • 0 kudos
Latest Reply
Ayushi_Suthar
Databricks Employee
  • 0 kudos

Hi @nidhin , Good Day!  The reason behind the below error while trying to access the external dbfs mount file using "with open" is that you are using a shared access mode cluster. 'FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/mnt/ra...

  • 0 kudos
Phani1
by Valued Contributor II
  • 2784 Views
  • 0 replies
  • 0 kudos

UCX Installation without CLI

 Hi Team, Can we install UCX Toolkit into databricks workspace without installing it via Databricks CLI? If its possible then How ?https://github.com/databrickslabs/ucx

  • 2784 Views
  • 0 replies
  • 0 kudos
Phani1
by Valued Contributor II
  • 4973 Views
  • 0 replies
  • 0 kudos

SAP Successfator

Hi Team,We are working on a new Data Product onboarding to the current Databricks Lakehouse Platform.The first step is foundation where we should get data from SAP success factors to S3+ Bronze layer and then do the initial setup of Lakehouse+Power B...

  • 4973 Views
  • 0 replies
  • 0 kudos
Phani1
by Valued Contributor II
  • 1790 Views
  • 0 replies
  • 1 kudos

cost finding and optimization

 Hi Team,Could you please suggest the best way to track the cost of Databricks objects/components? Could you please share any best practices for optimizing costs and conducting detailed cost analysis?Regards,Phanindra

  • 1790 Views
  • 0 replies
  • 1 kudos
Phani1
by Valued Contributor II
  • 1088 Views
  • 1 replies
  • 0 kudos

Capture changes at the object level in Databricks

Could you please suggest how to capture changes at the object level in Databricks, such as notebooks changes, table DDL changes, view DDL , functions DDL, and workflows etc. changes? We would like to build a dashboard for changes.

  • 1088 Views
  • 1 replies
  • 0 kudos
Latest Reply
Ayushi_Suthar
Databricks Employee
  • 0 kudos

Hi @Phani1 , Good Day!  Could you kindly clarify on your question about the capture change? What type of change do you want to capture? If you want to see what modifications the user made to the Notebooks, tables, and workflow, you can check the audi...

  • 0 kudos
wellington
by New Contributor III
  • 1105 Views
  • 1 replies
  • 0 kudos

Log notebook activities

Hi friends;I'm working on a project where we are 4 programmers. We are working in a single environment, using only the "Workspaces" folder. Each has its own user, which is managed by Azure AD.We had a peak in consumption on the 5th Feb. So I can see ...

  • 1105 Views
  • 1 replies
  • 0 kudos
Latest Reply
wellington
New Contributor III
  • 0 kudos

Hi @Retired_mod , thanks for your quick answer.There is no other way to monitor notebook runs. I ask this because adding tags to the cluster and workspace does not solve my problem, considering that everyone uses the same cluster and the same workspa...

  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels