cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

T0M
by New Contributor III
  • 181 Views
  • 2 replies
  • 1 kudos

DLT Pipeline Validate will always spawn new cluster

Hi all!I've started learning DLT-Pipelines but I am struggling with the development of a pipeline.As far as I understand it, once I click on “Validate” a cluster will spin-up and stay (by default for 2hours), if the pipeline is in “Development” mode....

  • 181 Views
  • 2 replies
  • 1 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 1 kudos

Hi @T0M, It is expected that the cluster restarts with each validation to ensure that your changes are accurately reflected.  In development mode, once you click "Validate," a cluster will spin up, and it stays active for up to two hours by default. ...

  • 1 kudos
1 More Replies
GerardAlexander
by New Contributor
  • 370 Views
  • 1 replies
  • 0 kudos

Creating Unity Catalog in Personal AZURE Portal Account

Seeking advice on the following:1. Given that I have a Personal - and not an Organization-based - AZURE Portal Account,    2. that I can see I am Global Admin and have Admin Role in Databricks,         3. then why can I not get "Manage Account" for a...

  • 370 Views
  • 1 replies
  • 0 kudos
Latest Reply
Takuya-Omi
Valued Contributor II
  • 0 kudos

@GerardAlexander Try signing in to the Account Console (https://accounts.azuredatabricks.net/login) using a user account with the appropriate permissions, rather than accessing it from the workspace.If you are unable to sign in, the following resourc...

  • 0 kudos
spd_dat
by New Contributor II
  • 527 Views
  • 0 replies
  • 0 kudos

Can AWS workspaces share subnets?

The docs state:"You can choose to share one subnet across multiple workspaces or both subnets across workspaces."as well as:"You can reuse existing security groups rather than create new ones."and on this page:"If you plan to share a VPC and subnets ...

  • 527 Views
  • 0 replies
  • 0 kudos
FanMichelle0729
by New Contributor II
  • 420 Views
  • 5 replies
  • 0 kudos

Serveless compute does need has cloud accout(AWS、Google 、Azure)

I am a Databricks beginner, and I would like to ask if the Compute created in the Databricks account , it means also exists in the cloud account (e.g., AWS)? If the AWS account is deactivated, the existing compute will not be usable. This is what I h...

  • 420 Views
  • 5 replies
  • 0 kudos
Latest Reply
Takuya-Omi
Valued Contributor II
  • 0 kudos

@FanMichelleTW No, Databricks recommends using serverless compute, and you can use serverless compute as well.To do so, open a notebook and check the top-right corner to see if a serverless compute option is in a Ready state. If it is, simply select ...

  • 0 kudos
4 More Replies
nathanmle
by New Contributor
  • 182 Views
  • 1 replies
  • 0 kudos

Where to find Jupyter Notebook course materials for Get Started with Databricks for Generative AI

Hello, I can't seem to find any way to gain access to the Jupyter Notebook demo source of "Get Started with Databricks for Generative AI" course.  Please help.  Thank you kindly in advance. 

  • 182 Views
  • 1 replies
  • 0 kudos
Latest Reply
Advika_
Databricks Employee
  • 0 kudos

Hello @nathanmle! We are sorry to inform you that we are no longer offering notebooks or the DBC files for the self-paced courses due to recent changes.If you’re interested in working on labs in a provided Databricks environment, you can purchase the...

  • 0 kudos
kiko_roy
by Contributor
  • 1165 Views
  • 3 replies
  • 0 kudos

Getting access token error when connecting from azure databricks to GCS bucket

I am creating a data frame by reading a table's data residing in Azure backed unity catalog. I need to write the dataframe or file to GCS bucket. I have configured the spark cluster config using the GCP service account json values. Also tried uploadi...

  • 1165 Views
  • 3 replies
  • 0 kudos
Latest Reply
KristiLogos
Contributor
  • 0 kudos

@kiko_roy  unfortunately that didnt' work. the error is stating its trying to get the access token from metadata server, I wonder why from the metadata server?

  • 0 kudos
2 More Replies
lukaseder
by New Contributor II
  • 287 Views
  • 2 replies
  • 0 kudos

How to fetch nested data structures in Databricks using JDBC

I've asked the question also here on stack overflow When using nested data structures in Databricks (e.g. `ARRAY` or `ROW`) using JDBC, it appears that the results can be fetched as JSON `String` values, e.g.: try (Statement s = connection.createStat...

  • 287 Views
  • 2 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hi @lukaseder, This looks to be a bug, but will get more details about it internally. The driver is relatively new. Have you tried with another version, for instance JDBC: 2.6.40

  • 0 kudos
1 More Replies
AChang
by New Contributor III
  • 5439 Views
  • 2 replies
  • 2 kudos

Model Serving Endpoint keeps failing with SIGKILL error

I am trying to deploy a model in the serving endpoints section, but it keeps failing after attempting to create for an hour. Here are the service logs:Container failed with: 9 +0000] [115] [INFO] Booting worker with pid: 115[2023-09-15 19:15:35 +0000...

Get Started Discussions
model registry
Model serving
serving endpoint
serving endpoints
  • 5439 Views
  • 2 replies
  • 2 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 2 kudos

Hello @AChang, This is a common issue when the memory requirements of your model exceed the available memory on your current compute resources.   Moving to a larger compute instance with more memory can help accommodate the memory requirements of yo...

  • 2 kudos
1 More Replies
enhancederroruk
by New Contributor II
  • 3000 Views
  • 3 replies
  • 2 kudos

Chrome/Edge high memory usage for Databricks tabs.

Is it normal for Databricks tabs to be using such high memory?The Chrome example I just got a screenshot of was this (rounded up/down)...3 x Databricks tabs for one user, sized at6gb, 4.5gb, and 2gbTotal = 12.5gbI know it gets higher than this too, I...

  • 3000 Views
  • 3 replies
  • 2 kudos
Latest Reply
OliverD
New Contributor II
  • 2 kudos

Boosting. I have the same issue and it's getting out of hand.

  • 2 kudos
2 More Replies
Phani1
by Valued Contributor II
  • 420 Views
  • 1 replies
  • 0 kudos

Databricks On-Premises or in Private Cloud

Hi All,Is it possible to store/process the data on-premises or in a private cloud with Databricks? Will this choice affect costs and performance? Please advise, as the customer wants the data stored on-premises or in a private cloud for security reas...

  • 420 Views
  • 1 replies
  • 0 kudos
Latest Reply
Takuya-Omi
Valued Contributor II
  • 0 kudos

@Phani1 Databricks does not provide a product that can be directly installed and self-managed on on-premises or private cloud environments. Instead, Databricks primarily operates as a managed service on public cloud platforms such as AWS, Azure, and ...

  • 0 kudos
Phani1
by Valued Contributor II
  • 208 Views
  • 1 replies
  • 1 kudos

Multiple metastores in Unity Catalog

Hi All,Can we have more than one meta store in a region in Unity Catalog? Having a single meta store per region helps keep metadata management organized, but customers are asking for multiple meta stores for different needs. Is it possible to have se...

  • 208 Views
  • 1 replies
  • 1 kudos
Latest Reply
Takuya-Omi
Valued Contributor II
  • 1 kudos

@Phani1 When attempting to create multiple metastores in a specific region, you may encounter the following error: This region already contains a metastore. Only a single metastore per region is allowed.Databricks recommends having only one metastore...

  • 1 kudos
Phani1
by Valued Contributor II
  • 175 Views
  • 1 replies
  • 0 kudos

EMR cluster pyspark scripts to databricks

Hi All,The PySpark scripts currently operating on the EMR cluster need to be migrated to Databricks. Are there any tools available that can assist in minimizing the time required for code conversion? Your suggestions would be appreciated.Regards,Phan...

  • 175 Views
  • 1 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hello @Phani1, This guide can help you: https://www.databricks.com/resources/guide/emr-databricks-migration-guide

  • 0 kudos
Phani1
by Valued Contributor II
  • 169 Views
  • 1 replies
  • 0 kudos

Airflow jobs migration to Databricks Workflows

Hi All,We need to move our Airflow jobs over to Databricks Workflows. Are there any tools out there that can help with this migration and make the process quicker? If you have any sample code or documents that could assist, I would really appreciate ...

  • 169 Views
  • 1 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hi @Phani1, Please see this post which can help you: https://community.databricks.com/t5/data-engineering/migrating-logic-from-airflow-dags-to-databricks-workflow/td-p/104501

  • 0 kudos
BS_THE_ANALYST
by New Contributor III
  • 508 Views
  • 4 replies
  • 3 kudos

Zero to Hero - Databricks

Hi all!In a nutshell, I want to go from zero to hero with Databricks. I'd like to pursue the Databricks Data Engineering pathway, I think that makes sense as I have a background with Alteryx.I'd really like to get hands on whilst learning. Are the le...

  • 508 Views
  • 4 replies
  • 3 kudos
Latest Reply
BS_THE_ANALYST
New Contributor III
  • 3 kudos

@MariuszK thanks for the link to your medium article. There's some great stuff in there!Good point about the 30 day Azure free trial for Databricks. 

  • 3 kudos
3 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels
Top Kudoed Authors