cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

fperry
by New Contributor III
  • 1450 Views
  • 6 replies
  • 0 kudos

Issue with Multiple Stateful Operations in Databricks Structured Streaming

Hi everyone,I'm working with Databricks structured streaming and have encountered an issue with stateful operations. Below is my pseudo-code: df = df.withWatermark("timestamp", "1 second") df_header = df.withColumn("message_id", F.col("payload.id"))...

  • 1450 Views
  • 6 replies
  • 0 kudos
Latest Reply
fperry
New Contributor III
  • 0 kudos

This should according to this blog post basically work, right? However, I'm getting the same errorMultiple Stateful Streaming Operators | Databricks BlogOr am I missing something? rate_df = spark.readStream.format("rate").option("rowsPerSecond", "1")...

  • 0 kudos
5 More Replies
Maria_fed
by New Contributor III
  • 4669 Views
  • 7 replies
  • 0 kudos

Need help migrating company customer and partner academy accounts to work properly

Hi, originally I accidentally made a customer academy account with my company that is a databricks partner. Then I made an account using my personal email and listed my company email as the partner email for the partner academy account. that account ...

  • 4669 Views
  • 7 replies
  • 0 kudos
Latest Reply
adage_123
New Contributor II
  • 0 kudos

I have the same problem. How do I change from customer to partner?

  • 0 kudos
6 More Replies
Floris
by New Contributor II
  • 948 Views
  • 3 replies
  • 0 kudos

Databricks Apps have become unavailable

We've become unable to access or renable Apps in our workspace. As of yesterday the tab "Apps" has dissapeared from the compute page in our workspaces. Additionally the toggle that we originally used to enable the public preview for apps has dissapea...

  • 948 Views
  • 3 replies
  • 0 kudos
Latest Reply
Floris
New Contributor II
  • 0 kudos

Hi @Renu_, thanks for the advice. I just did a double check in the account panel but I couldn't find anything there that seemed related to enabling Apps that we have access to. Would you happen to know which specific preview/setting we would need to ...

  • 0 kudos
2 More Replies
aswinkks
by New Contributor III
  • 1083 Views
  • 2 replies
  • 2 kudos

Resolved! Unstructured Data (Images) training in Databricks

I'm looking for a solution where1. Need a delta table that saves the pointers(path) of the images in volume2. Train a Pytorch, tensorflow or keras model with the data from delta lake.I tried multiple methods :1. Storing the data as vectors or binary ...

  • 1083 Views
  • 2 replies
  • 2 kudos
Latest Reply
BigRoux
Databricks Employee
  • 2 kudos

I managed to find a few solution accelerators that are in the ballpark, albeit not exact, to what you are trying to accomplish.   Have a look: 1. https://www.databricks.com/solutions/accelerators/digital-pathology 2. https://www.databricks.com/resour...

  • 2 kudos
1 More Replies
freshmint
by New Contributor II
  • 978 Views
  • 4 replies
  • 0 kudos

How to get Databricks usage invoices?

Hey guys,I'm wondering if there are people who wanted to see invoices? I've been using Databricks and I registered my credit card. I've been paying for it.Now I just want to see the invoices but I can't find it. Is there anybody who experienced simil...

  • 978 Views
  • 4 replies
  • 0 kudos
Latest Reply
Advika
Databricks Employee
  • 0 kudos

Hello @freshmint! To clarify, are you looking for invoices related to courses you've purchased, or are you referring to other Databricks services?

  • 0 kudos
3 More Replies
DbricksLearner1
by New Contributor
  • 446 Views
  • 1 replies
  • 0 kudos

How do I display output from applyinPandas function?

I'm using databricks version 13.3. I have a function which I'm calling by using the applyInPandas function. I need to see the attributes of my df dataset which I'm using inside my function. My sample code looks likedef train_model(df):# Copy input Da...

  • 446 Views
  • 1 replies
  • 0 kudos
Latest Reply
BigRoux
Databricks Employee
  • 0 kudos

Here are some idead/approaches to consider:   To inspect the attributes of a df dataset within a function used in applyInPandas on a Databricks Runtime 13.3 cluster, you can use debugging techniques that help you explore the structure and content of ...

  • 0 kudos
shubham_007
by Contributor III
  • 450 Views
  • 1 replies
  • 0 kudos

Assistance Needed with Databricks DQX Framework for Data Quality

 Hi Community Experts,I hope this message finds you well. Our team is currently working on enhancing data quality within our Databricks environment and we are utilizing the Databricks DQX framework for this purpose. We are seeking detailed guidance a...

  • 450 Views
  • 1 replies
  • 0 kudos
Latest Reply
Advika
Databricks Employee
  • 0 kudos

Hello @shubham_007! It looks like this post duplicates the one you recently posted. A response has already been provided there. I recommend continuing the discussion in that thread to keep the conversation focused and organised

  • 0 kudos
POB756
by New Contributor II
  • 755 Views
  • 3 replies
  • 0 kudos

The case of the phantom files!

I developed a notebook that uses *.py files as module imports.  On my cluster, the logic in the notebook works fine, my custom modules get loaded, code executes.Using an AzDO pipeline, I deploy the notebook and supporting files to a separate workspac...

  • 755 Views
  • 3 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

Got it.My background is not SWE, I have always been a 'data guy', but I definitely appreciate a proper dev workflow (ci/cd, git integration, tests).When we started using databricks like 7 or 8 years ago, we went for notebooks as this got us up to spe...

  • 0 kudos
2 More Replies
CT123456
by New Contributor II
  • 1440 Views
  • 3 replies
  • 0 kudos

Access S3 Tables from Databricks

Hi, When I was looking for how to access S3 Tables from Databricks, I came across this AWS blog post: https://aws.amazon.com/blogs/big-data/access-amazon-s3-iceberg-tables-from-databricks-using-aws-glue-iceberg-rest-catalog-in-amazon-sagemaker-lakeho...

  • 1440 Views
  • 3 replies
  • 0 kudos
Latest Reply
ashishmor
New Contributor II
  • 0 kudos

Not sure if it Databricks is supporting same way as snowflake but this is what I found (https://docs.databricks.com/aws/en/connect/storage/amazon-s3) This was updated in February 2025. Hope It helps.

  • 0 kudos
2 More Replies
Henrik
by New Contributor III
  • 2650 Views
  • 3 replies
  • 2 kudos

UI menu customisation

If I want to customize the UI menu so some users/groups can't for example create jobs or make experiments etc.As I see it, when unity catalog is enabled, everyone can create jobs (If they have attach to cluster permission). But in my organization, th...

  • 2650 Views
  • 3 replies
  • 2 kudos
Latest Reply
pdiamond
Contributor
  • 2 kudos

I would also like a way to customize what users see when they log in. For example we don't want most of them to even have the option of creating a Genie Space so ideally we could simply remove "Genie" from their menu

  • 2 kudos
2 More Replies
Klusener
by Contributor
  • 1655 Views
  • 11 replies
  • 14 kudos

Smaller dataset causing OOM on large cluster

I have a pyspark job reading the input data volume of just ~50-55GB Parquet data from a delta table on Databricks. Job is using n2-highmem-4 GCP VM and 1-15 worker with autoscaling on databricks. Each workerVM of type n2-highmem-4 has 32GB memory and...

  • 1655 Views
  • 11 replies
  • 14 kudos
Latest Reply
mark_ott
Databricks Employee
  • 14 kudos

Next, use the repartition(n) to increase your dataframe to twice the number you got earlier. For example, if num_partitions was 30, then repartition(60) prior to running your query.  With half the data in each Memory Partition, I'm guessing you won't...

  • 14 kudos
10 More Replies
Arnold_Souza
by New Contributor III
  • 10392 Views
  • 9 replies
  • 1 kudos

How to move a metastore to a new Storage Account in unity catalog?

Hello, I would like to change the Metastore location in Databricks Account Console. I have one metastore created that is in an undesired container/storage account. I could see that it's not possible to edit a metastore that is already created. I coul...

1.JPG
  • 10392 Views
  • 9 replies
  • 1 kudos
Latest Reply
Bakkie
New Contributor III
  • 1 kudos

@TugrulA we repopulated our SQL warehouse by rerunning all Pipelines (Delta Live Tables) that read our raw data from an Azure storage account (different than the Metastore storage container) and write the bronze/silver/gold tables to the (new) Metast...

  • 1 kudos
8 More Replies
TugrulA
by New Contributor II
  • 1418 Views
  • 2 replies
  • 0 kudos

Databricks Unity Catalog Metastore

Hey everyone,I deleted my Unity Catalog metastore and now want to point it to another Azure storage account (ADLS). However, once a metastore is created, its storage location cannot be changed. Therefore, I deleted the existing metastore and created ...

  • 1418 Views
  • 2 replies
  • 0 kudos
Latest Reply
sarahbhord
Databricks Employee
  • 0 kudos

Hey TugrulA - 1. Deleting a Unity Catalog Metastore permanently removes all associated objects, and the new metastore wont automatically include original objects. Unfortunately automatic recover is not possible. While UC allows UNDROP for individual ...

  • 0 kudos
1 More Replies
mrstevegross
by Contributor III
  • 1007 Views
  • 2 replies
  • 0 kudos

Resolved! Trying to understand why a cluster reports as "terminating" right after being created

We use a "warmup" mechanism to get our DBR instance pool into a state where it has at-least-N instances. The logic is:For N repetitions:Request a new DBR cluster in the pool (which causes the pool to request an AWS instance)Wait for the cluster to re...

  • 1007 Views
  • 2 replies
  • 0 kudos
Latest Reply
mrstevegross
Contributor III
  • 0 kudos

Aha, found it. I monitored the pool status via the DBR UI, and when a cluster *started* being provisioned, I clicked into it. Then I looked at the event log, and found useful information about failed steps. The underlying error was indeed AWS related...

  • 0 kudos
1 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels