cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Community Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
rudyevers
by • New Contributor III
  • 3372 Views
  • 6 replies
  • 5 kudos

Resolved! Unity catalog - external table lastUpdateversion

We are currently upgrading our Lakehouse to use the Unity Catalog benefits. We will mostly use external tables because alle our DETLA tables are already stored in Azure Storage. I try to figure out how to update the table property "delta.lastUpdateve...

Community Discussions
Reply
  • 3372 Views
  • 6 replies
  • 5 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 5 kudos

I am in the same boat.That is the reason I opted to use managed tables instead.  OK; it means migrating tables and changing notebooks but besides not having to struggle with external tables, you also get something in return (liquid clustering f.e.).

  • 5 kudos
5 More Replies
AlexVB
by • New Contributor III
  • 2478 Views
  • 0 replies
  • 3 kudos

Metabase support

Databricks x MetabaseHi, as someone who previously used Metabase as their self-service BI tool in their org, I was disappointed to see that Databricks is not supported officially: https://www.metabase.com/data_sources/The community drivers project is...

Community Discussions
Reply
  • 2478 Views
  • 0 replies
  • 3 kudos
neca36
by • New Contributor
  • 1071 Views
  • 1 replies
  • 0 kudos

Databricks data engineer associate got paused

Hi team,I've faced a disappointing experience during my first certification attempt and need help in resolving the issue.While attending the certification - Databricks data engineer associate on each 2-3 questions I kept receiving a message that the ...

Community Discussions
Reply
  • 1071 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Thank you for posting your concern on Community! To expedite your request, please list your concerns on our ticketing portal. Our support staff would be able to act faster on the resolution (our standard resolution time is 24-48 hours).

  • 0 kudos
Shree23
by • New Contributor III
  • 1985 Views
  • 6 replies
  • 0 kudos

Primary key and not null

Hi Expert, how we can get primary key and not null and cluster index in table creation%Sqlcreate table table1 values (id int , product char) expected outputcreate table table1 values (id int  not null primary key, product char) and cluster index  

Community Discussions
Reply
  • 1985 Views
  • 6 replies
  • 0 kudos
View Replies
Latest Reply
Shree23
New Contributor III
  • 0 kudos

sugggestion pls

  • 0 kudos
5 More Replies
mvmiller
by • New Contributor III
  • 802 Views
  • 1 replies
  • 0 kudos

Sharing compute between tasks of a job

Is there a way to set up a workflow with multiple tasks, so that different tasks can share the same compute resource, at the same time?I understand that an instance pool may be an option, here. Wasn't sure if there were other possible options to cons...

Community Discussions
Reply
  • 802 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @mvmiller , Certainly! When orchestrating workflows with multiple tasks, it’s essential to optimize resource usage.   Here are a couple of approaches you can consider: GitHub Actions: By default, GitHub Actions runs multiple jobs in parallel. Howe...

  • 0 kudos
Bhanu1
by • New Contributor III
  • 1009 Views
  • 1 replies
  • 0 kudos

Thoughts on how to improve string search queries

Please see sample code I am running below. What options can I explore to improve speed of query execution in such a scenario? Current full code takes about 4 hrs to run on 1.5 billion rows. Thanks!SELECT fullVisitorId ,VisitId ,EventDate ,PagePath ,d...

Community Discussions
Reply
  • 1009 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Bhanu1, When dealing with large datasets and slow query execution, there are several strategies you can explore to improve performance. Let’s dive into some options: Indexing: Indexing is a critical technique for enhancing SQL query performance o...

  • 0 kudos
rpl
by • New Contributor III
  • 1319 Views
  • 1 replies
  • 0 kudos

Resolved! Read file with Delta Live Tables from external location (Unity Catalog)

As far as I understand, Delta Live Tables should now support reading data from an external location, but I can’t get it to work. I’ve added an ADLS container to Unity Catalog as an external location. There’s a folder in the container containing an ex...

Community Discussions
Delta Live Tables
Unity Catalog
Reply
  • 1319 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
rpl
New Contributor III
  • 0 kudos

I misspelled the folder name; I got it working now  The error message could have been more informative

  • 0 kudos
Jun_NN
by • New Contributor
  • 4775 Views
  • 2 replies
  • 1 kudos

Deleted the s3 bucket assocated with metastore

I deleted the aws s3 bucket for the databricks metastore by mistake.How to fix this? can I re-create the s3 bucket? Or can I delete the metastore (I don't have much data in it), and re-generate one? Thank you!

Community Discussions
Reply
  • 4775 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @Jun_NN , Indeed, deleting the AWS S3 bucket for the Databricks metastore can be a nerve-wracking task. However, there are strategies to recover from such situations. Let’s explore some options: Regenerate Metastore: If your metastore doesn’t ...

  • 1 kudos
1 More Replies
hagarciaj
by • New Contributor
  • 609 Views
  • 1 replies
  • 0 kudos

Highly Performant Data Ingestion and Processing Pipelines

Hi everyone,I am working on a project that requires highly performant pipelines for managing data ingestion, validation, and processing large data volumes from IOT devices.I am interested in knowing:- The best way to ingest from EventHub/Kafka sinks-...

Community Discussions
Reply
  • 609 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @hagarciaj, Certainly! Handling data pipelines for large volumes from IoT devices is crucial.   Let’s dive into each aspect:   Ingestion from EventHub/Kafka Sinks: Azure Event Hubs provides an Apache Kafka endpoint, allowing you to connect using t...

  • 0 kudos
Adil
by • New Contributor
  • 1094 Views
  • 1 replies
  • 0 kudos

Find value in any column in a table

Hi,I'm not sure if this is a possible scenario, but is there, by any chance a way to query all the columns of a table for searching a value? Explanation: I want to search for a specific value in all the columns of a databricks table. I don't know whi...

Community Discussions
Reply
  • 1094 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Adil, Certainly! When you need to search for a specific value across all columns in a table, you can use SQL queries to achieve this. You can construct a query that checks each column for the desired value.   Here are a few approaches you can con...

  • 0 kudos
osalawu
by • New Contributor
  • 697 Views
  • 2 replies
  • 0 kudos

Got Suspended while taking Databricks Certified Data Analyst Associate Assessment

Hi Team, What I experienced today from the proctor was so not nice, my experience with proctor today was very frustrating and pathetic, I was taking my assessment today 11-26-2023, 2pm, I was already on between question 22nd to 25th when my assessmen...

Community Discussions
Reply
  • 697 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Cert-Team
Honored Contributor III
  • 0 kudos

@osalawu Sorry to hear you had an issue with your exam. In order to protect your Webassessor account information, please file a ticket with our support team. Please include your Webassessor login ID, the exam, and a couple of dates and times that wil...

  • 0 kudos
1 More Replies
Klusener
by • New Contributor
  • 795 Views
  • 0 replies
  • 0 kudos

Arguments parsing in Databricks python jobs

On Databricks created a job task with task type as Python script from s3. However, when arguments are passed via Parameters option, running into unrecognized arguments' error.Code in s3 file:import argparse def parse_arguments(): parser = argpar...

Community Discussions
Reply
  • 795 Views
  • 0 replies
  • 0 kudos
eallain
by • New Contributor
  • 804 Views
  • 0 replies
  • 0 kudos

Structured Streaming - Kafka Offset Management

In my team, we decided to move from spark streaming to structured streaming, mainly cause it says that it's legacy and we want to benefit new features from structured streaming.However we have an issue with committing offsets.Previously on spark stre...

Community Discussions
Reply
  • 804 Views
  • 0 replies
  • 0 kudos
RahulPatidar
by • New Contributor II
  • 3967 Views
  • 5 replies
  • 0 kudos

Resolved! dataset.cache() not working : NoSuchObjectException(message:There is no database named global_temp)

ERROR RetryingHMSHandler: NoSuchObjectException(message:There is no database named global_temp)at org.apache.hadoop.hive.metastore.ObjectStore.getMDatabase(ObjectStore.java:508)at org.apache.hadoop.hive.metastore.ObjectStore.getDatabase(ObjectStore.j...

Community Discussions
Reply
  • 3967 Views
  • 5 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @RahulPatidar , The error message you’re encountering, “NoSuchObjectException (message: There is no database named global_temp),” is related to the use of the special database called “global_temp” in Spark.Here’s what you need to know: Global Tem...

  • 0 kudos
4 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels