cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

namankhamesara
by New Contributor II
  • 1372 Views
  • 1 replies
  • 0 kudos

Discrepancy in Performance Reading Delta Tables from S3 in PySpark

Hello Databricks Community,I've encountered a puzzling performance difference while reading Delta tables from S3 using PySpark, particularly when applying filters and projections. I'm seeking insights to understand this variation better.I've attempte...

  • 1372 Views
  • 1 replies
  • 0 kudos
Latest Reply
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Hello Databricks Community,I've encountered a puzzling performance difference while reading Delta tables from S3 using PySpark, particularly when applying filters and projections. I'm seeking insights to understand this variation better.I've attempte...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
RozaZaharieva
by New Contributor
  • 1502 Views
  • 0 replies
  • 0 kudos

set up Azure Databricks workspace and Unity catalog - how to automate not using Terraform

Hi everyone, I am looking for a way to automate initial setup of Azure Databricks workspace and Unity Catalog but can't find anything on this topic other than Terraform. Can you share if this is possible with powershell, for example. Thank you un adv...

  • 1502 Views
  • 0 replies
  • 0 kudos
SaiNeelakantam
by New Contributor
  • 1526 Views
  • 1 replies
  • 0 kudos

How to enable "Create Vector Search Index" button in DB workspace?

How to enable "Create Vector Search Index" button in DB workspace?Following is the screenshot from the Microsoft Ignite 2023 Databricks presentation:

  • 1526 Views
  • 1 replies
  • 0 kudos
Latest Reply
PL_db
Databricks Employee
  • 0 kudos

The feature is in public preview only in some regions, you can check the available regions in the documentation here.  In addition there are certain requirements, such as a UC enabled workspace and Serverless Compute enabled, you can check all requir...

  • 0 kudos
SamGreene
by Contributor
  • 3055 Views
  • 5 replies
  • 0 kudos

CONVERT_TIMEZONE issue in DLT

I can run a query that uses the CONVERT_TIMEZONE function in a SQL notebook.  When I move the code to my DLT notebook the pipeline produces this error:Cannot resolve function `CONVERT_TIMEZONE`Here is the line:  CONVERT_TIMEZONE('UTC', 'America/Phoen...

  • 3055 Views
  • 5 replies
  • 0 kudos
Latest Reply
annn
New Contributor II
  • 0 kudos

Yes, the notebook is set to SQL and the convert_timezone function is within a select statement.

  • 0 kudos
4 More Replies
Ak_0926
by New Contributor
  • 3089 Views
  • 2 replies
  • 1 kudos

Can we get the actual query execution plan programmatically after a query is executed? Apart from UI

Let's say i have run a query and it showed me results. we can find the respective query execution plan on the UI. Is there any way we can get that execution plan through programmatically or through API?

  • 3089 Views
  • 2 replies
  • 1 kudos
Latest Reply
Walter_C
Databricks Employee
  • 1 kudos

You can obtain the query execution plan programmatically using the EXPLAIN statement in SQL. The EXPLAIN statement displays the execution plan that the database planner generates for the supplied statement. The execution plan shows how the table(s) r...

  • 1 kudos
1 More Replies
Danny_Lee
by Valued Contributor
  • 2349 Views
  • 2 replies
  • 4 kudos

Top Kudoed Author 🌟🤩🧑‍🎤

I recently saw a link to the Kudos Leaderboard for the Community Discussions.  It has always been my hope and fantasy , ever since I was a little child that I would someday be the #1 Kudoed Author on Community Discusions on community.Databricks.com....

KudosOprahGIF.gif
  • 2349 Views
  • 2 replies
  • 4 kudos
Latest Reply
Danny_Lee
Valued Contributor
  • 4 kudos

Thanks @DB_Paul - I'm on my way!   

  • 4 kudos
1 More Replies
Anku_
by New Contributor II
  • 1487 Views
  • 2 replies
  • 0 kudos

New to PySpark

Hi all,I am trying to get the domain from an email field using below expression; but getting an error.Kindly help. df.select(df.email, substring(df.email,instr(df.email,'@'),length(df.email).alias('domain')))

  • 1487 Views
  • 2 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

In your case, you want to extract the domain from the email, which starts from the position just after '@'. So, you should add 1 to the position of '@'. Also, the length of the substring should be the difference between the total length of the email ...

  • 0 kudos
1 More Replies
kickbuttowski
by New Contributor II
  • 1015 Views
  • 1 replies
  • 0 kudos

Issue in inferring schema for streaming dataframe using json files

Below is the pileine design in databricks and it's not working out , kindly look on this and let me know whether it will work or not , I'm getting json files of different schemas from directory under the root directory and it read all the files using...

  • 1015 Views
  • 1 replies
  • 0 kudos
Latest Reply
AmanSehgal
Honored Contributor III
  • 0 kudos

Could you please share some sample of your dataset and code snippet of what you're trying to implement?

  • 0 kudos
pernilak
by New Contributor III
  • 2758 Views
  • 2 replies
  • 3 kudos

Resolved! Pros and cons of physically separating data in different storage accounts and containers

When setting up Unity Catalog, it is recommended by Databricks to figure out your data isolation model when it comes to physically separating your data into different storage accounts and/or contaners. There are so many options, it can be hard to be ...

  • 2758 Views
  • 2 replies
  • 3 kudos
Latest Reply
raphaelblg
Databricks Employee
  • 3 kudos

Hello @pernilak , Thanks for reaching out to Databricks Community! My name is Raphael, and I'll be helping out. Should all catalogs and the metastore reside in the same storage account (but different containers)   Yes, Databricks recommends having o...

  • 3 kudos
1 More Replies
Mesh
by New Contributor II
  • 6828 Views
  • 1 replies
  • 0 kudos

Optimizing for Recall in Azure AutoML UI

Hi all, I've been using Azure AutoML and noticed that I can choose 'recall' as my optimization metric in the notebook but not in the Azure AutoML UI. The Databricks documentation also doesn't list 'recall' as an optimization metric.Is there a reason ...

  • 6828 Views
  • 1 replies
  • 0 kudos
Latest Reply
Mesh
New Contributor II
  • 0 kudos

On the databricks notebook itself, I can see that databricks.automl supports using recall as a primary metric Help on function classify in module databricks.automl: :param primary_metric: primary metric to select the best model. Each trial will...

  • 0 kudos
swapnilmd
by New Contributor
  • 884 Views
  • 1 replies
  • 0 kudos

Databricks Web Editor's Cell like UI in local IDE

I want to have databricks related developement locally.There is extension that allows to run local python file on remote databricks cluster.But I want to have cell like structure that is present in databricks UI for python files in local IDE as well....

  • 884 Views
  • 1 replies
  • 0 kudos
Latest Reply
daniel_sahal
Esteemed Contributor
  • 0 kudos

@swapnilmd You can use VSCode extension for Databricks.https://docs.databricks.com/en/dev-tools/vscode-ext/index.html

  • 0 kudos
NhanNguyen
by Contributor II
  • 1421 Views
  • 3 replies
  • 1 kudos

[Memory utilization in Metrics Tab still display after terminate a cluster]

Hi All,Could you guys help me to check this?I run a cluster and then terminate that cluster but when i navigate to the Metrics tab of Cluster still see the Memory utilization show metrics.Thanks

jensen22_0-1710993062168.png
  • 1421 Views
  • 3 replies
  • 1 kudos
Latest Reply
NhanNguyen
Contributor II
  • 1 kudos

here are my cluster display and my simple notebook:

  • 1 kudos
2 More Replies
HakanNordgren
by New Contributor II
  • 1911 Views
  • 3 replies
  • 0 kudos

databricks-jdbc lists `spark_catalog` among catalogs for Standard tier Azure workspace

databricks-jdbc lists `spark_catalog` among catalogs for Standard tier Azure workspace. The UI lists `hive_metastore`. It would be better if these two were consistent.

  • 1911 Views
  • 3 replies
  • 0 kudos
Latest Reply
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
databricks-jdbc lists `spark_catalog` among catalogs for Standard tier Azure workspace. The UI lists `hive_metastore`. It would be better if these two were consistent.

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
2 More Replies
Chris_Konsur
by New Contributor III
  • 2023 Views
  • 3 replies
  • 1 kudos

an autoloader in file notification mode to get files from S3 on AWS -Error

I configured an autoloader in file notification mode to get files from S3 on AWS.spark.readStream\.format("cloudFiles")\.option("cloudFiles.format", "json")\.option("cloudFiles.inferColumnTypes", "true")\.option("cloudFiles.schemaLocation", "dbfs:/au...

  • 2023 Views
  • 3 replies
  • 1 kudos
Latest Reply
Selz
New Contributor II
  • 1 kudos

In case anyone else stumbles across this, I was able to fix my issue by setting up an instance profile with the file notification permissions and attaching the instance profile to the job cluster. It wasn't clear from the documentation that the file ...

  • 1 kudos
2 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Top Kudoed Authors