cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Sreekanth_N
by New Contributor II
  • 3194 Views
  • 2 replies
  • 0 kudos

'NotebookHandler' object has no attribute 'setContext' in pyspark streaming in AWS

I am facing issue while calling dbutils.notebook.run() inside of pyspark streaming with concurrent.executor. At first the error is "pyspark.sql.utils.IllegalArgumentException: Context not valid. If you are calling this outside the main thread,you mus...

  • 3194 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kevin3
New Contributor III
  • 0 kudos

The error message you're encountering in PySpark when using dbutils.notebook.run() suggests that the context in which you are attempting to call the run() method is not valid. PySpark notebooks in Databricks have certain requirements when it comes to...

  • 0 kudos
1 More Replies
Khushisi
by New Contributor II
  • 701 Views
  • 0 replies
  • 0 kudos

Databricks to make a machine learning model

Hey all,I've been using a voice cloning AI and it's working well. I'm thinking of using Databricks to make a machine learning model for speech tech. I want to start with personal content creation. Any tips or advice would be great!

  • 701 Views
  • 0 replies
  • 0 kudos
Phani1
by Valued Contributor II
  • 1530 Views
  • 0 replies
  • 0 kudos

Billing usage per user

Hi Team ,Unity catalog is not enabled in our workspace, We would like to know the billing usage information per user ,could you please help us how to get these details( by using notebook level script).Regards,Phanindra

  • 1530 Views
  • 0 replies
  • 0 kudos
anandreddy23
by New Contributor III
  • 4784 Views
  • 1 replies
  • 0 kudos

unpersist doesn't clear

from pyspark.sql import SparkSessionfrom pyspark import SparkContext, SparkConffrom pyspark.storagelevel import StorageLevelspark = SparkSession.builder.appName('TEST').config('spark.ui.port','4098').enableHiveSupport().getOrCreate()df4 = spark.sql('...

  • 4784 Views
  • 1 replies
  • 0 kudos
Latest Reply
anandreddy23
New Contributor III
  • 0 kudos

Thank you so much for taking time and explaining the concepts

  • 0 kudos
MFrandsen
by New Contributor
  • 935 Views
  • 0 replies
  • 0 kudos

Question for exam project

For my exam i have to do a small project for the company im interning at. I am creating a datawarehouse where i will have to transfer data from another database, and then transforming it to a star schema. would databricks be good for this, or is it t...

  • 935 Views
  • 0 replies
  • 0 kudos
Phani1
by Valued Contributor II
  • 998 Views
  • 2 replies
  • 0 kudos

checklist for : process to move and deploy in the prod

Hi Team,Could you please help me with best practices to move and deploy (code, workspace, notebooks, etc) in the prod?Regards,Phanindra

  • 998 Views
  • 2 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

the most important is to use Repos!Link your workspace with git and use feature branches and pull requests to promote code/notebooks.Check the databricks docs on Repos.  If you have further questions; shoot.

  • 0 kudos
1 More Replies
Phani1
by Valued Contributor II
  • 2281 Views
  • 1 replies
  • 0 kudos

Archival Strategy for Delta tables

 Hi Team, We would like to define the archival strategy for data. Could you please share best practices /guide me on the below are the 3 use cases Case-1: On-Prem SQL and Oracle Data which is more than 20 years and they wanted to bring them into clou...

  • 2281 Views
  • 1 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

case 1: I'd extract the data from the db to a data lake (cold storage if that is possible, that is cheaper) using an ETL tool like Data Factory, Glue etc.  Then the archiving can take place.  Perhaps also create a backup of the data on a 2nd data lak...

  • 0 kudos
Phani1
by Valued Contributor II
  • 2730 Views
  • 2 replies
  • 0 kudos

Databricks setup/deployment checklist/best practices

Hi Team, could you please share or guide us on any checklist/best practices for Databricks setup/deployment?

  • 2730 Views
  • 2 replies
  • 0 kudos
Latest Reply
icyflame92
New Contributor II
  • 0 kudos

Hi @Phani1 , here are some best practices https://github.com/Azure/AzureDatabricksBestPractices/tree/master and you could take these points as your "checklist".Choose the right Databricks Workspace:Decide on the appropriate Azure region for your Data...

  • 0 kudos
1 More Replies
Shivam_Patil
by New Contributor II
  • 2504 Views
  • 2 replies
  • 0 kudos

Databricks Certified Associate Developer for Apache Spark 3.0 - Python sample paper

I am trying to get sample papers for Databricks Certified Associate Developer for Apache Spark 3.0 - Python before I attempt my exam 

  • 2504 Views
  • 2 replies
  • 0 kudos
Latest Reply
Cert-Bricks
Databricks Employee
  • 0 kudos

Hi, there, This exam has a Practice Exam, I think that's what you are looking for. 

  • 0 kudos
1 More Replies
Phani1
by Valued Contributor II
  • 929 Views
  • 1 replies
  • 0 kudos

Infrastructure reverse engineering

How can we do infrastructure reverse engineering in Databricks? Any kind of solution accelerator or tool that can scan through the environment and identify the solutions /components that are used in the Databricks environment and the complexity withi...

Get Started Discussions
reverse engineering
  • 929 Views
  • 1 replies
  • 0 kudos
Latest Reply
Phani1
Valued Contributor II
  • 0 kudos

Hi Team,Could you please help us on this.Regards,Phanindra

  • 0 kudos
harry546
by New Contributor III
  • 1209 Views
  • 0 replies
  • 0 kudos

Unable to log metrics to table/driver using spark StreamingQueryListener

Hi Everyone,Hope everyone is enjoying their holiday time.I have spark streaming job (copy activity from location A to location B) and a Query Listener and attached to the streaming job.As mentioned over the attachments I can see numInputRows gets log...

Get Started Discussions
Streamingmetrics
StreamingQueryListener
  • 1209 Views
  • 0 replies
  • 0 kudos
MiroFuoli
by New Contributor II
  • 1134 Views
  • 0 replies
  • 0 kudos

Unity Catalog - Limited Options for Connection Objects

I’m currently trying to create a Foreign Catalog based on a Connection object of type SQLSERVER. This would allow me to directly access our on-premises MS SQL database from within Azure Databricks using Unity Catalog.As I’m part of a large organizati...

Get Started Discussions
Connection
Foreign Catalog
SQL warehouse
Unity Catalog
  • 1134 Views
  • 0 replies
  • 0 kudos
kelvinlui1234
by New Contributor
  • 1116 Views
  • 1 replies
  • 0 kudos

Can not able to connect to ADLS Gen2 using ABFSS (urgent)

Hi all,I was having an error when trying to connect to azure data lake. "SSLHandshakeExceptio" Please see the detail below.Please help!

MicrosoftTeams-image (3).png
  • 1116 Views
  • 1 replies
  • 0 kudos
Latest Reply
ChishGayl
New Contributor II
  • 0 kudos

The error message you provided, "SSLHandshakeExceptio," indicates that there is a problem with the SSL handshake between your system and the Azure Data Lake. This issue can be caused by various factors, such as incorrect settings, network issues, or ...

  • 0 kudos
AFox
by Contributor
  • 872 Views
  • 1 replies
  • 0 kudos

Databricks Community Post Editor Issues

Anyone else constantly having errors with this editor when using any of the 'features' like code sample?Can we please have a Markdown Editor or at least the ability to edit the HTML this tool creates to fix all the bugs it makes?

  • 872 Views
  • 1 replies
  • 0 kudos
Latest Reply
AFox
Contributor
  • 0 kudos

Here is a fun one: "The message body contains h d, which is not permitted in this community. Please remove this content before sending your post."Had to add the space between h and d to be able to post it.  This means code samples can't contain `ch d...

  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels
Top Kudoed Authors