cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Phani1
by Valued Contributor
  • 1421 Views
  • 1 replies
  • 1 kudos

Unity catalog Migration

Hi Team,Could you please help me to understand,  1)Why we need to migrate Unity catalog? if we are not migrating what benefits we will not get?2) How to migrate Unity catalog (What all are objects needs to migrate and any tool) ? Regards,Phanindra

  • 1421 Views
  • 1 replies
  • 1 kudos
Latest Reply
Kaniz
Community Manager
  • 1 kudos

Hi @Phani1, Good Question!   Why Migrate to Unity Catalog?   Unity Catalog is a unified governance solution for Databricks workspaces. Without it, each Databricks workspace connects to a Hive metastore and maintains a separate service for Table Acces...

  • 1 kudos
Sreekanth_N
by New Contributor II
  • 1373 Views
  • 3 replies
  • 0 kudos

'NotebookHandler' object has no attribute 'setContext' in pyspark streaming in AWS

I am facing issue while calling dbutils.notebook.run() inside of pyspark streaming with concurrent.executor. At first the error is "pyspark.sql.utils.IllegalArgumentException: Context not valid. If you are calling this outside the main thread,you mus...

  • 1373 Views
  • 3 replies
  • 0 kudos
Latest Reply
Kevin3
New Contributor III
  • 0 kudos

The error message you're encountering in PySpark when using dbutils.notebook.run() suggests that the context in which you are attempting to call the run() method is not valid. PySpark notebooks in Databricks have certain requirements when it comes to...

  • 0 kudos
2 More Replies
anandreddy23
by New Contributor II
  • 1889 Views
  • 2 replies
  • 1 kudos

unpersist doesn't clear

from pyspark.sql import SparkSessionfrom pyspark import SparkContext, SparkConffrom pyspark.storagelevel import StorageLevelspark = SparkSession.builder.appName('TEST').config('spark.ui.port','4098').enableHiveSupport().getOrCreate()df4 = spark.sql('...

  • 1889 Views
  • 2 replies
  • 1 kudos
Latest Reply
anandreddy23
New Contributor II
  • 1 kudos

Thank you so much for taking time and explaining the concepts

  • 1 kudos
1 More Replies
doublesteakhous
by New Contributor
  • 540 Views
  • 1 replies
  • 0 kudos

SQL UDFs not visible in notebooks

We are using a serverless SQL warehouse and managed tables in unity catalog in Azure Databricks. When usind the designated catalog tab, I can see and filter for functions, but when I am developing in a notebook, there are only tables and views visibl...

  • 540 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @doublesteakhous, When working with Azure Databricks Notebooks, you might notice that the smaller catalog view to the side only displays tables and views, but not functions. However, there is a way to access and explore functions within your noteb...

  • 0 kudos
valjas
by New Contributor III
  • 2421 Views
  • 3 replies
  • 1 kudos

Clusters are really slow

We have two environments for our Azure Databricks. Dev and Prod. We had clusters created and tested in Dev environment, then they were exported to the prod environment through APIs. The clusters in Dev are performing as expected. Whereas, the cluster...

  • 2421 Views
  • 3 replies
  • 1 kudos
Latest Reply
Kaniz
Community Manager
  • 1 kudos

Hi @valjas, Workspace Creation and Cluster Performance: Actions taken during the creation of a workspace can indeed impact cluster performance. When setting up a workspace, consider the following factors: Configuration Settings: Ensure that the wor...

  • 1 kudos
2 More Replies
Employee_HT
by New Contributor
  • 718 Views
  • 1 replies
  • 0 kudos

Error while using the latest version of databricks in an Mulesoft Application

HI All,I use the latest version of Databricks (2.6.34) in my Mulesoft Application to and the code it deployed successfully in local and while testing i have observed the below mentioned error.Kindly check and help resolving the issue.at com.databrick...

  • 718 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @Employee_HT,  The error stack trace you’ve provided seems to be related to Databricks and JDBC. Let’s break it down: The error starts with a call to com.databricks.client.jdbc42.internal.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4j...

  • 0 kudos
lrajagopalan
by New Contributor
  • 381 Views
  • 1 replies
  • 0 kudos

Disable DLT notifications in development mode

Hello,We are using DLT pipelines for many of our jobs with notifications on failures to slack.Wondering if there is a clean way to disable the alerts when in development mode. It does make sense to have it turned off in dev, doesn't it?

  • 381 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @lrajagopalan, Yes, it makes sense to turn off notifications during development mode to avoid unnecessary alerts. Notifications in Databricks are usually configured at the job level or task level. 

  • 0 kudos
gyapar
by New Contributor II
  • 2731 Views
  • 1 replies
  • 0 kudos

Job Clusters With Multiple Tasks

Hi all,I'm trying to do creating one job cluster with one configuration or specification which has a workflow and this workflow needs to have 3 dependent tasks as a straight line. For example, t1->t2->t3. In databricks there are some constraints also...

  • 2731 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @gyapar, Certainly! Let’s dive into your questions about Databricks job clusters, orchestration, and scaling.   Utilizing Databricks Job Clusters: A job cluster in Databricks is a non-interactive way to run an application, such as an ETL job or d...

  • 0 kudos
enhancederroruk
by New Contributor II
  • 986 Views
  • 2 replies
  • 0 kudos

Chrome/Edge high memory usage for Databricks tabs.

Is it normal for Databricks tabs to be using such high memory?The Chrome example I just got a screenshot of was this (rounded up/down)...3 x Databricks tabs for one user, sized at6gb, 4.5gb, and 2gbTotal = 12.5gbI know it gets higher than this too, I...

  • 986 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @enhancederroruk, Databricks is a powerful platform for big data analytics and machine learning, but it can indeed be memory-intensive, especially when running complex workloads.   Let’s explore some aspects related to memory usage in Databricks: ...

  • 0 kudos
1 More Replies
Phani1
by Valued Contributor
  • 425 Views
  • 2 replies
  • 0 kudos

checklist for : process to move and deploy in the prod

Hi Team,Could you please help me with best practices to move and deploy (code, workspace, notebooks, etc) in the prod?Regards,Phanindra

  • 425 Views
  • 2 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

the most important is to use Repos!Link your workspace with git and use feature branches and pull requests to promote code/notebooks.Check the databricks docs on Repos.  If you have further questions; shoot.

  • 0 kudos
1 More Replies
Phani1
by Valued Contributor
  • 829 Views
  • 1 replies
  • 0 kudos

Archival Strategy for Delta tables

 Hi Team, We would like to define the archival strategy for data. Could you please share best practices /guide me on the below are the 3 use cases Case-1: On-Prem SQL and Oracle Data which is more than 20 years and they wanted to bring them into clou...

  • 829 Views
  • 1 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

case 1: I'd extract the data from the db to a data lake (cold storage if that is possible, that is cheaper) using an ETL tool like Data Factory, Glue etc.  Then the archiving can take place.  Perhaps also create a backup of the data on a 2nd data lak...

  • 0 kudos
Phani1
by Valued Contributor
  • 1088 Views
  • 2 replies
  • 0 kudos

Databricks setup/deployment checklist/best practices

Hi Team, could you please share or guide us on any checklist/best practices for Databricks setup/deployment?

  • 1088 Views
  • 2 replies
  • 0 kudos
Latest Reply
icyflame92
New Contributor II
  • 0 kudos

Hi @Phani1 , here are some best practices https://github.com/Azure/AzureDatabricksBestPractices/tree/master and you could take these points as your "checklist".Choose the right Databricks Workspace:Decide on the appropriate Azure region for your Data...

  • 0 kudos
1 More Replies
karthik_p
by Esteemed Contributor
  • 468 Views
  • 1 replies
  • 0 kudos

we have enabled compute schema in system tables, not able to see cluster and node_type tables

we have enabled system tables, under system tables we are able to see compute schema, under that we are not able to see cluster and node_type in azure databricks. do we have any limitations for above tables 

  • 468 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Certainly, karthik_p! Let’s address your query regarding the cluster and node_type tables in Azure Databricks. Cluster Tables:Cluster-related information is essential for managing and monitoring your Databricks clusters. However, there are some consi...

  • 0 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels
Top Kudoed Authors