cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

abueno
by Contributor
  • 1164 Views
  • 1 replies
  • 0 kudos

Databricks Pyspark filter several columns with similar criteria

I am querying a table from the Databricks Catalog which I have to filter several columns with the same criteria.  below is what I have created so far.  I have 10 columns that I have filter with a set of criteria from (dx_list1) and another 10 that I ...

  • 1164 Views
  • 1 replies
  • 0 kudos
Latest Reply
filipniziol
Esteemed Contributor
  • 0 kudos

Hi @abueno ,As I understand the logic you want to implement is:1. For every pair of columns:First Column (DX_i): Must be in dx_list1Second Column (DX_{i+1}): Must be in dx_list22. The condition for each pair is:col('DX_i').isin(dx_list1) OR col('DX_{...

  • 0 kudos
ayush19
by New Contributor III
  • 2420 Views
  • 3 replies
  • 1 kudos

How to retrieve Spark Session inside java jar library installed on Cluster

I have a java app in form of jar package. This jar is installed on a Databricks cluster. This jar package reads and writes to few tables in databricks. In order to achieve that, I need SparkSession available in the code. Given that spark session is a...

  • 2420 Views
  • 3 replies
  • 1 kudos
Latest Reply
IslaGray
New Contributor II
  • 1 kudos

Thanks for the update, I will try it too.

  • 1 kudos
2 More Replies
harshshah
by New Contributor
  • 743 Views
  • 0 replies
  • 0 kudos

Course registration issue

Hi All,I'm trying to register for "Spark Developer course" which is free course but I don't see this course under the course catalogue and if I try to directly access using a link I get 403 forbidden error. Can anyone help me figure out the issue.

  • 743 Views
  • 0 replies
  • 0 kudos
rseshadri
by New Contributor III
  • 2802 Views
  • 7 replies
  • 0 kudos

Unable to install the Databricks in AWS

I have followed the instructions given in the following link https://docs.databricks.com/en/admin/workspace/quick-start.html#use-aws-quickstart-to-create-a-databricks-workspace%22There was an error in CFT while deploying the workspace. Please help[ER...

  • 2802 Views
  • 7 replies
  • 0 kudos
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 0 kudos

Hi @rseshadri ,I'm more familiar with Azure, but this error is most commonly caused by entering the wrong credentials in the Cloudformation template.Try to launch the Quickstart again from the Databricks Account Console and double check that you ente...

  • 0 kudos
6 More Replies
reddybhargava
by New Contributor II
  • 891 Views
  • 2 replies
  • 1 kudos

How to delete object permissions using REST API

We have REST API methods to set,update,Get object permissions documented here https://docs.databricks.com/api/workspace/permissionsIs there a way to revoke the permissions which are granted through REST API

  • 891 Views
  • 2 replies
  • 1 kudos
Latest Reply
reddybhargava
New Contributor II
  • 1 kudos

Thank you for your response!We have a situation where we assigned CAN_BIND permissions to a group for a service principal using the following:$ vim grant-service-principal.json{"access_control_list": [{"user_name": "{username}","permission_level": "C...

  • 1 kudos
1 More Replies
Sardenberg
by New Contributor II
  • 13273 Views
  • 3 replies
  • 1 kudos

How list all USERs FROM a especific GROUP USING SQL?

I want to list, using sql editor, all users name from a specific group.Reading documentation, I only learned how to show the groups or the users, using simples filters, like:SHOW GROUPS LIKE '*XPTO*';SHOW GROUPS WITH USER `test@gmail.com`SHOW USERS L...

  • 13273 Views
  • 3 replies
  • 1 kudos
Latest Reply
Robin63
New Contributor II
  • 1 kudos

I don't think it's possible yet. Unfortunately, I look in all system tables and command and didn't found this kind of things.But with a Python notebook, like what did the AI, you can reconstruct it:first you list all the users withSHOW USERSthen you ...

  • 1 kudos
2 More Replies
devendra_tomar
by New Contributor
  • 2902 Views
  • 3 replies
  • 0 kudos

How to Read Data from Databricks Worker Nodes in Unity Catalog Volume

I am currently working on a similarity search use case where we need to extract text from PDF files and create a vector index. We have stored our PDF files in a Unity Catalog Volume, and I can successfully read these files from the driver node.Here's...

  • 2902 Views
  • 3 replies
  • 0 kudos
Latest Reply
ayush007
New Contributor II
  • 0 kudos

I am also looking for it in databricks runtime version 15.4

  • 0 kudos
2 More Replies
anialos
by New Contributor III
  • 1413 Views
  • 2 replies
  • 0 kudos

Resolved! Databricks Certified Associate Developer for Apache Spark 3 Got Suspended. Require Immediate support

Hello @Cert-Team  @Certificate Team,Request Id# 00544837I encountered a pathetic experience while attempting my Databricks Certified Associate Developer for Apache Spark 3 certification exam. I had answered more than 44 questions   I applied for Data...

  • 1413 Views
  • 2 replies
  • 0 kudos
Latest Reply
anialos
New Contributor III
  • 0 kudos

The issue is resolved . Please close the ticket 

  • 0 kudos
1 More Replies
HemanthG
by New Contributor
  • 1364 Views
  • 3 replies
  • 0 kudos

Unable to install databricks free trial on AWS

I was trying to install databricks on AWS using these instructions "https://docs.databricks.com/en/admin/workspace/quick-start.html#use-aws-quickstart-to-create-a-databricks-workspace". But when CloudFormation Template stack executed "createCredentia...

  • 1364 Views
  • 3 replies
  • 0 kudos
Latest Reply
rseshadri
New Contributor III
  • 0 kudos

I am facing the same error when installing the Databricks in AWS, Please find attached the error below and help [ERROR] 2024-10-08T12:11:56.458Z ad8dd24a-67ec-4941-8096-3f8d883994b2 Exception: Expecting value: line 1 column 1 (char 0)Traceback (most ...

  • 0 kudos
2 More Replies
shubbansal27
by New Contributor II
  • 16622 Views
  • 9 replies
  • 3 kudos

How to disable task in databricks job workflow

I don't see any option to disable the task. I have created one workflow which contains multiple tasks.sometime i need to disable some task for my experiment.  but there is no straightforward way for this.  I feel "disable" button is must have option ...

  • 16622 Views
  • 9 replies
  • 3 kudos
Latest Reply
VickiT
New Contributor II
  • 3 kudos

I know this post was a while ago, but I've hit this issue, and come up with using if/else conditions between the tasks with a Job parameter of "IsActiveTask2" with a True or False, and then I can have the Task2 dependant on the success of this condit...

  • 3 kudos
8 More Replies
Himanshu4
by New Contributor II
  • 1205 Views
  • 1 replies
  • 0 kudos

Automating the Migration of ML Models and Metadata to Unity Catalog

Hello everyone,I am currently working on a project where I need to migrate ML models from the Model Registry of one Databricks workspace to the Unity Catalog (schema/catalog) of another workspace. This migration needs to include not only the models t...

  • 1205 Views
  • 1 replies
  • 0 kudos
Latest Reply
gchandra
Databricks Employee
  • 0 kudos

Deploy as Code is recommended. https://learn.microsoft.com/en-us/azure/databricks/machine-learning/mlops/deployment-patterns#deploy-code If you need to copy the model as a Pickle file then use https://github.com/mlflow/mlflow-export-import CICD imple...

  • 0 kudos
Anish_2
by New Contributor II
  • 1516 Views
  • 1 replies
  • 0 kudos

Unable to clone delta live table

Hello Team,In order to avoid any cost of running workflow for historical data, i am trying DEEP clone in order to copy data of streaming table from PROD workspace to QA workspace but it is giving below error. Please suggest some efficient data copy m...

Get Started Discussions
Databricks
Delta Live Tables
  • 1516 Views
  • 1 replies
  • 0 kudos
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 0 kudos

Hi @Anish_2 ,This is well known limitation of DLT. You can read about it at limitation section in documentation. Conclusion is following. If you want to have ability to use all feature of delta protocol, it's better not to use dlt framework. It's gre...

  • 0 kudos
satya1206
by New Contributor II
  • 1121 Views
  • 1 replies
  • 0 kudos

Price of UCX assessment and migration

Hi,I am planning to migrate few tables and workflows from Hive metastore to Unity catalog( For example 8 tables and 2 small) workflows. What is the cost of using UCX (Unity Catalog Upgrade eXperience) for assessment and migration of artifacts from a ...

  • 1121 Views
  • 1 replies
  • 0 kudos
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 0 kudos

Hi @satya1206 ,There is no place involved when it comes to using ucx. It's free tool, open source tool created by databricks labs. More info you can find on below: https://github.com/databrickslabs/ucx

  • 0 kudos

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels