cancel
Showing results for 
Search instead for 
Did you mean: 
Page Title

Welcome to the Databricks Community

Discover the latest insights, collaborate with peers, get help from experts and make meaningful connections

102819members
53254posts
cancel
Showing results for 
Search instead for 
Did you mean: 
Unity Catalog Lakeguard: Industry-first and only data governance for multi-user Apache™ Spark cluste

Run Scala, Python and SQL workloads on shared, cost-efficient multi-user compute. We are thrilled to announce Unity Catalog Lakeguard, which allows you to run Apache Spark™ workloads in SQL, Python, and Scala with full data governance on the Databric...

  • 400 Views
  • 1 replies
  • 1 kudos
Thursday
Announcing the General Availability of Databricks Asset Bundles

Get started with Databricks Asset Bundles (DABs) today! Simplify project management, versioning, testing, and deployment for your data and AI projects on the Databricks Platform. Embrace software engineering best practices with DABs—source control, c...

  • 1007 Views
  • 1 replies
  • 2 kudos
Wednesday
Register now and save 50% on training at Data + AI Summit!

For a limited time, we're offering 50% off training and certification at Data + AI Summit with the following code: TRAIN50FOTY. This offer expires on May 3, 2024. Register today and add training. Read more here!

  • 366 Views
  • 0 replies
  • 0 kudos
Tuesday
How to successfully build GenAI applications

The Big Book of Generative AI brings together best practices and know-how for building production-quality GenAI applications. You’ll find technical content and code samples that will help you do everything from deploying your first application to bui...

  • 828 Views
  • 1 replies
  • 2 kudos
Monday

Community Activity

koushiknpvs
by New Contributor II
  • 80 Views
  • 2 replies
  • 0 kudos

Databricks Certification Exam Suspended. Kindly help to reschedule

Hi, My exam got suspended in the middle of answering. My name is Koushik Nandiraju. My Email id is koushiknandiraju@gmail.com  Can you guys help me out please? Here is the request id - #00466317

  • 80 Views
  • 2 replies
  • 0 kudos
Latest Reply
Cert-TeamOPS
New Contributor II
  • 0 kudos

  Hi @koushiknpvs Thank you for raising a ticket with Databricks Training & Certification regarding your issue, We are working on your request. Please allow us 24 hours to revert back to you. 

  • 0 kudos
1 More Replies
Sujitha
by Community Manager
  • 1007 Views
  • 1 replies
  • 2 kudos

Announcing the General Availability of Databricks Asset Bundles

Get started with Databricks Asset Bundles (DABs) today! Simplify project management, versioning, testing, and deployment for your data and AI projects on the Databricks Platform. Embrace software engineering best practices with DABs—source control, c...

Screenshot 2024-04-25 at 8.09.57 AM.png
  • 1007 Views
  • 1 replies
  • 2 kudos
Latest Reply
Ajay-Pandey
Esteemed Contributor III
  • 2 kudos

This was most awaited features.I am already using Bundle for workflow CICD.

  • 2 kudos
Pri333
by New Contributor II
  • 3351 Views
  • 11 replies
  • 1 kudos

Databricks exam got suspended. Need urgent help

Hello Team, I faced multiple interruptions and challenges while attempting my certification, The exam got paused 4 times where twice I had to show my desk and my entire room. Eventually my exam got suspended. I was not involved in any unfair means an...

  • 3351 Views
  • 11 replies
  • 1 kudos
Latest Reply
munish-pratap
New Contributor II
  • 1 kudos

Hi @Cert-Team Would require you help on below support ticket.Request ID : #00467143I faced some interruptions and challenges while attempting my certification, The exam got paused 2 times and after that I had to show my desk and my entire room. Event...

  • 1 kudos
10 More Replies
corp
by Visitor
  • 41 Views
  • 1 replies
  • 0 kudos

inter connected notebook

How to use inter connected notebook, available in databricks?

  • 41 Views
  • 1 replies
  • 0 kudos
Latest Reply
mhiltner
Vistor
  • 0 kudos

Do you mean running one notebook from another and using variables and functions defined in the other one? If that's what you're seeking, try using the magic command %run + notebook path.  You can find some documentation about it here: https://docs.da...

  • 0 kudos
legobricks
by New Contributor
  • 92 Views
  • 2 replies
  • 0 kudos

Unable to mount GCS bucket with underscores in the name

I have two buckets with the same configurations and labels.One is named my-bucket and the other is my_bucket. I am able to mount my-bucket but get an opaque error message when trying to mount my_bucket. Is this known/expected behavior? Are underscore...

  • 92 Views
  • 2 replies
  • 0 kudos
Latest Reply
Walter_C
Valued Contributor II
  • 0 kudos

Hello it seems this is a known issue which is being worked by our engineering team. This is because when handling the mount point information we are using java.net.URI which does not accept _ in the hostname. 

  • 0 kudos
1 More Replies
Yohannes
by Visitor
  • 50 Views
  • 1 replies
  • 0 kudos

Databricks cli workflow

Is there a way that I can set up and configure a Databricks workflow job and tasks from Databricks cli or api tools by using python? Any help would be appreciated. #databricksworkflow #databricks 

  • 50 Views
  • 1 replies
  • 0 kudos
Latest Reply
steyler-db
New Contributor III
  • 0 kudos

Hello and yes, you can set up and configure a Databricks workflow job and tasks using Databricks CLI or API tools with Python. Here are some resources and steps to guide you:   Create and run Databricks Jobs: This document: ( https://docs.databrick...

  • 0 kudos
Anonymous
by Not applicable
  • 1779 Views
  • 4 replies
  • 4 kudos

Resolved! Play the BIG DATA GAME | By Firebolt

https://www.firebolt.io/big-data-gameThe most fun our Bricksters have had in a while at work is thanks to a little BIG DATA thing called The BIG DATA GAME ️This game is the cure for the mid-week blues. The Big Data Game is a simple yet awesome online...

Image Image
  • 1779 Views
  • 4 replies
  • 4 kudos
Latest Reply
FeliciaWilliam
New Contributor III
  • 4 kudos

You got me interested

  • 4 kudos
3 More Replies
de-hru
by New Contributor III
  • 812 Views
  • 2 replies
  • 1 kudos

Address Validation, Correction and Enrichment with Databricks Spark Engine

Hi all!In our project, we're thinking about "Validation, Correction and Enrichment of Postal Addresses" with Databricks. For sure we'd need some kind of batch processing, because we have millions of addresses in our system.I'm aware of Address Valida...

  • 812 Views
  • 2 replies
  • 1 kudos
Latest Reply
Sam99
Visitor
  • 1 kudos

Happy to help. Feel free to reach out https://www.linkedin.com/in/saleh-sultan-143ab036?utm_source=share&utm_campaign=share_via&utm_content=profile&utm_medium=android_app

  • 1 kudos
1 More Replies
Phani1
by Valued Contributor
  • 144 Views
  • 1 replies
  • 0 kudos

temporary tables or dataframes

Hi Team,We have to generate over 70 intermediate tables. Should we use temporary tables or dataframes, or should we create delta tables and truncate and reload? Having too many temporary tables could lead to memory problems. In this situation, what i...

Certifications
temp table
  • 144 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Valued Contributor II
  • 0 kudos

Using temporary tables or dataframes can be a good approach when the data is only needed for the duration of a single session. However, as you mentioned, having too many temporary tables could lead to memory problems. On the other hand, Delta tables ...

  • 0 kudos
Phani1
by Valued Contributor
  • 63 Views
  • 1 replies
  • 0 kudos

udf in databricks

Hi Team,Is there a particular reason why we should avoid using UDF and instead convert to DataFrame code?Are there any restrictions or limitations (in terms of performance or governance) when using UDFs in Databricks? Regards,Janga

  • 63 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Valued Contributor II
  • 0 kudos

Hello some of the things you need to take in consideration is that:UDFs might introduce significant processing bottlenecks into code execution. Databricks uses a number of different optimizers automatically for code written with included Apache Spark...

  • 0 kudos
ande
by New Contributor
  • 74 Views
  • 1 replies
  • 0 kudos

IP address for accessing external SFTP server

I am trying to pull in data to my Databricks workspace via an external SFTP server. I am using Azure for my compute. To access the SFTP server they need to whitelist my IP address. My IP address in Azure Databricks seems to be constantly changing fro...

  • 74 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Valued Contributor II
  • 0 kudos

Azure Databricks, like many cloud services, does not provide static IP addresses for outbound connections. This is because the compute resources are dynamically allocated and can change over time. One potential workaround could be to use a Virtual N...

  • 0 kudos
DC3
by New Contributor
  • 97 Views
  • 1 replies
  • 0 kudos

Unable to access unity catalog volume via /Volumes in notebook

I have set up a volume in unity catalog in the format catalog/schema/volume, and granted all permissions to all users on the catalog, schema and volume.From the notebook I can see the /Volumes directory in the root of the file system but am unable to...

  • 97 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Valued Contributor II
  • 0 kudos

  Ensure that you have the necessary privileges on the catalog, schema, and volume, to access a volume, you must have the USE CATALOG privilege on the Volume’s parent catalog and the USE SCHEMA privilege on its parent schema. If you're trying to cr...

  • 0 kudos
MOUNIKASIMHADRI
by New Contributor
  • 192 Views
  • 1 replies
  • 0 kudos

Insufficient Permissions Issue on Databricks

I have encountered a technical issue on Databricks.While executing commands both in Spark and SQL within the Databricks environment, I’ve run into permission-related errors from selecting files from DBFS. "org.apache.spark.SparkSecurityException: [IN...

  • 192 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Valued Contributor II
  • 0 kudos

Hello Mounika, many thanks for your question, are you using a shared access cluster? If yes, shared clusters requires you to grant Select permission on Any file to be able to access DBFS as mentioned on this doc https://docs.databricks.com/en/data-go...

  • 0 kudos
dbx_687_3__1b3Q
by New Contributor III
  • 72 Views
  • 1 replies
  • 0 kudos

Impersonating a user

How do I impersonate a user? I can't find any documentation that explains how to do this or even hint that it's possible.Use case: I perform administrative tasks like assign grants and roles to catalogs, schemas, and tables for the benefit of busines...

  • 72 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Valued Contributor II
  • 0 kudos

Hello, many thanks for your question. Right now impersonation is not possible in the Databricks environment, one possible solution might be that if you are an account admin you can remove your admin permissions from the account console on the specifi...

  • 0 kudos
Joaquim
by New Contributor II
  • 252 Views
  • 1 replies
  • 0 kudos

New admin question: How do you enable R on a existing cluster?

Hello Community. I have a user trying to use R and receive the error message illustrated on the attachment. I can't seem to find correct documentation on enabling R on an existing cluster. Would anyone be able to point me in the right direction? Than...

  • 252 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Valued Contributor II
  • 0 kudos

Hello Joaquim,Your issue might be related to the access mode of your cluster which probably has been selected to be Shared Access Mode.Shared cluster only allows Python, SQL and Scala languages, you might need to change the access mode to be Single U...

  • 0 kudos

Latest from our Blog

Mastering the Spark UI

Almost anyone who uses Spark or Databricks is aware of the Spark UI, and we all know that it’s a super powerful tool in the right hands. It can reveal what’s going wrong and any inefficiencies in you...

787Views 4kudos