cancel
Showing results for 
Search instead for 
Did you mean: 
Community Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Daniel-K
by New Contributor
  • 583 Views
  • 1 replies
  • 0 kudos

DATA+AI Summit Day 4

The Expo Booths are a treasure hunt, met many interesting companies doing wonderful things in Generative AI #Data+AISummit

  • 583 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Daniel-K, We're thrilled to hear that you had a great experience at DAIS 2023! Your feedback is valuable to us, and we appreciate you taking the time to share it on the community platform. We wanted to let you know that the Databricks Community T...

  • 0 kudos
ManiMar
by New Contributor II
  • 649 Views
  • 1 replies
  • 2 kudos

MLOps on Azure: API vs SDK vs Databricks CLI?

Hello fellow community members,In our organization, we have developed, deployed and utilized an API-based MLOps pipeline using Azure DevOps.The CI/CD pipeline has been developed and refined for about 18 months or so, and I have to say that it is pret...

  • 649 Views
  • 1 replies
  • 2 kudos
Latest Reply
raphaelblg
Honored Contributor
  • 2 kudos

Hello @ManiMar,  In my opinion it's up to you to choose, and you're in the right path by comparing the pros/cons of each approach. I'd like to highlight that one of the advantages of the Databricks CLI is being able to use Databricks Asset Bundles. I...

  • 2 kudos
Aslam_shaikk99
by New Contributor
  • 443 Views
  • 1 replies
  • 0 kudos

Got suspended which attempting Databricks certified Associate Developer for Apache Spark 3.0 Python

Hi Team, My Databricks Certified exam got suspended.I was continuously in front of the camera and an alert appeared and then my exam resumed. Then later a support person asked me to show the entire table and entire room, I have showed around the room...

  • 443 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Aslam_shaikk99, I'm sorry to hear your exam was suspended. Thank you for filing a ticket with our support team. Please allow the support team 24-48 hours to resolve. In the meantime, you can review the following documentation: Room requirements B...

  • 0 kudos
divakar_siva
by New Contributor
  • 506 Views
  • 1 replies
  • 0 kudos

Learnings from DIAS

Flying in from Singapore to attend the DIAS 2023 to evaluate the Databricks  platforms and workflows for our cloud migration. The Advanced Data engineering course provides alot of insights and details for many niche use-cases. This experience will he...

  • 506 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Rafa82, We're thrilled to hear that you had a great experience at DAIS 2023! Your feedback is valuable to us, and we appreciate you taking the time to share it on the community platform.   We wanted to let you know that the Databricks Community T...

  • 0 kudos
restlessmodem
by New Contributor
  • 555 Views
  • 1 replies
  • 0 kudos

Data + AI Summit Experience

Hey everyone,currently attending Data + AI Summit and definitely seeing value of implementing Streaming and Unity Catalog in our current pipelines. Thanks to Tom for a great Training Session on Advanced Data Engineering! restlessmodem

  • 555 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @restlessmodem, We're thrilled to hear that you had a great experience at DAIS 2023! Your feedback is valuable to us, and we appreciate you taking the time to share it on the community platform.   We wanted to let you know that the Databricks Comm...

  • 0 kudos
SabrinaG
by New Contributor
  • 513 Views
  • 1 replies
  • 0 kudos

Great experience at Summit 23!

I came across Databricks last year when I was working on a client project. Client had data in large volume (over 100m rows) and after transforming the data it eventually went up to 50 billion records. Databricks helped my team to drastically improved...

  • 513 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Rafa82, We're thrilled to hear that you had a great experience at DAIS 2023! Your feedback is valuable to us, and we appreciate you taking the time to share it on the community platform.   We wanted to let you know that the Databricks Community T...

  • 0 kudos
brian_zavareh
by New Contributor III
  • 946 Views
  • 1 replies
  • 0 kudos

Insights and Learnings from Databricks Summit: A Deep Dive into Data Engineering and Delta Live Tab

 1. Why am I here?I am present at the Databricks summit because I'm motivated to deepen my understanding of its groundbreaking technology. I'm passionate about learning how to leverage this advanced platform to better optimize and streamline the proc...

  • 946 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Rafa82, We're thrilled to hear that you had a great experience at DAIS 2023! Your feedback is valuable to us, and we appreciate you taking the time to share it on the community platform. We wanted to let you know that the Databricks Community Tea...

  • 0 kudos
AbbasNvs
by New Contributor
  • 517 Views
  • 1 replies
  • 0 kudos

Data Engineering Hub

I have been using databricks from last 2 years and have got involved with arranging when I was in my masters. Databricks did not made data engineering easy for me but helped me become a data engineer. I got into this 2 all day training from Advanced ...

  • 517 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @AbbasNvs, We're thrilled to hear that you had a great experience at DAIS 2023! Your feedback is valuable to us, and we appreciate you taking the time to share it on the community platform. We wanted to let you know that the Databricks Community T...

  • 0 kudos
ariharansiva00
by New Contributor III
  • 980 Views
  • 5 replies
  • 0 kudos

DataBricks Certification Exam Got Suspended. Require support for the same.

Hello Team, I encountered Pathetic experience while attempting my 1st DataBricks certification. Abruptly, Proctor asked me to show my desk, after showing he/she asked multiple times.. wasted my time and then suspended my exam without giving any reaso...

  • 980 Views
  • 5 replies
  • 0 kudos
Latest Reply
ariharansiva00
New Contributor III
  • 0 kudos

@Kaniz @Cert-Team @Sujitha I have sent multiple emails to the Support team to reschedule my exam with Date, but I have not received any confirmation from them.Please look into this issue and reschedule the exam as soon as possible. This certification...

  • 0 kudos
4 More Replies
manupmanoos
by New Contributor III
  • 1561 Views
  • 4 replies
  • 1 kudos

How to save the best model checkpoi through the epochs of a deep learning network through callbacks?

I have create a neural network and I am training the model with the code as below.  The code fails to write to the databricks file storage. is there any other way to write the checkpoint to databricks storage or to an s3 bucket directly?custom_early_...

  • 1561 Views
  • 4 replies
  • 1 kudos
Latest Reply
friazaati67
New Contributor III
  • 1 kudos

I'm having trouble saving to local storage in Databricks DBFS due to an "invalid operation" message. Although I can save models to S3 with my AWS credentials, I need to save the best model checkpoint during training locally. For more details please c...

  • 1 kudos
3 More Replies
pshuk
by New Contributor III
  • 537 Views
  • 1 replies
  • 1 kudos

capture return value from databricks job to local machine by CLI

Hi,I want to run a python code on databricks notebook and return the value to my local machine. Here is the summary:I upload files to volumes on databricks. I generate a md5 for local file. Once the upload is finished, I create a python script with t...

  • 537 Views
  • 1 replies
  • 1 kudos
Latest Reply
raphaelblg
Honored Contributor
  • 1 kudos

Hello @pshuk, You could check the below CLI commands: get-run-output Get the output for a single run. This is the REST API reference, which relates to the CLI command: https://docs.databricks.com/api/workspace/jobs/getrunoutput export-run  There's al...

  • 1 kudos
Frantz
by New Contributor III
  • 1438 Views
  • 2 replies
  • 0 kudos

Resolved! Error Code: METASTORE_DOES_NOT_EXIST when using Databricks API

Hello, I'm attempting to use the databricks API to list the catalogs in the metastore. When I send the GET request to `/api/2.1/unity-catalog/catalogs` , I get this error I have checked multiple times and yes, we do have a metastore associated with t...

Frantz_0-1716331980508.png
  • 1438 Views
  • 2 replies
  • 0 kudos
Latest Reply
Frantz
New Contributor III
  • 0 kudos

Turns out I was using the wrong databricks host url when querying from postman. I was using my Azure instance instead of my AWS instance.

  • 0 kudos
1 More Replies
subham0611
by New Contributor II
  • 664 Views
  • 1 replies
  • 0 kudos

Parallel kafka consumer in spark structured streaming

Hi,I have a spark streaming job which reads from kafka and process data and write to delta lake.Number of kafka partition: 100number of executor: 2 (4 core each)So we have 8 cores total which are reading from 100 partitions of a topic. I wanted to un...

  • 664 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @subham0611, In Spark Streaming, the number of threads is not explicitly controlled by the user. Instead, the parallelism is determined by the number of partitions in the Kafka topic. Each partition is consumed by a single Spark task. When you ...

  • 0 kudos
MohsenJ
by Contributor
  • 682 Views
  • 1 replies
  • 1 kudos

how to develop Notebooks on vscode for git repos?

I am able to use vscode extension + databricks connect to develop Notebooks on my local computer and run them on my databricks cluster. However I can not figure out how to develop the Notebooks that have the file `.py` extension but identified by Dat...

  • 682 Views
  • 1 replies
  • 1 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @MohsenJ, You can indeed develop and run Databricks notebooks with .py extension on your local computer using the Databricks extension for Visual Studio Code and Databricks Connect1. Here are the steps you need to follow: Enable Databricks Con...

  • 1 kudos
unity_Catalog
by New Contributor
  • 416 Views
  • 1 replies
  • 0 kudos

Assessment(Assessment job need to be deployed using Terraform)

Assessment(Assessment job need to be deployed using Terraform)1.Install latest version of UCX 2.UCX will add the assessment job and queries to the workspace3.Run the assessment using ClusterHow to write code for this by using Terraform. Can anyone he...

  • 416 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @unity_Catalog, To deploy an assessment job using Terraform, you’ll need to write a Terraform configuration that includes the following steps: Install the latest version of UCX: This will likely involve using a provider to manage software inst...

  • 0 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!