cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
tirato
by • New Contributor II
  • 2881 Views
  • 3 replies
  • 2 kudos

Resolved! Cannot import-dir from AzureDevops, but works fine locally.

Hello,as i'm trying to create a CI/CD for the project, I'm finding myself stuck.Tried to upload the Notebooks from my Azure DevOps Release and I'm getting 403-forbidden access.I used 'cat ~/.databrickscfg file and matched with the local config that I...

Data Engineering
Reply
  • 2881 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
valeryuaba
New Contributor III
  • 2 kudos

Hey everyone! I can totally relate to the frustration of encountering authentication issues when setting up a CI/CD pipeline. It's great that you're able to import the notebooks locally, but facing difficulties on Azure DevOps can be quite puzzling.F...

  • 2 kudos
2 More Replies
brickster_2018
by • Databricks Employee
  • 3163 Views
  • 2 replies
  • 1 kudos

Resolved! Cluster Health Dashboard

Is there a cluster health dashboard which has the details of the total number of running interactive cluster, the total number of job clusters? Also Flag clusters with issues. 

Data Engineering
Reply
  • 3163 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
valeryuaba
New Contributor III
  • 1 kudos

Thanks!

  • 1 kudos
1 More Replies
thushar
by • Contributor
  • 2585 Views
  • 2 replies
  • 0 kudos

Connect to Azure DevOps repository using service principle

My source code is in the VSTS repository and I am using PAT token to connect VSTS from Azure data bricks notebook and then building packages and installing my cluster. For the production environment, I can't use PAT token, so is there any way to conn...

Data Engineering
Reply
  • 2585 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
martinez
New Contributor III
  • 0 kudos

Hey everyoneI've been working with Azure DevOps and VSTS repositories, and I can relate to the challenges of connecting them securely. thushar, I understand your concern about using a PAT token for production environments. Fortunately, there is indee...

  • 0 kudos
1 More Replies
boyelana
by • Contributor III
  • 2572 Views
  • 3 replies
  • 7 kudos

Resolved! How to start with Databricks in Google Cloud?

I am looking through Google Cloud Platform and I am looking to get started with Databricks on GCP. Happy if anyone can point me in the direction that can provide guidance on how to get started.Thansk

Data Engineering
Reply
  • 2572 Views
  • 3 replies
  • 7 kudos
View Replies
Latest Reply
martinez
New Contributor III
  • 7 kudos

Hey boyelana Databricks on Google Cloud Platform is definitely an interesting and powerful combination, and I'm thrilled to see that you're looking to get started with it, boyelana!To begin your journey with Databricks on GCP, there are a few steps y...

  • 7 kudos
2 More Replies
erigaud
by • Honored Contributor
  • 2668 Views
  • 2 replies
  • 2 kudos

Resolved! Access to personal access token via python

Is there a way to get an existing personal access token via python ? Either through and sdk or a rest endpoint ? Or is the only way to do that to store the PAT in a key vault and retrieve it via a secret scope ? Thank you !

Data Engineering
Reply
  • 2668 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Ajay-Pandey  Hope everything is going great. Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so ...

  • 2 kudos
1 More Replies
Sujitha
by • Databricks Employee
  • 2016 Views
  • 3 replies
  • 2 kudos

KB Feedback Discussion In addition to the Databricks Community, we have a Support team that maintains a Knowledge Base (KB). The KB contains answers t...

KB Feedback DiscussionIn addition to the Databricks Community, we have a Support team that maintains a Knowledge Base (KB). The KB contains answers to common questions about Databricks, as well as information on optimisation and troubleshooting.These...

Data Engineering
Reply
  • 2016 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
martinez
New Contributor III
  • 2 kudos

Thanks for sharing!  

  • 2 kudos
2 More Replies
VitorGhiotti
by • New Contributor II
  • 1233 Views
  • 2 replies
  • 1 kudos

Python error on install epmwebapi library

The error below occurred when trying to install the mentioned library. how do i fix this

error.png
Data Engineering
Reply
  • 1233 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Hemant
Valued Contributor II
  • 1 kudos

Hi @VitorGhiotti, I am able to install this package, can you share your cluster configuration and are you using a private endpoint?

  • 1 kudos
1 More Replies
superanna
by • New Contributor II
  • 902 Views
  • 1 replies
  • 1 kudos

Yes, still illegal. And I also don’t understand why it is equated with drugs, but alcohol is not! Not a single murder has yet been committed under can...

Yes, still illegal. And I also don’t understand why it is equated with drugs, but alcohol is not! Not a single murder has yet been committed under cannabis, not a single war has been unleashed. It's just that people who don't use don't understand how...

Data Engineering
Reply
  • 902 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Mz_Yvette
New Contributor II
  • 1 kudos

You are absolutely right! I have found it to be a big relief medically. I have nerve conditions which is not operable. The legal medical pills almost literally killed me, and if it wasn't for my husband's quick thinking, I wouldn't be here to share t...

  • 1 kudos
Priyag1
by • Honored Contributor II
  • 3339 Views
  • 4 replies
  • 4 kudos

Data preparation in Databricks

Data preparation in Databricks Good data is important to ensure accurate and useful results. To get good data following tasks must be done Cleaning and formatting data - Handling missing values or outliers, ensuring data is in the correct format, and...

Data Engineering
Reply
  • 3339 Views
  • 4 replies
  • 4 kudos
View Replies
Latest Reply
dplante
Contributor II
  • 4 kudos

Data governance and data lineage are other things to call out.Here's a cheat sheet  that is also useful -> Data Preparation Cheatsheet

  • 4 kudos
3 More Replies
gpierard
by • New Contributor III
  • 741 Views
  • 1 replies
  • 0 kudos

Badge not received for Databricks Certified Data Engineer Associate

Hello,I passed the certification but haven't received a badge. In fact, I created my databricks academy account only after completing the test. Could you please ensure I do receive that certification? Thanks 

Data Engineering
Reply
  • 741 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @gpierard  Thank you for reaching out!  Please submit a ticket to our Training Team here: https://help.databricks.com/s/contact-us?ReqType=training  and our team will get back to you shortly. 

  • 0 kudos
matty_f
by • New Contributor II
  • 7126 Views
  • 1 replies
  • 0 kudos

Migration scripts for distribution, embedded in library

I'm working on a python package that can be installed via pip. The package will manage a delta table for the user, and new versions of the package may need to run migrations on this tableIs this an okay format to use?def migrate(table_path): mm_p...

Data Engineering
Reply
  • 7126 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
matty_f
New Contributor II
  • 0 kudos

Not much community happening here 

  • 0 kudos
Dekova
by • New Contributor II
  • 833 Views
  • 0 replies
  • 0 kudos

Structured Streaming & Workspace Job Limits

In "Advanced Data Engineering with Databricks", the section on Bronze Ingestion Patterns mentions that workspaces have limits of 5000 jobs triggered in an hour. As a solution, it suggest multiplex streaming to a single bronze table and then using sub...

Screenshot 2023-07-14 at 9.49.43 PM.png
Data Engineering
structured streaming
Reply
  • 833 Views
  • 0 replies
  • 0 kudos
brickster_2018
by • Databricks Employee
  • 4096 Views
  • 2 replies
  • 3 kudos

Resolved! Can I install notebook scoped JAR/Maven libraries?

The notebook scoped libraries are very handy. Is it possible to leverage the same for maven jars or application jars as well?

Data Engineering
Reply
  • 4096 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Pratik_Ghosh
New Contributor II
  • 3 kudos

Any further update on this topic?

  • 3 kudos
1 More Replies
Ruby8376
by • Valued Contributor
  • 745 Views
  • 0 replies
  • 0 kudos

Schema definition help in scala notebook in databricks !!!!!!!1

I am building schema for an incoming avro file(json message) and creating a final dataframe for it. The schema built looks fine as per the json sample message provided but I am getting null values in all the fields. Can somebody look at this code and...

Data Engineering
Reply
  • 745 Views
  • 0 replies
  • 0 kudos
erigaud
by • Honored Contributor
  • 15934 Views
  • 5 replies
  • 4 kudos

Resolved! Gracefully stop a job based on condition

Hello, I have a job with many tasks running on a schedule, and the first tasks checks a condition. Based on the condition, I would either want to continue the job as normal, or to stop right away and don't run all the other tasks. Is there a way to d...

Data Engineering
Reply
  • 15934 Views
  • 5 replies
  • 4 kudos
View Replies
Latest Reply
erigaud
Honored Contributor
  • 4 kudos

I think the best way to accomplish this would be to either propagate the check, as mentionned by @menotron, or have the initial task in another job, and only run the second job if the condition is met. Obviously it depends on the use case. Thank you ...

  • 4 kudos
4 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels