cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

mshettar
by • New Contributor II
  • 5118 Views
  • 2 replies
  • 0 kudos

Databricks CLI's workspace export_dir command adds unnecessary edits despite not making any change in the workspace

databricks workspace export_dir / export command with overwrite option enabled adds non-existent changes in the target directory. 1. It introduces new line deletion and 2. add/deletion of MAGIC comments despite not making any meaningful changes in th...

Screenshot 2023-06-06 at 2.44.48 PM
  • 5118 Views
  • 2 replies
  • 0 kudos
Latest Reply
RyanHager
Contributor
  • 0 kudos

I am encountering this issue as well and it did not happen previously.  Additionally, you see this pattern if you are using repos internally and make a change to a notebook in another section.

  • 0 kudos
1 More Replies
Chakra
by • New Contributor II
  • 2941 Views
  • 1 replies
  • 1 kudos

Create job cluster with a docker image in azure data factory

Is there a way to create a job cluster in azure data factory with a docker image either through API or UI

  • 2941 Views
  • 1 replies
  • 1 kudos
Latest Reply
m_szklarczyk
New Contributor II
  • 1 kudos

Does anyone handle how to run from ADF custom image as jobs compute?

  • 1 kudos
Shay83
by • New Contributor II
  • 3238 Views
  • 1 replies
  • 3 kudos

Resolved! stream from specific time

Hello,How should I start stream a delta table from specific point in time?

  • 3238 Views
  • 1 replies
  • 3 kudos
Latest Reply
Lakshay
Databricks Employee
  • 3 kudos

If you are streaming from a Delta table, you can specify the starting version or timestamp. You can refer the document for complete detail: https://docs.databricks.com/structured-streaming/delta-lake.html#specify-initial-position

  • 3 kudos
User16826992666
by • Databricks Employee
  • 5432 Views
  • 5 replies
  • 1 kudos

When developing a Delta Live Table, can I test my code in the notebook?

Just wondering about the dev flow when building Delta Live Tables. I write my code in the notebook so it would be useful to be able to test it out from within that environment.

  • 5432 Views
  • 5 replies
  • 1 kudos
Latest Reply
Lakshay
Databricks Employee
  • 1 kudos

You need to create a DLT pipeline to test the code. 

  • 1 kudos
4 More Replies
TonyLe
by • New Contributor
  • 3529 Views
  • 2 replies
  • 1 kudos

IllegalArgumentException: requirement failed: Invalid uri

Hi all, I'm trying to connect to MongoDB using the Databricks notebook. I keep getting the error that my MongoDB uri is invalid. The uri works when connecting from my local machine using the Rust driver. I pretty much followed the tutorial that was g...

  • 3529 Views
  • 2 replies
  • 1 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 1 kudos

Also take in mind firewall issues in case the mongodb is on-prem or on some other location.

  • 1 kudos
1 More Replies
tirato
by • New Contributor II
  • 6963 Views
  • 3 replies
  • 2 kudos

Resolved! Cannot import-dir from AzureDevops, but works fine locally.

Hello,as i'm trying to create a CI/CD for the project, I'm finding myself stuck.Tried to upload the Notebooks from my Azure DevOps Release and I'm getting 403-forbidden access.I used 'cat ~/.databrickscfg file and matched with the local config that I...

  • 6963 Views
  • 3 replies
  • 2 kudos
Latest Reply
valeryuaba
New Contributor III
  • 2 kudos

Hey everyone! I can totally relate to the frustration of encountering authentication issues when setting up a CI/CD pipeline. It's great that you're able to import the notebooks locally, but facing difficulties on Azure DevOps can be quite puzzling.F...

  • 2 kudos
2 More Replies
brickster_2018
by • Databricks Employee
  • 5057 Views
  • 2 replies
  • 1 kudos

Resolved! Cluster Health Dashboard

Is there a cluster health dashboard which has the details of the total number of running interactive cluster, the total number of job clusters? Also Flag clusters with issues. 

  • 5057 Views
  • 2 replies
  • 1 kudos
Latest Reply
valeryuaba
New Contributor III
  • 1 kudos

Thanks!

  • 1 kudos
1 More Replies
thushar
by • Databricks Partner
  • 4439 Views
  • 2 replies
  • 0 kudos

Connect to Azure DevOps repository using service principle

My source code is in the VSTS repository and I am using PAT token to connect VSTS from Azure data bricks notebook and then building packages and installing my cluster. For the production environment, I can't use PAT token, so is there any way to conn...

  • 4439 Views
  • 2 replies
  • 0 kudos
Latest Reply
martinez
New Contributor III
  • 0 kudos

Hey everyoneI've been working with Azure DevOps and VSTS repositories, and I can relate to the challenges of connecting them securely. thushar, I understand your concern about using a PAT token for production environments. Fortunately, there is indee...

  • 0 kudos
1 More Replies
boyelana
by • Contributor III
  • 5650 Views
  • 3 replies
  • 7 kudos

Resolved! How to start with Databricks in Google Cloud?

I am looking through Google Cloud Platform and I am looking to get started with Databricks on GCP. Happy if anyone can point me in the direction that can provide guidance on how to get started.Thansk

  • 5650 Views
  • 3 replies
  • 7 kudos
Latest Reply
martinez
New Contributor III
  • 7 kudos

Hey boyelana Databricks on Google Cloud Platform is definitely an interesting and powerful combination, and I'm thrilled to see that you're looking to get started with it, boyelana!To begin your journey with Databricks on GCP, there are a few steps y...

  • 7 kudos
2 More Replies
erigaud
by • Honored Contributor
  • 4816 Views
  • 2 replies
  • 2 kudos

Resolved! Access to personal access token via python

Is there a way to get an existing personal access token via python ? Either through and sdk or a rest endpoint ? Or is the only way to do that to store the PAT in a key vault and retrieve it via a secret scope ? Thank you !

  • 4816 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Ajay-Pandey  Hope everything is going great. Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so ...

  • 2 kudos
1 More Replies
Sujitha
by • Databricks Employee
  • 3654 Views
  • 3 replies
  • 2 kudos

KB Feedback Discussion In addition to the Databricks Community, we have a Support team that maintains a Knowledge Base (KB). The KB contains answers t...

KB Feedback DiscussionIn addition to the Databricks Community, we have a Support team that maintains a Knowledge Base (KB). The KB contains answers to common questions about Databricks, as well as information on optimisation and troubleshooting.These...

  • 3654 Views
  • 3 replies
  • 2 kudos
Latest Reply
martinez
New Contributor III
  • 2 kudos

Thanks for sharing!  

  • 2 kudos
2 More Replies
VitorGhiotti
by • New Contributor II
  • 2373 Views
  • 2 replies
  • 1 kudos

Python error on install epmwebapi library

The error below occurred when trying to install the mentioned library. how do i fix this

error.png
  • 2373 Views
  • 2 replies
  • 1 kudos
Latest Reply
Hemant
Valued Contributor II
  • 1 kudos

Hi @VitorGhiotti, I am able to install this package, can you share your cluster configuration and are you using a private endpoint?

  • 1 kudos
1 More Replies
superanna
by • New Contributor II
  • 1955 Views
  • 1 replies
  • 1 kudos

Yes, still illegal. And I also don’t understand why it is equated with drugs, but alcohol is not! Not a single murder has yet been committed under can...

Yes, still illegal. And I also don’t understand why it is equated with drugs, but alcohol is not! Not a single murder has yet been committed under cannabis, not a single war has been unleashed. It's just that people who don't use don't understand how...

  • 1955 Views
  • 1 replies
  • 1 kudos
Latest Reply
Mz_Yvette
New Contributor II
  • 1 kudos

You are absolutely right! I have found it to be a big relief medically. I have nerve conditions which is not operable. The legal medical pills almost literally killed me, and if it wasn't for my husband's quick thinking, I wouldn't be here to share t...

  • 1 kudos
Priyag1
by • Honored Contributor II
  • 6815 Views
  • 4 replies
  • 4 kudos

Data preparation in Databricks

Data preparation in Databricks Good data is important to ensure accurate and useful results. To get good data following tasks must be done Cleaning and formatting data - Handling missing values or outliers, ensuring data is in the correct format, and...

  • 6815 Views
  • 4 replies
  • 4 kudos
Latest Reply
dplante
Contributor II
  • 4 kudos

Data governance and data lineage are other things to call out.Here's a cheat sheet  that is also useful -> Data Preparation Cheatsheet

  • 4 kudos
3 More Replies
Labels