cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

databicky
by Contributor II
  • 4005 Views
  • 7 replies
  • 8 kudos

Resolved! How can we move the excel file from adls to share point

I have one excel file in the adls, i want to move that file into sharepoint, but i tried this method in data factory, but in sink sharepoint is not available, is there any possible way to do this?​

  • 4005 Views
  • 7 replies
  • 8 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 8 kudos

Hi @Mohammed sadamusean​(Customer)​ , We haven’t heard from you on the last response from @Daniel Sahal​ ​ and @KVNARK .​, and I was checking back to see if their suggestions helped you. Or else, If you have any solution, please do share that with th...

  • 8 kudos
6 More Replies
SeliLi_52097
by New Contributor III
  • 2840 Views
  • 5 replies
  • 5 kudos

Databricks Academy webpage showing insecure connection (in Chrome)

When I was trying to visit the Databricks Academy website https://customer-academy.databricks.com, it showed insecure connection as below.This happened at 8 January 2023 (AEDT) around 12:30pm.

Screen Shot 2023-01-08 at 12.15.54 pm
  • 2840 Views
  • 5 replies
  • 5 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 5 kudos

Hi @Selina Li​, Thank you for reaching out!Let us look into this for you, and we'll circle back with an update.

  • 5 kudos
4 More Replies
prashantp
by New Contributor III
  • 2109 Views
  • 2 replies
  • 1 kudos

Resolved! CloudFormation failure while creating workspace

The cloud formation script is fails to create workspace due to following error. The solution to this error is not available in the standard troubleshooting guide. Following is the error message from cloud watch logs. It seems like a bug in the api, b...

  • 2109 Views
  • 2 replies
  • 1 kudos
Latest Reply
prashantp
New Contributor III
  • 1 kudos

I found a solution to this by creating a new role following these articles -https://docs.databricks.com/administration-guide/account-api/iam-role.htmlhttps://docs.databricks.com/administration-guide/account-api/aws-storage.html

  • 1 kudos
1 More Replies
Tacuma
by New Contributor II
  • 1539 Views
  • 4 replies
  • 1 kudos

Scheduling jobs with Airflow result in each task running multiple jobs.

Hey everyone, I'm experiementing with running containerized pyspark jobs in Databricks, and orchestrating them with airflow. I am however, encountering an issue here. When I trigger an airflow DAG, and I look at the logs, I see that airflow is spinni...

  • 1539 Views
  • 4 replies
  • 1 kudos
Latest Reply
Tacuma
New Contributor II
  • 1 kudos

Both, I guess? Yes, all jobs share the same config - the question I have is why in the same airflow task log, there are 3 jobs runs. I'm hoping that there's something in the configs and may give me some kind of clue.

  • 1 kudos
3 More Replies
databicky
by Contributor II
  • 2102 Views
  • 3 replies
  • 0 kudos

how to check the particular column value in spark dataframe ?

if i want​ to check the the particular column in dataframe is need to contain zero, if its not have zero means , it need to get fail

  • 2102 Views
  • 3 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Mohammed sadamusean​  (Customer)​, We haven’t heard from you since the last response from @Mateusz Łomański​ (Customer)​ , and I was checking back to see if his suggestions helped you.Or else, If you have any solution, please share it with the co...

  • 0 kudos
2 More Replies
Jennifer
by New Contributor III
  • 2178 Views
  • 1 replies
  • 0 kudos

How do I update an aggregate table using a Delta live table

I have am using delta live tables to stream events and I have a raw table for all the events and a downstream aggregate table. I need to add the new aggregated number to the downstream table aggregate column. But I didn't find any recipe talking abou...

  • 2178 Views
  • 1 replies
  • 0 kudos
Latest Reply
Jennifer
New Contributor III
  • 0 kudos

Maybe my code is correct already since I use dlt.read("my_raw_table") instead of delta.read_stream("my_raw_table"). So the col_aggr is recalculated completely every time my_raw_table is updated.

  • 0 kudos
pasiasty2077
by New Contributor
  • 7268 Views
  • 2 replies
  • 1 kudos

Partition filter is skipped when table is used in where condition, why?

Hi,maybe someone can help me i do want to run very narrow query SELECT * FROM my_table WHERE snapshot_date IN('2023-01-06', '2023-01-07')   -- part of the physical plan: -- Location: PreparedDeltaFileIndex [dbfs:/...] -- PartitionFilters: [cast(snaps...

  • 7268 Views
  • 2 replies
  • 1 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @Mariusz J​, We haven’t heard from you since the last response from @Werner Stinckens​, and I was checking back to see if his suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to oth...

  • 1 kudos
1 More Replies
Valon98
by New Contributor III
  • 12055 Views
  • 11 replies
  • 5 kudos

Resolved! During execution of a cell "RuntimeException: The python kernel is unresponsive."

Hi all, I am running a preprocessing to create my trainset and test set. Does anyone know why during the execution my cell gives the error "RuntimeException: The python kernel is unresponsive." ? How can I solve it?

  • 12055 Views
  • 11 replies
  • 5 kudos
Latest Reply
Anonymous
Not applicable
  • 5 kudos

Hey there @Valerio Goretti​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from...

  • 5 kudos
10 More Replies
BigMF
by New Contributor III
  • 1803 Views
  • 2 replies
  • 2 kudos

How do I maintain Azure Account Console access after Global Admin access is removed?

Hello, I don't know if I should have created a separate question or added to this one.I've read the documentation and as far as I can tell, have followed it correctly but I'm still having issues accessing the Account Console unless the user I'm loggi...

image image image
  • 1803 Views
  • 2 replies
  • 2 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 2 kudos

Hi @Mark Ferguson​  We haven’t heard from you since the last response from @Debayan Mukherjee​, and I was checking back to see if his suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful t...

  • 2 kudos
1 More Replies
DanielBarbosa
by New Contributor III
  • 4747 Views
  • 3 replies
  • 3 kudos

Running jobs using notebooks in a remote Azure DevOps Services (Repos) Git repository is generating "Notebook not found" error.

By reading the documentation, we checked the possibility of running jobs in the Azure Databricks Workspace workflow using Azure DevOps Services repository source codes.The instructions in the documentation were followed and we configured the git info...

image image image image
  • 4747 Views
  • 3 replies
  • 3 kudos
Latest Reply
Ulf
New Contributor II
  • 3 kudos

I have the same challenge when integrating with Github repos. However I did not succeed including: '# Databricks notebook source' in the top of python files. Do you have any additional suggestions for solving this problem? @Vaibhav Sethi​ 

  • 3 kudos
2 More Replies
Prabha
by New Contributor II
  • 1836 Views
  • 4 replies
  • 0 kudos

DataBricks_dataengineer_result

Hi Team,I've successfully passed the Databricks Data Engineer Associate Certified exam on 30th October 2022. but still have not received the certificate.Please find the reference below.  I have registered the exam with the email id vprabhakaran1987@...

image
  • 1836 Views
  • 4 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Prabhakaran velusamy​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from y...

  • 0 kudos
3 More Replies
Sai1996
by New Contributor II
  • 930 Views
  • 2 replies
  • 1 kudos

Screenshot_20221031-222242_Chrome

Hi Databricks Team,​I​ have completed my databricks Certified data engineer Associate exam on Oct 30 but not received badge or certificate yet. Raised case alo but no response..could someone helpUserid: rupinisarguru128@gmail.com ​​​

  • 930 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Rupini Ravichandran​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from yo...

  • 1 kudos
1 More Replies
Own
by Contributor
  • 804 Views
  • 3 replies
  • 0 kudos
  • 804 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Shubham Sharma​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 0 kudos
2 More Replies
Renga87
by New Contributor II
  • 1709 Views
  • 4 replies
  • 2 kudos

Clarification on Voucher code

Is voucher code is mandatory to register exam "Databricks Associate developer for Apache spark 3.0" If so how to get it. Note : Created account in databricks portal recently and username is ksrenga87@gmail.com

  • 1709 Views
  • 4 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Rengaraja Sundararaj​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from y...

  • 2 kudos
3 More Replies
confused_dev
by New Contributor II
  • 23741 Views
  • 3 replies
  • 3 kudos

Python mocking dbutils in unittests

I am trying to write some unittests using pytest, but I am coming accross the problem of how to mock my dbutils method when dbutils isn't being defined in my notebook.Is there a way to do this so that I can unit test individual functions that are uti...

  • 23741 Views
  • 3 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Jake P​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 3 kudos
2 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels