cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

JasonN
by • New Contributor II
  • 3927 Views
  • 2 replies
  • 2 kudos

Resolved! DLT Cluster accessing to S3 bucket without Instance Profile attached

Hi Team,Can anyone please help me figure out how to configure Delta Live Tables Cluster accessing AWS S3 bucket without Instance profile defined in Cluster's JSON?The idea is, the user who is running the DLT Cluster has Storage Credentials and Extern...

  • 3927 Views
  • 2 replies
  • 2 kudos
Latest Reply
Vivian_Wilfred
Databricks Employee
  • 2 kudos

Hi @Jason Nam​ , DLT and unity catalog are not integrated yet. The cluster-notebook setup uses UC and can access S3 but not the DLT jobs. Please check the limitations in this document (7th point):https://docs.databricks.com/release-notes/unity-catalo...

  • 2 kudos
1 More Replies
SailajaB
by • Databricks Partner
  • 20224 Views
  • 10 replies
  • 10 kudos

Resolved! Is there a way to capture the notebook logs from ADF pipeline?

Hi,I would like to capture notebook custom log exceptions(python) from ADF pipeline based on the exceptions pipeline should got succeed or failed.Is there any mechanism to implement it. In my testing ADF pipeline is successful irrespective of the log...

  • 20224 Views
  • 10 replies
  • 10 kudos
Latest Reply
GurpreetSethi
Databricks Employee
  • 10 kudos

Hi SailajaB,Try this out.Notebook, once executed successfully return a long JSON formatted output. We need to specify appropriate nodes to fetch the output. In below screenshot we can see that when notebook ran it returns empName & empCity as output....

  • 10 kudos
9 More Replies
vjraitila
by • Databricks Partner
  • 4210 Views
  • 3 replies
  • 5 kudos

Strategy for streaming ETL and Delta Lake before Delta Live Tables existed

What was the established architectural pattern for doing streaming ETL with Delta Lake before DLT was a thing? And incidentally, what approach would you take in the context of delta-oss today? The pipeline definitions would not have had to be declara...

  • 4210 Views
  • 3 replies
  • 5 kudos
Latest Reply
Vidula
Databricks Partner
  • 5 kudos

Hi @Veli-Jussi Raitila​ Does @Shanmugavel Chandrakasu​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 5 kudos
2 More Replies
sternman
by • New Contributor II
  • 4078 Views
  • 2 replies
  • 2 kudos

Can I change the VMs IP address from dynamic to static on Azure without breaking anything?

Hello community;I am using Databricks on Azure and I am curious to know the implications of changing the IP address of the underlying VM's from dynamic to static - that is, the IP addresses that the clusters get, I would just like them to always have...

  • 4078 Views
  • 2 replies
  • 2 kudos
Latest Reply
Vidula
Databricks Partner
  • 2 kudos

Hi @Jonathan Stern​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 2 kudos
1 More Replies
shesa
by • New Contributor
  • 2222 Views
  • 2 replies
  • 0 kudos

Unable to login to https://community.cloud.databricks.com/login.html

Unable to login to https://community.cloud.databricks.com/login.htmlYesterday only I registered, able to login and create cluster.Today unable to login with same credential. I do not receive any email also while trying Password reset.

  • 2222 Views
  • 2 replies
  • 0 kudos
Latest Reply
Vidula
Databricks Partner
  • 0 kudos

Hi @SHESADEV SAMAL​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 0 kudos
1 More Replies
Aaron1234567
by • Databricks Partner
  • 11502 Views
  • 9 replies
  • 3 kudos

Delta Live Table Pipeline - Azure cluster fail

Failed to launch pipeline cluster 0802-171503-4m02lexd: The operation could not be performed on your account with the following error message: azure_error_code: OperationNotAllowed, azure_error_message: Operation could not be completed as it results ...

  • 11502 Views
  • 9 replies
  • 3 kudos
Latest Reply
Vidula
Databricks Partner
  • 3 kudos

Hi there @Aaron LeBato​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you...

  • 3 kudos
8 More Replies
Brendon_Daugher
by • New Contributor II
  • 2582 Views
  • 0 replies
  • 0 kudos

Understanding Dependency Update Failure

Heyooooo!I'm using Azure Databricks and sparklyr to do some geospatial analysis.Before I actually work with Spark Dataframes, I've been using the R packagesstarsandsfto do some preprocessing on my data so that it's easier to interact with later.In or...

  • 2582 Views
  • 0 replies
  • 0 kudos
Anonymous
by • Not applicable
  • 15126 Views
  • 0 replies
  • 1 kudos

Yay! It's September! ���� On September 22nd we are hosting another Community Social - we're doing these monthly ! We want to mak...

Yay! It's September! On September 22nd we are hosting another Community Social - we're doing these monthly ! We want to make sure that we all have the chance to connect as a community often. Come network, talk data, and just get social! Join us for o...

  • 15126 Views
  • 0 replies
  • 1 kudos
arthur_wang
by • Databricks Partner
  • 7141 Views
  • 2 replies
  • 1 kudos

How does Task Orchestration compare to Airflow (for Databricks-only jobs)?

One of my clients has been orchestration Databricks notebooks using Airflow + REST API. They're curious about the pros/cons of switching these jobs to Databricks jobs with Task Orchestration.I know there are all sorts of considerations - for example,...

  • 7141 Views
  • 2 replies
  • 1 kudos
Latest Reply
Shourya
Databricks Partner
  • 1 kudos

@Kaniz Fatma​ Hello Kaniz, I'm currently working with a major Enterprise Client looking to make the choice between the Airflow vs Databricks for Jobs scheduling. Our Entire code base is in Databricks and we are trying to figure out the complexities t...

  • 1 kudos
1 More Replies
datatello
by • New Contributor II
  • 4415 Views
  • 3 replies
  • 1 kudos

Exponentially slower joins using Pyspark

I'm new to Pyspark, but I've stumbled across an odd issue when I perform joins, where the action seems to take exponentially longer every time I add a new join to a function I'm writing.I'm trying to join a dataset of ~3 million records to one of ~17...

  • 4415 Views
  • 3 replies
  • 1 kudos
Latest Reply
Vidula
Databricks Partner
  • 1 kudos

Hi @Lee Bevers​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 1 kudos
2 More Replies
byrdman
by • New Contributor III
  • 4571 Views
  • 3 replies
  • 1 kudos

databricks multistage jobs in workflow Params not passing

I am using a multi-stage job calling different notebooks all have the same PARAMNAME that needs to be passed in. one the second and third job, I input the new a different PARAM's value .. but those values do not show up when it runs the task. I...

image image
  • 4571 Views
  • 3 replies
  • 1 kudos
Latest Reply
Prabakar
Databricks Employee
  • 1 kudos

Hi @David Byrd​ this is already a known thing and we have raised it to our engineering team. If you have the same key but different values in the parameters, then its most likely takes the first value for the key and will use the same for all the tas...

  • 1 kudos
2 More Replies
pmt
by • New Contributor III
  • 6667 Views
  • 7 replies
  • 1 kudos

Handling Changing Schema in CDC DLT

We are building a DLT pipeline and the autoloader is handling schema evolution fine. However, further down the pipeline we are trying to load that streamed data with the apply_changes() function into a new table and, from the looks of it, doesn't see...

  • 6667 Views
  • 7 replies
  • 1 kudos
Latest Reply
Vidula
Databricks Partner
  • 1 kudos

Hey there @Palani Thangaraj​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear fro...

  • 1 kudos
6 More Replies
liamod
by • Databricks Partner
  • 7833 Views
  • 2 replies
  • 8 kudos

Resolved! AWS and Azure on the same Databricks workspace.

Can I use compute instances from different providers i.e. AWS and Azure on the same Databricks workspace?

  • 7833 Views
  • 2 replies
  • 8 kudos
Latest Reply
Prabakar
Databricks Employee
  • 8 kudos

Hi @Liam ODonoghue​ there are a few methods using which you can connect AWS and Azure resources. In such cases, it involves only your accounts. But with Databricks, you need to handle two accounts for both cloud providers. Let's say if you create a w...

  • 8 kudos
1 More Replies
Rayan
by • New Contributor II
  • 2108 Views
  • 2 replies
  • 2 kudos

Terraform managing workspaces

Hi , Do we have any references? I am actually looking for managing workspaces and other databricks related stuff with terraform, trying to create in the form of modules so that everytime i need to create a new workspace i just need to call the worksp...

  • 2108 Views
  • 2 replies
  • 2 kudos
Latest Reply
Vidula
Databricks Partner
  • 2 kudos

Hi there @Rayan D​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...

  • 2 kudos
1 More Replies
Labels