cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

gazzyjuruj
by Contributor II
  • 9405 Views
  • 2 replies
  • 1 kudos

Error 503 first byte timeout

Hi,I'm receiving error while logging in or signing up on CE.Error 503 first byte timeoutfirst byte timeoutError 54113Details: cache-bom4725-BOM 1659706650 579764974Varnish cache serverscreenshot attached below:- Thanks, any help is appreciated from t...

Screenshot 2022-08-05 at 7.11.10 PM
  • 9405 Views
  • 2 replies
  • 1 kudos
Latest Reply
Vidula
Honored Contributor
  • 1 kudos

Hi @Ghazanfar Uruj​ Does @Prabakar Ammeappin​ answer help? If it does, would you be happy to mark it as best? If it doesn't, please tell us so we can help you.We'd love to hear from you.Thanks!

  • 1 kudos
1 More Replies
Wally
by New Contributor II
  • 1052 Views
  • 2 replies
  • 2 kudos

Databricks Sql schedule button not saving

The schedule button isn't saving my schedule information for a databricks sql query. After I hit save and open the schedule again it has reverted to 'Never'. The query itself according to the past executions pane is not running according to the sched...

  • 1052 Views
  • 2 replies
  • 2 kudos
Latest Reply
Vidula
Honored Contributor
  • 2 kudos

Hi @Wally Plourde​ Does @Rohit Rajendran​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 2 kudos
1 More Replies
Isaac_Low
by New Contributor II
  • 1382 Views
  • 2 replies
  • 0 kudos
  • 1382 Views
  • 2 replies
  • 0 kudos
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hey there @IssacClose​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....

  • 0 kudos
1 More Replies
Barb
by New Contributor III
  • 3231 Views
  • 2 replies
  • 0 kudos

How do I get a list of the tables that I personally created?

I know that I can get a list of all of the table names in a given 'database' by using (if the 'database' was named "scratch"): show tables from scratchHow do I get a list just like that, but that only lists the tables that I created?

  • 3231 Views
  • 2 replies
  • 0 kudos
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hi @Barb Krienke​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 0 kudos
1 More Replies
KNP
by New Contributor
  • 2340 Views
  • 2 replies
  • 0 kudos

passing array as a parameter to PandasUDF

Hi Team,My python dataframe is as below.The raw data is quite a long series of approx 5000 numbers. My requirement is to go through each row in RawData column and calculate 2 metrics. I have created a function in Python and it works absolutely fine. ...

image
  • 2340 Views
  • 2 replies
  • 0 kudos
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hello @Kausthub NP​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 0 kudos
1 More Replies
HariharaSam
by Contributor
  • 13561 Views
  • 4 replies
  • 4 kudos

Using variables in Spark SQL

Is there a way to declare variables in Spark SQL like we do it in T-SQL?

  • 13561 Views
  • 4 replies
  • 4 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 4 kudos

Hi @Hariharan Sambath​, We haven't heard from you on the last response from @Debayan Mukherjee​ and @Hubert Dudek​, and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please share it with the community as...

  • 4 kudos
3 More Replies
JasonN
by New Contributor II
  • 1461 Views
  • 2 replies
  • 2 kudos

Resolved! DLT Cluster accessing to S3 bucket without Instance Profile attached

Hi Team,Can anyone please help me figure out how to configure Delta Live Tables Cluster accessing AWS S3 bucket without Instance profile defined in Cluster's JSON?The idea is, the user who is running the DLT Cluster has Storage Credentials and Extern...

  • 1461 Views
  • 2 replies
  • 2 kudos
Latest Reply
Vivian_Wilfred
Honored Contributor
  • 2 kudos

Hi @Jason Nam​ , DLT and unity catalog are not integrated yet. The cluster-notebook setup uses UC and can access S3 but not the DLT jobs. Please check the limitations in this document (7th point):https://docs.databricks.com/release-notes/unity-catalo...

  • 2 kudos
1 More Replies
SailajaB
by Valued Contributor III
  • 9350 Views
  • 10 replies
  • 10 kudos

Resolved! Is there a way to capture the notebook logs from ADF pipeline?

Hi,I would like to capture notebook custom log exceptions(python) from ADF pipeline based on the exceptions pipeline should got succeed or failed.Is there any mechanism to implement it. In my testing ADF pipeline is successful irrespective of the log...

  • 9350 Views
  • 10 replies
  • 10 kudos
Latest Reply
User16826994569
New Contributor III
  • 10 kudos

Hi SailajaB,Try this out.Notebook, once executed successfully return a long JSON formatted output. We need to specify appropriate nodes to fetch the output. In below screenshot we can see that when notebook ran it returns empName & empCity as output....

  • 10 kudos
9 More Replies
vjraitila
by New Contributor III
  • 1411 Views
  • 3 replies
  • 5 kudos

Strategy for streaming ETL and Delta Lake before Delta Live Tables existed

What was the established architectural pattern for doing streaming ETL with Delta Lake before DLT was a thing? And incidentally, what approach would you take in the context of delta-oss today? The pipeline definitions would not have had to be declara...

  • 1411 Views
  • 3 replies
  • 5 kudos
Latest Reply
Vidula
Honored Contributor
  • 5 kudos

Hi @Veli-Jussi Raitila​ Does @Shanmugavel Chandrakasu​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 5 kudos
2 More Replies
sternman
by New Contributor II
  • 2086 Views
  • 2 replies
  • 2 kudos

Can I change the VMs IP address from dynamic to static on Azure without breaking anything?

Hello community;I am using Databricks on Azure and I am curious to know the implications of changing the IP address of the underlying VM's from dynamic to static - that is, the IP addresses that the clusters get, I would just like them to always have...

  • 2086 Views
  • 2 replies
  • 2 kudos
Latest Reply
Vidula
Honored Contributor
  • 2 kudos

Hi @Jonathan Stern​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 2 kudos
1 More Replies
shesa
by New Contributor
  • 808 Views
  • 2 replies
  • 0 kudos

Unable to login to https://community.cloud.databricks.com/login.html

Unable to login to https://community.cloud.databricks.com/login.htmlYesterday only I registered, able to login and create cluster.Today unable to login with same credential. I do not receive any email also while trying Password reset.

  • 808 Views
  • 2 replies
  • 0 kudos
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hi @SHESADEV SAMAL​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 0 kudos
1 More Replies
Aaron1234567
by New Contributor III
  • 3097 Views
  • 9 replies
  • 3 kudos

Delta Live Table Pipeline - Azure cluster fail

Failed to launch pipeline cluster 0802-171503-4m02lexd: The operation could not be performed on your account with the following error message: azure_error_code: OperationNotAllowed, azure_error_message: Operation could not be completed as it results ...

  • 3097 Views
  • 9 replies
  • 3 kudos
Latest Reply
Vidula
Honored Contributor
  • 3 kudos

Hi there @Aaron LeBato​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you...

  • 3 kudos
8 More Replies
Brendon_Daugher
by New Contributor II
  • 879 Views
  • 0 replies
  • 0 kudos

Understanding Dependency Update Failure

Heyooooo!I'm using Azure Databricks and sparklyr to do some geospatial analysis.Before I actually work with Spark Dataframes, I've been using the R packagesstarsandsfto do some preprocessing on my data so that it's easier to interact with later.In or...

  • 879 Views
  • 0 replies
  • 0 kudos
Anonymous
by Not applicable
  • 8040 Views
  • 0 replies
  • 1 kudos

Yay! It's September! ���� On September 22nd we are hosting another Community Social - we're doing these monthly ! We want to mak...

Yay! It's September! On September 22nd we are hosting another Community Social - we're doing these monthly ! We want to make sure that we all have the chance to connect as a community often. Come network, talk data, and just get social! Join us for o...

  • 8040 Views
  • 0 replies
  • 1 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels