cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Digan_Parikh
by Valued Contributor
  • 7863 Views
  • 2 replies
  • 3 kudos

Resolved! Default Query Limit 1000

By default, we return back up to 1000 query results when a user runs a cell in Databricks. E.g., if you run display(storeData) and you have ten million customers, the UI will show the first 1000 results. If you graph that by age of customer, similarl...

  • 7863 Views
  • 2 replies
  • 3 kudos
Latest Reply
User16805453151
New Contributor III
  • 3 kudos

This is simple in Databricks SQL, just uncheck LIMIT 1000 in the drop down.

  • 3 kudos
1 More Replies
Digan_Parikh
by Valued Contributor
  • 2111 Views
  • 2 replies
  • 2 kudos

Resolved! Default Query Limit 1000

Is there any way to change the 1000 for the display row limit at workspace, cluster and notebook level?

  • 2111 Views
  • 2 replies
  • 2 kudos
Latest Reply
User16805453151
New Contributor III
  • 2 kudos

This is simple in Databricks SQL, just uncheck LIMIT 1000 in the drop down.

  • 2 kudos
1 More Replies
labromb
by Contributor
  • 5014 Views
  • 6 replies
  • 8 kudos

Resolved! Create Databricks tables dynamically

Hi, I would like to be able to do something like this...create table if not exists table1using parquetlocation = '/mnt/somelocationsome location needs to be a concatenation of static and code generated string. Documentation suggests that location onl...

  • 5014 Views
  • 6 replies
  • 8 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 8 kudos

Hi @Brian Labrom​ ​, We haven’t heard from you since the last response from @Prasanth Mathesh​ and @Pat Sienkiewicz​, and I was checking back to see if you have a resolution yet. If you have any solution, please share it with the community, as it can...

  • 8 kudos
5 More Replies
atul1146
by New Contributor III
  • 1625 Views
  • 2 replies
  • 5 kudos

Resolved! Databricks set up in Prod environment

Hi! can anyone please help me with a documentation which can help me set up integration between data bricks with AWS without a QuickStart default cloud formation template. I would want to use my own CFT rather than using the default due to security ...

  • 1625 Views
  • 2 replies
  • 5 kudos
Latest Reply
Pat
Honored Contributor III
  • 5 kudos

Hi @Atul S​ ,I think that terraform is recommended way to go with Databricks deployment. I mean it's also supported now by the Databricks support.I haven't look much on the CloudFormation setup, because we decided to go with the Terraform in the comp...

  • 5 kudos
1 More Replies
mishravishakha
by New Contributor
  • 1087 Views
  • 2 replies
  • 1 kudos

Resolved! Unable to login Databric partner academy account.

I have registered on databricks partner academy but could not confirm the registration through the mail now that the link is expired and I am unable to log in databricks partner academy account.Please help me with this issue.

  • 1087 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @vishkaha mishra​ Hope all is well! Just wanted to check in if you were able to resolve your issue. If yes, please mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 1 kudos
1 More Replies
AriB101
by New Contributor
  • 792 Views
  • 2 replies
  • 1 kudos

Resolved! Attended 18Oct22 webinar but didnt recieve voucher

Hi @Kaniz Fatma​ Attended the webinar on 18th Oct uploaded the datalakehouse cert but didnt recieve voucher as of now,also didnt recieve data engg associate certificate.​Please help!!

  • 792 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Arindam Bose​ Just a friendly- follow-up, have you got your certification and badge? If yes, please mark the answer as best.Thanks and Regards

  • 1 kudos
1 More Replies
RK_AV
by New Contributor III
  • 3094 Views
  • 6 replies
  • 8 kudos

Resolved! Autoloader cluster

I wanted to setup Autoloader to process files from Azure Data Lake (Blob) automatically whenever new files arrive. For this to work, I wanted to know if AutoLoader requires that the cluster is on all the time.

  • 3094 Views
  • 6 replies
  • 8 kudos
Latest Reply
asif5494
New Contributor III
  • 8 kudos

@Kaniz Fatma​ , If my cluster is not active, and I have uploaded 50 files in storage location, then where this Auto Loader will list out these 50 files. Will it use any checkpoint location, if yes, then how can I set the checkpoint location in Cloud ...

  • 8 kudos
5 More Replies
khishore
by Contributor
  • 2916 Views
  • 9 replies
  • 6 kudos

Resolved! i haven't received my certificate or the badge for Databricks Certified Data Engineer Associate

Hi @Lindsay Olson​ @Kaniz Fatma​ ,I have cleared my Databricks Certified Data Engineer Associate on 29 October 2022. but haven't received my badge or certificate yet .Can you guys please help .Thanks

  • 2916 Views
  • 9 replies
  • 6 kudos
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Great to hear that @khishore selvanathan​.

  • 6 kudos
8 More Replies
133994
by New Contributor III
  • 1618 Views
  • 6 replies
  • 4 kudos

Resolved! Databricks Certified Data Engineer Associate Certificate not received

Hello,I passed Databricks Certified Data Engineer Associate on 30 October 2022, but still didn't receive my certificate/badge. Could you please help me to obtain it?Regards,Ali

  • 1618 Views
  • 6 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @ali.ganbarov ali.ganbarov​ We are really sorry for the delays.The certification has been issued but due to the lag in the system, it is taking time. Our team is working on it. Please visit the credible site once.Thanks and Regards

  • 4 kudos
5 More Replies
129876
by New Contributor III
  • 3503 Views
  • 6 replies
  • 10 kudos

Schedule job runs with different parameters?

Is it possible to schedule different runs for job with parameters? I have a notebook that generates data based on the supplied parameter but would like to schedule runs instead of manually starting them. I assume that this would be possible using the...

  • 3503 Views
  • 6 replies
  • 10 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 10 kudos

You can pass parameters for your task. Each task type has different requirements for formatting and passing the parameters. https://docs.databricks.com/workflows/jobs/jobs.html#create-a-jobREST API can also pass parameters fro jobs. Tokens replace pa...

  • 10 kudos
5 More Replies
Sandy21
by New Contributor III
  • 1310 Views
  • 1 replies
  • 2 kudos

Resolved! Cluster Configuration Best Practices

I have a cluster with the configuration of 400 GB RAM, 160 Cores.Which of the following would be the ideal configuration to use in case of one or more VM failures?Cluster A: Total RAM 400 GB      Total Cores 160   Total VMs: 1   400 GB/Exec & 160 c...

  • 1310 Views
  • 1 replies
  • 2 kudos
Latest Reply
karthik_p
Esteemed Contributor
  • 2 kudos

@Santhosh Raj​ can you please confirm cluster sizes you are taking are related to driver and worker node. how much you want to allocate to Driver and Worker? once we are sure about type of driver and worker we would like to pick, we need to enable au...

  • 2 kudos
lzha174
by Contributor
  • 3148 Views
  • 3 replies
  • 3 kudos

Resolved! ipywidgets stopped displaying today

everything was working yesterday, but today it stopped working as below: The example from the DB website does not work either with the same error. The page source says  This is affecting my work~~~a bit annoying, is DB people going to look into this ...

image image
  • 3148 Views
  • 3 replies
  • 3 kudos
Latest Reply
lzha174
Contributor
  • 3 kudos

Today its back to work! I got a pop up window sayingthis should be the reason it was broken

  • 3 kudos
2 More Replies
ossinova
by Contributor II
  • 1244 Views
  • 2 replies
  • 3 kudos

Resolved! Shortcut for changing cell language (adding magic command)

Is there an option or shortcut (or is it in the pipeline) to quickly change / insert a cell to a specific language in Databricks?Triggering B + P would for instance add a new cell below with %pythonTriggering M would for instance change that cell to ...

  • 1244 Views
  • 2 replies
  • 3 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 3 kudos

Hi @Oscar Dyremyhr​  ​, It would mean a lot if you could select the "Best Answer" to help others find the correct answer faster.This makes that answer appear right after the question, so it's easier to find within a thread.It also helps us mark the q...

  • 3 kudos
1 More Replies
Sajid1
by Contributor
  • 22123 Views
  • 4 replies
  • 5 kudos

Resolved! Parse Syntax error ,can anyone guide me what is going wrong here

Select case WHEN {{ Month }} = 0 then add_months(current_date(),-13 ) elseWHEN {{ Month }}> month(add_months(current_date(),-1)) then add_months(to_date(concat(year(current_date())-1,'-',{{Month}},'-',1)),-13)             else add_months(to_date(conc...

  • 22123 Views
  • 4 replies
  • 5 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 5 kudos

Hi @Sajid Thavalengal Rahiman​ ​, It would mean a lot if you could select the "Best Answer" to help others find the correct answer faster.This makes that answer appear right after the question, so it's easier to find within a thread.It also helps us ...

  • 5 kudos
3 More Replies
Constantino
by New Contributor III
  • 1256 Views
  • 2 replies
  • 2 kudos

CMK for managed services automatic rotation

The docs for the CMK for workspace storage states:After you add a customer-managed key for storage, you cannot later rotate the key by setting a different key ARN for the workspace. However, AWS provides automatic CMK master key rotation, which rotat...

  • 1256 Views
  • 2 replies
  • 2 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 2 kudos

Hi @Constantino Schillebeeckx​ , You can update/rotate CMK at a later time (on a running workspace). Please refer: https://docs.databricks.com/security/keys/customer-managed-keys-managed-services-aws.html?_ga=2.214562071.1895504292.1667411694-6435253...

  • 2 kudos
1 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels