- 11913 Views
- 4 replies
- 8 kudos
Resolved! Create Databricks tables dynamically
Hi, I would like to be able to do something like this...create table if not exists table1using parquetlocation = '/mnt/somelocationsome location needs to be a concatenation of static and code generated string. Documentation suggests that location onl...
- 11913 Views
- 4 replies
- 8 kudos
- 8 kudos
FString Python can be used. example > spark.sql(f"CREATE TABLE {table_name} (id INT, name STRING, value DOUBLE, state STRING)")
- 8 kudos
- 4578 Views
- 2 replies
- 5 kudos
Resolved! Databricks set up in Prod environment
Hi! can anyone please help me with a documentation which can help me set up integration between data bricks with AWS without a QuickStart default cloud formation template. I would want to use my own CFT rather than using the default due to security ...
- 4578 Views
- 2 replies
- 5 kudos
- 5 kudos
Hi @Atul S​ ,I think that terraform is recommended way to go with Databricks deployment. I mean it's also supported now by the Databricks support.I haven't look much on the CloudFormation setup, because we decided to go with the Terraform in the comp...
- 5 kudos
- 12141 Views
- 1 replies
- 4 kudos
Resolved! Accept widget value during runtime from user
list1 = ('alpha', 'beta', 'gamma', 'eta', 'Theta')list2 = ('alpha', 'beta')df1 = spark.createDataFrame(list1, 'String').withColumnRenamed('value', 'dataset')df2 = spark.createDataFrame(list2, 'String').withColumnRenamed('value', 'dataset')df = df1.ex...
- 12141 Views
- 1 replies
- 4 kudos
- 4 kudos
Every time the user change selection in the widget, you get that selection—behavior after the dropdown change you can define in widget settings.
- 4 kudos
- 8227 Views
- 0 replies
- 5 kudos
Unable to create a cluster in Databricks with a new subscription
I have upgraded my expired Student subscription to 'Azure subscription 1' in Azure portal today. I want to use Databricks for personal projects as pay-as-you-go.When I go to my Databricks workspace and to my notebook and try to create a cluster,Comp...
- 8227 Views
- 0 replies
- 5 kudos
- 1528 Views
- 1 replies
- 1 kudos
Attended 18Oct22 webinar but didnt recieve voucher
Hi @Kaniz Fatma​ Attended the webinar on 18th Oct uploaded the datalakehouse cert but didnt recieve voucher as of now,also didnt recieve data engg associate certificate.​Please help!!
- 1528 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Arindam Bose​ Just a friendly- follow-up, have you got your certification and badge? If yes, please mark the answer as best.Thanks and Regards
- 1 kudos
- 2763 Views
- 0 replies
- 2 kudos
I will appreciate if there is a black friday deal on the Databricks Data Engineering Associate course or if I can get a personal coupon.
I will appreciate if there is a black friday deal on the Databricks Data Engineering Associate course or if I can get a personal coupon.
- 2763 Views
- 0 replies
- 2 kudos
- 3603 Views
- 0 replies
- 6 kudos
Catch when a notebook fails and terminate command in threaded parallel notebook run
I have a command that is running notebooks in parallel using threading. I want the command to fail whenever one of the notebooks that is running fails. Right now it is just continuing to run the command.Below is the command line that I'm currently ru...
- 3603 Views
- 0 replies
- 6 kudos
- 7146 Views
- 5 replies
- 8 kudos
Autoloader cluster
I wanted to setup Autoloader to process files from Azure Data Lake (Blob) automatically whenever new files arrive. For this to work, I wanted to know if AutoLoader requires that the cluster is on all the time.
- 7146 Views
- 5 replies
- 8 kudos
- 8 kudos
@Kaniz Fatma​ , If my cluster is not active, and I have uploaded 50 files in storage location, then where this Auto Loader will list out these 50 files. Will it use any checkpoint location, if yes, then how can I set the checkpoint location in Cloud ...
- 8 kudos
- 1736 Views
- 0 replies
- 1 kudos
I have created account but forgot to check community edition option while signing up. How this can be solved now.
I have created account but forgot to check community edition option while signing up. How this can be solved now.
- 1736 Views
- 0 replies
- 1 kudos
- 4171 Views
- 6 replies
- 4 kudos
Resolved! Databricks Certified Data Engineer Associate Certificate not received
Hello,I passed Databricks Certified Data Engineer Associate on 30 October 2022, but still didn't receive my certificate/badge. Could you please help me to obtain it?Regards,Ali
- 4171 Views
- 6 replies
- 4 kudos
- 4 kudos
Hi @ali.ganbarov ali.ganbarov​ We are really sorry for the delays.The certification has been issued but due to the lag in the system, it is taking time. Our team is working on it. Please visit the credible site once.Thanks and Regards
- 4 kudos
- 7971 Views
- 4 replies
- 7 kudos
Schedule job runs with different parameters?
Is it possible to schedule different runs for job with parameters? I have a notebook that generates data based on the supplied parameter but would like to schedule runs instead of manually starting them. I assume that this would be possible using the...
- 7971 Views
- 4 replies
- 7 kudos
- 7 kudos
You can pass parameters for your task. Each task type has different requirements for formatting and passing the parameters. https://docs.databricks.com/workflows/jobs/jobs.html#create-a-jobREST API can also pass parameters fro jobs. Tokens replace pa...
- 7 kudos
- 4541 Views
- 1 replies
- 2 kudos
Resolved! Cluster Configuration Best Practices
I have a cluster with the configuration of 400 GB RAM, 160 Cores.Which of the following would be the ideal configuration to use in case of one or more VM failures?Cluster A: Total RAM 400 GB Total Cores 160 Total VMs: 1 400 GB/Exec & 160 c...
- 4541 Views
- 1 replies
- 2 kudos
- 2 kudos
@Santhosh Raj​ can you please confirm cluster sizes you are taking are related to driver and worker node. how much you want to allocate to Driver and Worker? once we are sure about type of driver and worker we would like to pick, we need to enable au...
- 2 kudos
- 9875 Views
- 3 replies
- 3 kudos
Resolved! ipywidgets stopped displaying today
everything was working yesterday, but today it stopped working as below: The example from the DB website does not work either with the same error. The page source says This is affecting my work~~~a bit annoying, is DB people going to look into this ...
- 9875 Views
- 3 replies
- 3 kudos
- 3 kudos
Today its back to work! I got a pop up window sayingthis should be the reason it was broken
- 3 kudos
- 3748 Views
- 0 replies
- 3 kudos
Cassandra connection from ADB
Hi , Could anyone please help to know the steps for connecting Cassandra from Azure Databricks? I have followed the steps in https://learn.microsoft.com/en-us/azure/databricks/_static/notebooks/azure/cassandra-azure.html But I am getting below error....
- 3748 Views
- 0 replies
- 3 kudos
- 1633 Views
- 0 replies
- 0 kudos
Databricks Jobs
Consider User A has deployed the job to prod. User B has scheduled the job through an external orchestration tool.User C has got the owner privileges from User A. Whose email id would be displayed while running the databricks job?
- 1633 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
