Hi @Kaniz Fatma Attended the webinar on 18th Oct uploaded the datalakehouse cert but didnt recieve voucher as of now,also didnt recieve data engg associate certificate.Please help!!
I wanted to setup Autoloader to process files from Azure Data Lake (Blob) automatically whenever new files arrive. For this to work, I wanted to know if AutoLoader requires that the cluster is on all the time.
@Kaniz Fatma , If my cluster is not active, and I have uploaded 50 files in storage location, then where this Auto Loader will list out these 50 files. Will it use any checkpoint location, if yes, then how can I set the checkpoint location in Cloud ...
Hi @Lindsay Olson @Kaniz Fatma ,I have cleared my Databricks Certified Data Engineer Associate on 29 October 2022. but haven't received my badge or certificate yet .Can you guys please help .Thanks
Hello,I passed Databricks Certified Data Engineer Associate on 30 October 2022, but still didn't receive my certificate/badge. Could you please help me to obtain it?Regards,Ali
Hi @ali.ganbarov ali.ganbarov We are really sorry for the delays.The certification has been issued but due to the lag in the system, it is taking time. Our team is working on it. Please visit the credible site once.Thanks and Regards
Is it possible to schedule different runs for job with parameters? I have a notebook that generates data based on the supplied parameter but would like to schedule runs instead of manually starting them. I assume that this would be possible using the...
You can pass parameters for your task. Each task type has different requirements for formatting and passing the parameters. https://docs.databricks.com/workflows/jobs/jobs.html#create-a-jobREST API can also pass parameters fro jobs. Tokens replace pa...
I have a cluster with the configuration of 400 GB RAM, 160 Cores.Which of the following would be the ideal configuration to use in case of one or more VM failures?Cluster A: Total RAM 400 GB Total Cores 160 Total VMs: 1 400 GB/Exec & 160 c...
@Santhosh Raj can you please confirm cluster sizes you are taking are related to driver and worker node. how much you want to allocate to Driver and Worker? once we are sure about type of driver and worker we would like to pick, we need to enable au...
everything was working yesterday, but today it stopped working as below: The example from the DB website does not work either with the same error. The page source says This is affecting my work~~~a bit annoying, is DB people going to look into this ...
Is there an option or shortcut (or is it in the pipeline) to quickly change / insert a cell to a specific language in Databricks?Triggering B + P would for instance add a new cell below with %pythonTriggering M would for instance change that cell to ...
Hi @Oscar Dyremyhr , It would mean a lot if you could select the "Best Answer" to help others find the correct answer faster.This makes that answer appear right after the question, so it's easier to find within a thread.It also helps us mark the q...
Select case WHEN {{ Month }} = 0 then add_months(current_date(),-13 ) elseWHEN {{ Month }}> month(add_months(current_date(),-1)) then add_months(to_date(concat(year(current_date())-1,'-',{{Month}},'-',1)),-13) else add_months(to_date(conc...
Hi @Sajid Thavalengal Rahiman , It would mean a lot if you could select the "Best Answer" to help others find the correct answer faster.This makes that answer appear right after the question, so it's easier to find within a thread.It also helps us ...
The docs for the CMK for workspace storage states:After you add a customer-managed key for storage, you cannot later rotate the key by setting a different key ARN for the workspace. However, AWS provides automatic CMK master key rotation, which rotat...
Hi @Constantino Schillebeeckx , You can update/rotate CMK at a later time (on a running workspace). Please refer: https://docs.databricks.com/security/keys/customer-managed-keys-managed-services-aws.html?_ga=2.214562071.1895504292.1667411694-6435253...
Hi there,I've recently installed Apache Sedona on my cluster, according to the detailed instructions here. My Databricks runtime version is 9.1 LTS (includes Apache Spark 3.1.2, Scala 2.12).The installation included the apache-sedona library from PyP...
Thank you @Debayan Mukherjee for the prompt reply. I've followed the instructions carefully, but now every time I try to run a cell in my notebook I get a "Cancelled" message. It clearly didn't work. Any advice?Your help is much appreciated.
Quite soon after moving from VMs to containers, I started crafting my own images. That way notebooks have all the necessary libraries already there and no need to do any Pipping/installing in the notebook.As requirements get more complex, now I'm at ...
Hi @Jari Turkia , Please check if this helps: https://developers.redhat.com/blog/2019/04/24/how-to-run-systemd-in-a-container#other_cool_features_about_podman_and_systemdAlso, you can run ubuntu /redhat linux OS inside containers which will have sys...
When there is a schema change while reading and writing to a stream, will the schema changes be automatically handled by sparkor do we need to include the option(mergeschema=True)?Eg:df.writeStream .option("mergeSchema", "true") .format("delta") .out...
mergeSchema doesn't support all operations. In some cases .option("overwriteSchema", "true") is needed. MergeSchema doesn't support:Dropping a columnChanging an existing column's data type (in place)Renaming column names that differ only by case (e.g...
I have a query that is hitting a table I have access too. Granting access to everyone is not an option. I am using this query in a SQL Dashboard. One of the where clause conditions uses a parameter populated by another query. I want this parameter qu...
