- 2528 Views
- 1 replies
- 2 kudos
Resolved! Cluster Configuration Best Practices
I have a cluster with the configuration of 400 GB RAM, 160 Cores.Which of the following would be the ideal configuration to use in case of one or more VM failures?Cluster A: Total RAM 400 GB Total Cores 160 Total VMs: 1 400 GB/Exec & 160 c...
- 2528 Views
- 1 replies
- 2 kudos
- 2 kudos
@Santhosh Raj​ can you please confirm cluster sizes you are taking are related to driver and worker node. how much you want to allocate to Driver and Worker? once we are sure about type of driver and worker we would like to pick, we need to enable au...
- 2 kudos
- 6547 Views
- 3 replies
- 3 kudos
Resolved! ipywidgets stopped displaying today
everything was working yesterday, but today it stopped working as below: The example from the DB website does not work either with the same error. The page source says This is affecting my work~~~a bit annoying, is DB people going to look into this ...
- 6547 Views
- 3 replies
- 3 kudos
- 3 kudos
Today its back to work! I got a pop up window sayingthis should be the reason it was broken
- 3 kudos
- 3035 Views
- 0 replies
- 3 kudos
Cassandra connection from ADB
Hi , Could anyone please help to know the steps for connecting Cassandra from Azure Databricks? I have followed the steps in https://learn.microsoft.com/en-us/azure/databricks/_static/notebooks/azure/cassandra-azure.html But I am getting below error....
- 3035 Views
- 0 replies
- 3 kudos
- 1317 Views
- 0 replies
- 0 kudos
Databricks Jobs
Consider User A has deployed the job to prod. User B has scheduled the job through an external orchestration tool.User C has got the owner privileges from User A. Whose email id would be displayed while running the databricks job?
- 1317 Views
- 0 replies
- 0 kudos
- 2501 Views
- 2 replies
- 2 kudos
CMK for managed services automatic rotation
The docs for the CMK for workspace storage states:After you add a customer-managed key for storage, you cannot later rotate the key by setting a different key ARN for the workspace. However, AWS provides automatic CMK master key rotation, which rotat...
- 2501 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Constantino Schillebeeckx​ , You can update/rotate CMK at a later time (on a running workspace). Please refer: https://docs.databricks.com/security/keys/customer-managed-keys-managed-services-aws.html?_ga=2.214562071.1895504292.1667411694-6435253...
- 2 kudos
- 34397 Views
- 33 replies
- 14 kudos
- 34397 Views
- 33 replies
- 14 kudos
- 14 kudos
I think plain python code will run with databricks connect (if it is a python program you are writing), and spark sql can be done by spark.sql(...).
- 14 kudos
- 2505 Views
- 2 replies
- 1 kudos
Can't import st_constructors module after installing Apache Sedona
Hi there,I've recently installed Apache Sedona on my cluster, according to the detailed instructions here. My Databricks runtime version is 9.1 LTS (includes Apache Spark 3.1.2, Scala 2.12).The installation included the apache-sedona library from PyP...
- 2505 Views
- 2 replies
- 1 kudos
- 1 kudos
Thank you @Debayan Mukherjee​ for the prompt reply. I've followed the instructions carefully, but now every time I try to run a cell in my notebook I get a "Cancelled" message. It clearly didn't work. Any advice?Your help is much appreciated.
- 1 kudos
- 8833 Views
- 5 replies
- 1 kudos
- 8833 Views
- 5 replies
- 1 kudos
- 1 kudos
Any string between double curly braces {{ }} is treated as a query parameter. widget can be an option here.
- 1 kudos
- 12564 Views
- 2 replies
- 6 kudos
Schema Evolution Issue in Streaming
When there is a schema change while reading and writing to a stream, will the schema changes be automatically handled by sparkor do we need to include the option(mergeschema=True)?Eg:df.writeStream .option("mergeSchema", "true") .format("delta") .out...
- 12564 Views
- 2 replies
- 6 kudos
- 6 kudos
mergeSchema doesn't support all operations. In some cases .option("overwriteSchema", "true") is needed. MergeSchema doesn't support:Dropping a columnChanging an existing column's data type (in place)Renaming column names that differ only by case (e.g...
- 6 kudos
- 35727 Views
- 3 replies
- 5 kudos
Resolved! Parse Syntax error ,can anyone guide me what is going wrong here
Select case WHEN {{ Month }} = 0 then add_months(current_date(),-13 ) elseWHEN {{ Month }}> month(add_months(current_date(),-1)) then add_months(to_date(concat(year(current_date())-1,'-',{{Month}},'-',1)),-13) else add_months(to_date(conc...
- 35727 Views
- 3 replies
- 5 kudos
- 5 kudos
Hi @Sajid Thavalengal Rahiman​ , Have you followed the recommendation given above? Also, could you please paste the whole error with the code?
- 5 kudos
- 1370 Views
- 0 replies
- 1 kudos
Voucher sending date
Completed the attached course today Nov 3rd. In order to avoid writing again and requesting a voucher, when is the voucher for this one expected to be sent?Thanks
- 1370 Views
- 0 replies
- 1 kudos
- 2012 Views
- 1 replies
- 0 kudos
Resolved! How do you run a query as the owner but use a parameter as a viewer
I have a query that is hitting a table I have access too. Granting access to everyone is not an option. I am using this query in a SQL Dashboard. One of the where clause conditions uses a parameter populated by another query. I want this parameter qu...
- 2012 Views
- 1 replies
- 0 kudos
- 0 kudos
It is not possible to do what I want. Somewhat seems like a security flaw but what ever
- 0 kudos
- 3979 Views
- 3 replies
- 2 kudos
Resolved! Orchestrate run of a folder
I'm needing to run the contents of a folder, which can change over time. Is there a way to set up a notebook that can orchestrate running all notebooks in a folder? My though was if I could retrieve a list of the notebooks I could create a loop to ru...
- 3979 Views
- 3 replies
- 2 kudos
- 2 kudos
List all notebooks by making API call and then run them by using dbutils.notebook.run:import requests ctx = dbutils.notebook.entry_point.getDbutils().notebook().getContext() host_name = ctx.tags().get("browserHostName").get() host_token = ctx.apiToke...
- 2 kudos
- 6314 Views
- 5 replies
- 5 kudos
Resolved! How do I clone a repo in Community Edition?
The e-learning videos on DBacademy say we should click on "Repos" and "Add Repo"I cannot find this in my Community Edition UII am a little frustrated that there are so many different versions of the UI and many videos show UI options that we cannot ...
- 6314 Views
- 5 replies
- 5 kudos
- 5 kudos
Hello, just import the .dbc file direct into your user workspace, as explained by Databricks here:https://www.databricks.training/step-by-step/importing-courseware-from-github/The simplest way
- 5 kudos
- 70619 Views
- 3 replies
- 9 kudos
Best practice for logging in Databricks notebooks?
What is the best practice for logging in Databricks notebooks? I have a bunch of notebooks that run in parallel through a workflow. I would like to keep track of everything that happens such as errors coming from a stream. I would like these logs to ...
- 70619 Views
- 3 replies
- 9 kudos
- 9 kudos
@Gimwell Young​ AS @Debayan Mukherjee​ mentioned if you configure verbose logging in workspace level, logs will be moved to your storage bucket that you have provided during configuration. from there you can pull logs into any of your licensed log mo...
- 9 kudos
| User | Count |
|---|---|
| 1613 | |
| 770 | |
| 349 | |
| 286 | |
| 253 |
