- 1691 Views
- 1 replies
- 2 kudos
docs.databricks.com
Are EBS volumes used by Databricks Clusters are encrypted especially the root volumes
- 1691 Views
- 1 replies
- 2 kudos
- 2 kudos
Yes these EBS volumes are encrypted. Earlier root volume encryptions were not supported but recently this encryption is also enabled (since Apr, 2021)please find more details on the below docs pagehttps://docs.databricks.com/clusters/configure.html#e...
- 2 kudos
- 8361 Views
- 6 replies
- 5 kudos
Resolved! /dbfs is empty
Why does /dbfs seem to be empty in my Databricks cluster ?If I run %sh ls /dbfsI get no output.I am looking for the databricks-datasets subdirectory ? I can't find it under /dbfs
- 8361 Views
- 6 replies
- 5 kudos
- 4946 Views
- 3 replies
- 2 kudos
Resolved! log error to cosmos db
Objective:- Retrieve objects from an S3 bucket using a 'get' api call, write the retrieved object to azure datalake and in case of errors like 404s (object not found) write the error message to cosmos DB"my_dataframe" consists of the a column (s3Obje...
- 4946 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Sandesh Puligundla​ issue is that you are using spark context inside foreachpartition. You can create a dataframe only on the spark driver. Few stack overflow references https://stackoverflow.com/questions/46964250/nullpointerexception-creatin...
- 2 kudos
- 4227 Views
- 2 replies
- 4 kudos
Resolved! Identifying the right tools for the job
Hi all, thank you for taking the time to attend to my post. A background to preface, my team and I have been prototyping an ML model that we would like to push into the production and deployment phase. We have been prototyping on Jupyter Notebooks bu...
- 4227 Views
- 2 replies
- 4 kudos
- 4 kudos
For production model serving, why not just use MLflow Model Serving? You just code it up/import it with the notebooks, then Log it using MLflow, then Register it with the MLflow Registry, then there is a nice UI to serve it using Model Serving. It wi...
- 4 kudos
- 1949 Views
- 1 replies
- 4 kudos
Enabling of Task Orchestration feature in Jobs via API as well Databricks supports the ability to orchestrate multiple tasks within a job. You must en...
Enabling of Task Orchestration feature in Jobs via API as wellDatabricks supports the ability to orchestrate multiple tasks within a job. You must enable this feature in the admin console. Once enabled, this feature cannot be disabled. To enable orch...
- 1949 Views
- 1 replies
- 4 kudos
- 4 kudos
@Mohit Miglani​ this will be really helpful for those who prefer CLI / API more than the UI.
- 4 kudos
- 5125 Views
- 3 replies
- 3 kudos
Passing parameters from DevOps Pipeline/release to DataBricks Notebook
Hi,This is all a bit new to me.Does anybody have any idea how to pass a parameter to the Databricks notebook.I have a DevOps pipeline/release that moves my databricks notebooks towards QA and Production environment. The only problem I am facing is th...
- 5125 Views
- 3 replies
- 3 kudos
- 3 kudos
@Mario Walle​ - If @Hubert Dudek​'s answer solved the issue, would you be happy to mark his answer as best so that it will be more visible to other members?
- 3 kudos
- 2599 Views
- 0 replies
- 0 kudos
On-prem DNS - entries to be added
Hello,I have a question about DNS addresses Databricks uses.In our network configuration, we are using custom VNet injection with no public IPs, and are required to use on-premises corporate DNS.Therefore we would like to add necessary entries to on-...
- 2599 Views
- 0 replies
- 0 kudos
- 5844 Views
- 4 replies
- 2 kudos
Resolved! Does Z-ordering speed up reading of a single file?
Situation: we have one partion per date, and it just so happens that each partition ends up (after optimize) as *a single* 128mb file. We partition on date, and zorder on userid, and our query is something like "find max value of column A where useri...
- 5844 Views
- 4 replies
- 2 kudos
- 2 kudos
Z-Order will make sure that in case you need to read multiple files, these files are co-located.For a single file this does not matter as a single file is always local to itself.If you are certain that your spark program will only read a single file,...
- 2 kudos
- 5169 Views
- 4 replies
- 0 kudos
Databricks JDBC 2.6.19 documentation
I am searching for the Databricks JDBC 2.6.19 documentation page. I can find release notes from the Databricks download page (https://databricks-bi-artifacts.s3.us-east-2.amazonaws.com/simbaspark-drivers/jdbc/2.6.19/docs/release-notes.txt) but on Mag...
- 5169 Views
- 4 replies
- 0 kudos
- 0 kudos
By the way what is still wild, is that the Simba docs say 2.6.16 does only support until Spark 2.4 while the release notes on Databricks download page say 2.6.16 already supports Spark 3.0. Strange that we get contradicting info from the actual driv...
- 0 kudos
- 4503 Views
- 1 replies
- 2 kudos
Resarting existing community edition clusters
I am new to Databricks community edition. I was following the quckstart guide and running through basic cluster management - create, start, etc. For whatever reason, I cannot restart an e3xisting cluster. There is nothing in the cluster event logs or...
- 4503 Views
- 1 replies
- 2 kudos
- 2 kudos
Community free edition is quite limited so it can be the reason.
- 2 kudos
- 1091 Views
- 0 replies
- 0 kudos
spacecoastdaily.com
This Vigor Now male improvement pill contains still up in the air trimmings that together work on working on your overall prosperity by boosting the levels and production of testosterone in your body. Such extended testosterone creation can certainly...
- 1091 Views
- 0 replies
- 0 kudos
- 18483 Views
- 11 replies
- 6 kudos
Resolved! Autocomplete parentheses, quotation marks, brackets and square stopped working
Hello guys, can someone help me?Autocomplete parentheses, quotation marks, brackets and square stopped working in python notebooks.How can I fix this?Daniel
- 18483 Views
- 11 replies
- 6 kudos
- 6 kudos
@Piper Wilson​ , @Werner Stinckens​ Thank you so much for your help.I made the suggestion of the @Jose Gonzalez​ and now it works.
- 6 kudos
- 4253 Views
- 2 replies
- 4 kudos
Resolved! Generating Spark SQL query using Python
I have a Spark SQL notebook on DB where I have a sql query likeSELECT * FROM table_name WHERE condition_1 = 'fname' OR condition_1 = 'lname' OR condition_1 = 'mname' AND condition_2 = 'apple' AND condition_3 ='orange'There are a lot ...
- 4253 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi @John Constantine​ ,I think you can also use arrays_overlap() for your OR statements docs here
- 4 kudos
- 4235 Views
- 4 replies
- 5 kudos
Resolved! Conditionally create a dataframe
I would like to implement a simple logic:if Df1 is empty return Df2 else newDf = Df1.union(Df2)May happened that Df1 is empty and the output is simply: []. In that case I do not need union.I have it like this but getting error when creating datafra...
- 4235 Views
- 4 replies
- 5 kudos
- 6215 Views
- 2 replies
- 1 kudos
- 6215 Views
- 2 replies
- 1 kudos
- 1 kudos
@Vaibhav Gour​ , It kinda depends on the case:if there are no workers available when your job starts, you get an error. As the cluster is unable to start so code cannot be executed. But this is not an autoscale issue.If you need to scale up, but for ...
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
