- 5274 Views
- 9 replies
- 1 kudos
Configure jobs throttling for ephemeral cluster ETLs
Hi,Is it possible to configure job throttling in order to queue jobs across a workspace after a given number of concurrent execution when using the ephemeral cluster pattern? The reason is mainly for cost control. We prefer reducing performance rathe...
- 5274 Views
- 9 replies
- 1 kudos
- 1 kudos
Thanks for the help josephk. I will continue to use an interactive cluster for the time being until the release of that new feature. Hopefully, it will allow my use case. Is there visibility on the roadmap for an ETA or more information on it?
- 1 kudos
- 1794 Views
- 1 replies
- 0 kudos
Installing python modules on databricks job clusters
Different than all-purpose clusters, the databricks job new cluster configuration window does not have a "Libraries" tab, in which specific python modules could be installed. What's the best practice for installing python modules on such clusters?
- 1794 Views
- 1 replies
- 0 kudos
- 0 kudos
It turns out that the option exists outside of the cluster configuration scope, in the task configuration window itself - under "Advanced options" -> "Add dependent libraries".
- 0 kudos
- 7584 Views
- 10 replies
- 12 kudos
Resolved! Databricks Certified Professional Data Scientist Exam Question Types
Hello,I am not seeing a lot of information regarding the Databricks Certified Professional Data Scientistexam. I took the Associate Developer in Apache Spark Exam last year and the materials for the exam seemed much more focused than what I found for...
- 7584 Views
- 10 replies
- 12 kudos
- 12 kudos
Hello @Sundar R​ , Yes I took the exam. Unfortunately I fail to reach the pass mark even though I got close. Things I could have did different:I focused so much in mastering each topics i.e. linear, logistic & regularized regression, ALS and etc. But...
- 12 kudos
- 3035 Views
- 2 replies
- 1 kudos
Resolved! Issues with using Databricks-Connect and Petastorm
Has anyone successfully used Petastorm + Databricks-Connect + Delta Lake?The use case is being able to use DeltaLake as a data store regardless of whether I want to use the databricks workspace or not for my training tasks.I'm using a cloud-hosted ju...
- 3035 Views
- 2 replies
- 1 kudos
- 1 kudos
because its janky or why? I don't need it for customer facing production. More so for if I'm using my own HPC or local workstation, but I want to access data from delta lake. Figured it was easier/preferable to setting up my own spark environment loc...
- 1 kudos
- 20640 Views
- 4 replies
- 5 kudos
Resolved! parquet file to include partitioned column in file
HI,I have a daily scheduled job which processes the data and write as parquet file in a specific folder structure like root_folder/{CountryCode}/parquetfiles. Where each day job will write new data for countrycode under the folder for countrycodeI am...
- 20640 Views
- 4 replies
- 5 kudos
- 5 kudos
Most external consumers will read partition as column when are properly configured (for example Azure Data Factory or Power BI).Only way around is that you will duplicate column with other name (you can not have the same name as it will generate conf...
- 5 kudos
- 962 Views
- 0 replies
- 0 kudos
Hi All, I hope you're doing well I am facing issue while installing an python library on ADB Cluster. lib - PyCaret ( latest version) its not gett...
Hi All,I hope you're doing wellI am facing issue while installing an python library on ADB Cluster.lib - PyCaret ( latest version)its not getting install and showing me 'Failed' Status.It would be great if you can help here !!Thanks
- 962 Views
- 0 replies
- 0 kudos
- 4358 Views
- 2 replies
- 1 kudos
Resolved! Cannot Get Databricks SQL to read external Hive Metastore
I have followed the documentation and using the same metastore config that is working in the Data Engineering context. When attempting to view the Databases, I get the error:Encountered an internal errorThe following information failed to load:The li...
- 4358 Views
- 2 replies
- 1 kudos
- 1 kudos
@Bilal Aslam​ I didn't think to look there before since I hadn't tried to run any queries. I see the failed SHOW DATABASES queries in history and they identify the error: Builtin jars can only be used when hive execution version == hive metastore v...
- 1 kudos
- 14268 Views
- 18 replies
- 8 kudos
Resolved! NoSuchMethodError: org.apache.spark.sql.catalyst.json.CreateJacksonParser on Databricks Cloud (but not on Spark Directly)
Hi,I'm working for Couchbase on the Couchbase Spark Connector and noticed something weird which I haven't been able to get to the bottom of so far.For query DataFrames we use the Datasource v2 API and we delegate the JSON parsing to the org.apache.sp...
- 14268 Views
- 18 replies
- 8 kudos
- 8 kudos
Since there hasn't been any progress on this for over a month, I applied a workaround and copied the classes into the connector source code so we don't have to rely on the databricks classloader. It seems to work in my testing and will be released wi...
- 8 kudos
- 3350 Views
- 3 replies
- 0 kudos
I cannot access my account, please help
Hi, when I am trying to log into my account, its showing "invalid email address and password". But,I am sure everything is correct. I request @Kaniz Fatma​ , @Harikrishnan Kunhumveettil​ and @Prabakar Ammeappin​ to please look into it and resolve thi...
- 3350 Views
- 3 replies
- 0 kudos
- 581 Views
- 0 replies
- 0 kudos
Grand Surgical, established in 2010, has been manufacturing high-quality Surgical instruments for all disciplines of surgery. Cardiac, Vascular, denta...
Grand Surgical, established in 2010, has been manufacturing high-quality Surgical instruments for all disciplines of surgery. Cardiac, Vascular, dental, ophthalmic.We develop and deliver hospitals and medical professionals worldwide with superior qua...
- 581 Views
- 0 replies
- 0 kudos
- 2290 Views
- 0 replies
- 2 kudos
Databricks Community Edition Not able to Login Account
I am studying databricks and I have an community edition account since November 19, 2021 and from December 22nd I am not able to login. "Invalid email address or password" error is thrown. When forgot password link is clicked no email is sent to regi...
- 2290 Views
- 0 replies
- 2 kudos
- 628 Views
- 0 replies
- 0 kudos
The Club Price is a leading supplier of high-quality, affordable products whose clientele spans all over Texas and neighboring areas since 1992. We ha...
The Club Price is a leading supplier of high-quality, affordable products whose clientele spans all over Texas and neighboring areas since 1992. We have an exemplary track record of increasing our partners’ sales with our variety in products. We have...
- 628 Views
- 0 replies
- 0 kudos
- 771 Views
- 0 replies
- 0 kudos
Rednirus Mart is a Third-Party Pharma Manufacturer and Supplier. If you are looking For Pharma Contract manufacturers For Ayurvedic Medicine Manufactu...
Rednirus Mart is a Third-Party Pharma Manufacturer and Supplier. If you are looking For Pharma Contract manufacturers For Ayurvedic Medicine Manufacturer Company in your region. Rednirus Mart is one of the leading one and their products are manufactu...
- 771 Views
- 0 replies
- 0 kudos
- 2961 Views
- 1 replies
- 3 kudos
Use '%sql' inside a python cmd cell?
Hi so I want to essentially execute a sql query if a condition is met. So one of my cells in my python notebook is a sql query (%sql followed by the query). Is there any way to put that in an 'IF' statement ie if an environment variable = some value,...
- 2961 Views
- 1 replies
- 3 kudos
- 3 kudos
in python cell just use:query = "SELECT 1"spark.sql(query)
- 3 kudos
- 9512 Views
- 4 replies
- 2 kudos
Upgrading from Spark 2.4 to 3.2: Recursive view errors when using
We get errors like this,Recursive view `x` detected (cycle: `x` -> `x`).. in our long-term working code, that has worked just fine in Spark 2.4.5 (Runtime 6.4), when we run it on a Spark 3.2 cluster (Runtime 10.0).It happens whenever we have,<x is a ...
- 9512 Views
- 4 replies
- 2 kudos
- 2 kudos
This is a breaking change introduced in Spark 3.1 From Migration Guide: SQL, Datasets and DataFrame - Spark 3.1.1 Documentation (apache.org)In Spark 3.1, the temporary view will have same behaviors with the permanent view, i.e. capture and store runt...
- 2 kudos
| User | Count |
|---|---|
| 1614 | |
| 771 | |
| 349 | |
| 286 | |
| 253 |
