- 4535 Views
- 5 replies
- 6 kudos
Resolved! register model - need python 3, but get only python 2
Hi all, I'm trying to register a model with python 3 support, but continue getting only python 2. I can see that runtime 6.0 and above get python 3 by default, but I don't see a way to set neither runtime version, nor python version during model regi...
- 4535 Views
- 5 replies
- 6 kudos
- 6 kudos
Hi team, thanks for getting back to me. Let's put this on hold for now. I will update once it's needed again. It was solely for education purpose and right now I have quite urgent stuff to do.Have a great day.
- 6 kudos
- 6697 Views
- 4 replies
- 1 kudos
Databricks interoperability between cloud environments
While Databricks is currently available and integrated into all three major cloud platforms (Azure, AWS, GCP) , following are pertinent questions that comes across in the real-world scenarios,1) Whether Databricks can be cloud agnostic (i.e.,) In ca...
- 6697 Views
- 4 replies
- 1 kudos
- 1 kudos
You'll be interested in the Unity Catalog.The notebooks should be the same across all the clouds and there are no syntax differences. The key things are going to be just changing paths from S3 to ADL2 and having different usernames/logins across the...
- 1 kudos
- 2995 Views
- 3 replies
- 1 kudos
python dataframe or hiveSql update based on predecessor value?
I have a million in rows that I need to update which looks for the highest count of the predecessor from the same source data and replaces the same value on a different row. For example.Original DF.sno Object Name shape rating1 Fruit apple round ...
- 2995 Views
- 3 replies
- 1 kudos
- 1 kudos
basically you have to create a dataframe (or use a window function, that will also work) which gives you the group combination with the most occurances. So a window/groupby on object, name, shape with a count().Then you have to determine which shape...
- 1 kudos
- 2472 Views
- 1 replies
- 4 kudos
collect_set/ collect_list Pushdown
Hello,I've noticed that Collect_Set and Collect_List are not pushed down to the database?Runtime DB 9.1LTSSpark 3.1.2Database: SnowflakeIs there any way to get a distinct set from a group by in a way that will push down the query to the database?
- 2472 Views
- 1 replies
- 4 kudos
- 4 kudos
Hm so collect_set does not get translated to listagg.Can you try the following?use a more recent version of dbrxuse delta lake as spark sourceuse the latest version of the snowflake connectorcheck if pushdown to snowflake is enabled
- 4 kudos
- 13671 Views
- 11 replies
- 0 kudos
Resolved! Unable to view exported notebooks in HTML format
My team and I noticed an issue lately where notebooks, when exported to HTML format, are not viewable in a stand-alone state anymore. Older notebooks which were exported have no issues, but newer exports are not viewable. The only way we can view t...
- 13671 Views
- 11 replies
- 0 kudos
- 0 kudos
I can confirm that the Community Edition now does correct readable HTML export.
- 0 kudos
- 4052 Views
- 3 replies
- 0 kudos
If our company has an Enterprise Git server deployed on a private network, can we use Repos?
Our team would like to use the Repos functionality but our security prevents outside traffic through public networks. Is there any way we can still use Repos?
- 4052 Views
- 3 replies
- 0 kudos
- 0 kudos
Please contact your account team for some options that are in preview right now.
- 0 kudos
- 7696 Views
- 2 replies
- 6 kudos
How to pass dynamic value in databricks
I have separate column value defined in 13 diffrent notebook and i want merge into 1 databrick notebook and want to pass dynamic parameter using databrick so it will help me to run in single databricks notebook .
- 7696 Views
- 2 replies
- 6 kudos
- 6 kudos
Hi @siddhesh Bhavar you can use widgets with the %run command to achieve this. https://docs.databricks.com/notebooks/widgets.html#use-widgets-with-run%run /path/to/notebook $X="10" $Y="1"
- 6 kudos
- 12783 Views
- 5 replies
- 12 kudos
The database and tables disappears when I delete the cluster
Hi guys,I have a trial databricks account, I realized that when I shutdown the cluster my databases and tables is disappear .. that is correct or thats is because my account is trial ?
- 12783 Views
- 5 replies
- 12 kudos
- 12 kudos
@William Scardua if it's an external hive metastore or Glue catalog you might be missing the configuration on the cluster. https://docs.databricks.com/data/metastores/index.htmlAlso as mentioned by @Hubert Dudek , if it's a community edition then t...
- 12 kudos
- 17012 Views
- 6 replies
- 3 kudos
Resolved! How do you create a Sandbox in your data environment ?
Hi guys,How do you create a Sandbox in your data environment ? have any idea ?Azzure/AWS + Data Lake + Databricks
- 17012 Views
- 6 replies
- 3 kudos
- 3 kudos
In a sandbox environment, you will find the Designer enabled. You can activate Designer by selecting the design icon Designer. on a page, or by choosing the Design menu item in the Settings Settings menu.
- 3 kudos
- 8051 Views
- 2 replies
- 11 kudos
Resolved! When trying to use pyodbc connector to write files to SQL server receiving error. java.lang.ClassNotFoundException Any alternatives or ways to fix this?
jdbcUsername = ******** jdbcPassword = *************** server_name = "jdbc:sqlserver://***********:******" database_name = "********" url = server_name + ";" + "databaseName=" + database_name + ";" table_name = "PatientTEST" try: df.write \ ...
- 8051 Views
- 2 replies
- 11 kudos
- 11 kudos
please check following code:df.write.jdbc( url="jdbc:sqlserver://<host>:1433;database=<db>;user=<user>;password=<password>;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30;driver=com.microsof...
- 11 kudos
- 22329 Views
- 11 replies
- 2 kudos
Resolved! Validate a schema of json in column
I have a dataframe like below with col2 as key-value pairs. I would like to filter col2 to only the rows with a valid schema. There could be many of pairs, sometimes less, sometimes more and this is fine as long as the structure is fine. Nulls in col...
- 22329 Views
- 11 replies
- 2 kudos
- 2 kudos
@Bartosz Wachocki - Thank you for sharing your solution and marking it as best.
- 2 kudos
- 18066 Views
- 13 replies
- 5 kudos
Pyspark vs Pandas
Would like to better understand the advantage of writing a python notebook in pyspark vs pandas. Does the entire notebook need to be written in pyspark to realize the performance benefits. I currently have a script using pandas for all my transformat...
- 18066 Views
- 13 replies
- 5 kudos
- 5 kudos
You can use the free Community Edition of Databricks that includes 10.0 runtime.
- 5 kudos
- 18515 Views
- 7 replies
- 7 kudos
Resolved! mlflow RESOURCE_ALREADY_EXISTS
I tried to log some run in my Databricks Workspace and I'm facing the following error: RESOURCE_ALREADY_EXISTS when I try to log any run.I could replicate the error with the following code:import mlflow import mlflow.sklearn from mlflow.tracking impo...
- 18515 Views
- 7 replies
- 7 kudos
- 7 kudos
Hi @Miguel Ángel Fernández it’s not recommended to “link” the Databricks and AML workspaces, as we are seeing more problems. You can refer to the instructions found below for using MLflow with AML. https://docs.microsoft.com/en-us/azure/machine-l...
- 7 kudos
- 8148 Views
- 4 replies
- 3 kudos
read percentage values in spark ( no casting )
I have a xlsx file which has a single column ;percentage30%40%50%-10%0.00%0%0.10%110%99.99%99.98%-99.99%-99.98%when i read this using Apache-Spark out put i get is,|percentage|+----------+| 0.3|| 0.4|| 0.5|| -0.1|| 0.0|| ...
- 8148 Views
- 4 replies
- 3 kudos
- 3 kudos
Affirmative. This is how excel stores percentages. What you see is just cell formatting.Databricks notebooks do not (yet?) have the possibility to format the output.But it is easy to use a BI tool on top of Databricks, where you can change the for...
- 3 kudos
- 52794 Views
- 18 replies
- 6 kudos
Resolved! java.lang.OutOfMemoryError: GC overhead limit exceeded. [ solved ]
solution :- i don't need to add any executor or driver memory all i had to do in my case was add this : - option("maxRowsInMemory", 1000). Before i could n't even read a 9mb file now i just read a 50mb file without any error.{ val df = spark.read .f...
- 52794 Views
- 18 replies
- 6 kudos
- 6 kudos
can you try without: .set("spark.driver.memory","4g") .set("spark.executor.memory", "6g")It is clearly show that there is no 4gb free on driver and 6gb free on executor (you can share hardware cluster details also).You can not also allocate 100% for ...
- 6 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
