- 3455 Views
- 3 replies
- 0 kudos
If our company has an Enterprise Git server deployed on a private network, can we use Repos?
Our team would like to use the Repos functionality but our security prevents outside traffic through public networks. Is there any way we can still use Repos?
- 3455 Views
- 3 replies
- 0 kudos
- 0 kudos
Please contact your account team for some options that are in preview right now.
- 0 kudos
- 7178 Views
- 2 replies
- 6 kudos
How to pass dynamic value in databricks
I have separate column value defined in 13 diffrent notebook and i want merge into 1 databrick notebook and want to pass dynamic parameter using databrick so it will help me to run in single databricks notebook .
- 7178 Views
- 2 replies
- 6 kudos
- 6 kudos
Hi @siddhesh Bhavar you can use widgets with the %run command to achieve this. https://docs.databricks.com/notebooks/widgets.html#use-widgets-with-run%run /path/to/notebook $X="10" $Y="1"
- 6 kudos
- 11218 Views
- 5 replies
- 12 kudos
The database and tables disappears when I delete the cluster
Hi guys,I have a trial databricks account, I realized that when I shutdown the cluster my databases and tables is disappear .. that is correct or thats is because my account is trial ?
- 11218 Views
- 5 replies
- 12 kudos
- 12 kudos
@William Scardua if it's an external hive metastore or Glue catalog you might be missing the configuration on the cluster. https://docs.databricks.com/data/metastores/index.htmlAlso as mentioned by @Hubert Dudek , if it's a community edition then t...
- 12 kudos
- 15452 Views
- 6 replies
- 3 kudos
Resolved! How do you create a Sandbox in your data environment ?
Hi guys,How do you create a Sandbox in your data environment ? have any idea ?Azzure/AWS + Data Lake + Databricks
- 15452 Views
- 6 replies
- 3 kudos
- 3 kudos
In a sandbox environment, you will find the Designer enabled. You can activate Designer by selecting the design icon Designer. on a page, or by choosing the Design menu item in the Settings Settings menu.
- 3 kudos
- 7614 Views
- 2 replies
- 11 kudos
Resolved! When trying to use pyodbc connector to write files to SQL server receiving error. java.lang.ClassNotFoundException Any alternatives or ways to fix this?
jdbcUsername = ******** jdbcPassword = *************** server_name = "jdbc:sqlserver://***********:******" database_name = "********" url = server_name + ";" + "databaseName=" + database_name + ";" table_name = "PatientTEST" try: df.write \ ...
- 7614 Views
- 2 replies
- 11 kudos
- 11 kudos
please check following code:df.write.jdbc( url="jdbc:sqlserver://<host>:1433;database=<db>;user=<user>;password=<password>;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30;driver=com.microsof...
- 11 kudos
- 19940 Views
- 11 replies
- 2 kudos
Resolved! Validate a schema of json in column
I have a dataframe like below with col2 as key-value pairs. I would like to filter col2 to only the rows with a valid schema. There could be many of pairs, sometimes less, sometimes more and this is fine as long as the structure is fine. Nulls in col...
- 19940 Views
- 11 replies
- 2 kudos
- 2 kudos
@Bartosz Wachocki - Thank you for sharing your solution and marking it as best.
- 2 kudos
- 14558 Views
- 13 replies
- 5 kudos
Pyspark vs Pandas
Would like to better understand the advantage of writing a python notebook in pyspark vs pandas. Does the entire notebook need to be written in pyspark to realize the performance benefits. I currently have a script using pandas for all my transformat...
- 14558 Views
- 13 replies
- 5 kudos
- 5 kudos
You can use the free Community Edition of Databricks that includes 10.0 runtime.
- 5 kudos
- 16210 Views
- 7 replies
- 7 kudos
Resolved! mlflow RESOURCE_ALREADY_EXISTS
I tried to log some run in my Databricks Workspace and I'm facing the following error: RESOURCE_ALREADY_EXISTS when I try to log any run.I could replicate the error with the following code:import mlflow import mlflow.sklearn from mlflow.tracking impo...
- 16210 Views
- 7 replies
- 7 kudos
- 7 kudos
Hi @Miguel Ángel Fernández it’s not recommended to “link” the Databricks and AML workspaces, as we are seeing more problems. You can refer to the instructions found below for using MLflow with AML. https://docs.microsoft.com/en-us/azure/machine-l...
- 7 kudos
- 7325 Views
- 4 replies
- 3 kudos
read percentage values in spark ( no casting )
I have a xlsx file which has a single column ;percentage30%40%50%-10%0.00%0%0.10%110%99.99%99.98%-99.99%-99.98%when i read this using Apache-Spark out put i get is,|percentage|+----------+| 0.3|| 0.4|| 0.5|| -0.1|| 0.0|| ...
- 7325 Views
- 4 replies
- 3 kudos
- 3 kudos
Affirmative. This is how excel stores percentages. What you see is just cell formatting.Databricks notebooks do not (yet?) have the possibility to format the output.But it is easy to use a BI tool on top of Databricks, where you can change the for...
- 3 kudos
- 45935 Views
- 18 replies
- 6 kudos
Resolved! java.lang.OutOfMemoryError: GC overhead limit exceeded. [ solved ]
solution :- i don't need to add any executor or driver memory all i had to do in my case was add this : - option("maxRowsInMemory", 1000). Before i could n't even read a 9mb file now i just read a 50mb file without any error.{ val df = spark.read .f...
- 45935 Views
- 18 replies
- 6 kudos
- 6 kudos
can you try without: .set("spark.driver.memory","4g") .set("spark.executor.memory", "6g")It is clearly show that there is no 4gb free on driver and 6gb free on executor (you can share hardware cluster details also).You can not also allocate 100% for ...
- 6 kudos
- 19988 Views
- 8 replies
- 6 kudos
How to send a list as parameter in databricks notebook task
Hi,How we can pass a list as parameter in data bricks notebook to run the notebook parallelly for list of values.Thank you
- 19988 Views
- 8 replies
- 6 kudos
- 6 kudos
another another way (in databricks you can achieve everything many ways) is to encode list using json library:import json print type(json.dumps([1, 2, 3])) #>> <type 'str'>
- 6 kudos
- 5670 Views
- 5 replies
- 6 kudos
Resolved! How to import SqlDWRelation from com.databricks.spark.sqldw
Hello, All - I'm working on a project using the SQL DataWarehouse connector built into Databricks (https://docs.databricks.com/data/data-sources/azure/synapse-analytics.html). From there, I'm trying to extract information from the logical plan / logi...
- 5670 Views
- 5 replies
- 6 kudos
- 6 kudos
@Werner Stinckens Thanks for the reply! The SQL DW Connector itself is working just fine and I can retrieve the results from the SQL DW. I'm trying to extract the metadata (i.e. the Server, Database, and Table name) from the logical plan (or throu...
- 6 kudos
- 2083 Views
- 0 replies
- 1 kudos
How do you upload a folder of csv files from your local machine into the Databricks platform?
I am working with time-series data, where each day is a separate csv file. I have tried to load a zip file to FileStore but then cannot use the magic command to unzip, most likely because it is in the tmp folder. Is there a workaround for this proble...
- 2083 Views
- 0 replies
- 1 kudos
- 5882 Views
- 6 replies
- 7 kudos
Resolved! How to get a minor DBR image?
In short, we aim to add a step to a CI job that would run tests in a container, which supposedly should mimic DBR of our clusters – currently we use 7.3 . We consider using one of databricksruntime images (possibly a standard:7.x for now, https://hub...
- 5882 Views
- 6 replies
- 7 kudos
- 7 kudos
Hi @Sepideh Ebrahimi , since cluster is Databricks proprietary, you ca not run it locally. as @Werner Stinckens said, you can build your own image but that has to be run in cluster. but there is databricks connect (https://docs.databricks.com/dev-...
- 7 kudos
- 10753 Views
- 5 replies
- 6 kudos
Apart from notebook , is it possible to deploy an application (Pyspark , or R+spark) as a package or file and execute them in Databricks ?
Hi,With the help of Databricks-connect i was able to connect the cluster to my local IDE like Pycharm and Rstudio desktop version and able to develop the application and committed the code in Git.When i try to add that repo to the Databricks workspac...
- 10753 Views
- 5 replies
- 6 kudos
- 6 kudos
may be you will be interested our db connect . not sure if that resolve your issue to connect with 3rd party tool and setup ur supported IDE notebook serverhttps://docs.databricks.com/dev-tools/databricks-connect.html
- 6 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
