Databricks Community

Error loading h2o model in mlflow

I'm getting the following error when I'm trying to load a h2o model using mlflow for prediction Error: Error Job with key $03017f00000132d4ffffffff$_990da74b0db027b33cc49d1d90934149 failed with an exception: java.lang.IllegalArgumentException:...

Query table access control metadata from Databricks SQL

I'm trying to create a dashboard in Databricks SQL, parameterized by table name. We have a metadata table which contains the names of all the eligible tables, and we use it to populate a drop-down box for the dashboard. This is a simplified version ...

Azure Databricks, how to learn to use practically?

Not sure whether better do ask this in an Azure or Spark subject, but I thought I might get responses appropriate to our use cases here. We have Azure Databricks set up and working, and not had any problems following along the tutorials, but I don't...

How is Databricks Spark different than Spark?

Hey guys, I am looking to create a real-time analytics application and I am pretty new to Data engineering. Any advice here would be appreciated. So I have been l appvalleyooking into spark streaming for my transformation process, so th tutuappe ove...

how to get unique values of a column in pyspark dataframe

like in pandas I usually do df['columnname'].unique()

Question About Access and Filter Data in Databricks

Hi guys, im new using databricks and i have a challenge in my new work. routerlogin I need to access to one the database (the database is on DBFS) result of some ETLS trough any service, can be ODBC or by some API. I need to connect there because I...

Multiple Clusters stuck in pending state during creation.

I have been trying to create a new cluster to use and multiple attempts have gotten stuck in pending: "Finding instances for new nodes, acquiring more instances if necessary" until they time out. Up to today I have had no problems creating clusters ...

How do I make approxSimilarityJoin work on 25k 300-d vectors?

I’m trying to use LSH approxSimilarityJoin on a dataset with ~25k 300-d vectors of floats. It gets stuck and eventually fails with ’Slave lost’ error. The size of cluster and memory are likely not a problem, the failure happens even with 16 nodes, 1...

Best Database for facial recognition/ Fast comparisons of Euclidean distance

Hello people,I'm trying to build a facial recognition application, and I have a working API, that takes in an image of a face and spits out a vector that encodes it. I need to run this on a million faces, store them in a db and when the system goes o...

Databricks delta lake

Can we use databricks delta lake as a data warehouse kind of thing where business analysts can explore data according to their needs ? Delta lake provides following features which I think supports this idea support to sql syntaxprovide ACID guarante...

On Azure Databricks, is it true that a customer cannot change the VNET CIDR range for a workspace and the only way to do so is to deploy a new workspace? If so, why?

Is possible to use jupyter extensions in databricks?

Hi, we need create an interactive map from ipyleaflet library and this use jupyterlab extensionjupyter labextension install @jupyter-widgets/jupyterlab-manager jupyter-leafletWe achieved to show with displayHTML but we lose the widget events

Azure Databricks cluster external IP ranges to be locked down for firewalls

The Spark driver / executors have callbacks to an external service. I need to find out the IP ranges to be locked down for the firewall on the external service. Here is what I found from Azure (https://www.microsoft.com/en-us/download/details.aspx?...

Cluster Pending

Hello, Relatively new to Databricks and I've been using the Community Edition for a little bit now. I've recently been having more and more issues with my clusters pending until they time out before ever starting up. I've seen a few other posts here...