Databricks Community

For Databricks on AWS, when I set up a Kinesis Endpoint as shown here https://docs.databricks.com/administration-guide/cloud-configurations/aws/customer-managed-vpc.html#firewall-appliance-infrastructure should I use Kinesis Firehose or Kinesis Streams?

What is the differentiator between delta sharing and other cloud sharing platform

What is differentiator between delta sharing and other cloud sharing platform.

Logging Lifetime Plot_history_alive as a model

Hi Databricks Community, I have followed the CLV Databricks accelator (https://databricks.com/notebooks/CLV_Part_1_Customer_Lifetimes.html) to do an initial CLV analysis. Thank you for sharing this with the community. My question is that in the note...

Databricks Error in SQL statement: ParseException: mismatched input 'Service_Date

I am running this script in Azure Databricks using spark SQL , getting this error below , I am new to Databricks so wondering any tips on troubleshooting thisError in SQL statement: ParseException:mismatched input 'Service_Date' expecting {'(', 'DESC...

Read metadata through JDBC driver

Dear all, The Spark JDBC driver (SparkJDBC42.jar) is unable to capture certain information from the below table structure: 1. table level comment 2. the TBLPROPERTIES key-value pair information 3. PARTITION BY information However, it captures the co...

Incremental updates to s3 csv files, autoloader, and delta lake updates

I'm using the Databricks autoloader to incrementally load a series of csv files on s3 which I update with an API. My tyipcal work process is to update only the latest year file each night. But, there are ocassions where previous years also get update...

Error loading h2o model in mlflow

I'm getting the following error when I'm trying to load a h2o model using mlflow for prediction Error: Error Job with key $03017f00000132d4ffffffff$_990da74b0db027b33cc49d1d90934149 failed with an exception: java.lang.IllegalArgumentException:...

Query table access control metadata from Databricks SQL

I'm trying to create a dashboard in Databricks SQL, parameterized by table name. We have a metadata table which contains the names of all the eligible tables, and we use it to populate a drop-down box for the dashboard. This is a simplified version ...

Azure Databricks, how to learn to use practically?

Not sure whether better do ask this in an Azure or Spark subject, but I thought I might get responses appropriate to our use cases here. We have Azure Databricks set up and working, and not had any problems following along the tutorials, but I don't...

How is Databricks Spark different than Spark?

Hey guys, I am looking to create a real-time analytics application and I am pretty new to Data engineering. Any advice here would be appreciated. So I have been l appvalleyooking into spark streaming for my transformation process, so th tutuappe ove...

how to get unique values of a column in pyspark dataframe

like in pandas I usually do df['columnname'].unique()

Question About Access and Filter Data in Databricks

Hi guys, im new using databricks and i have a challenge in my new work. routerlogin I need to access to one the database (the database is on DBFS) result of some ETLS trough any service, can be ODBC or by some API. I need to connect there because I...

Multiple Clusters stuck in pending state during creation.

I have been trying to create a new cluster to use and multiple attempts have gotten stuck in pending: "Finding instances for new nodes, acquiring more instances if necessary" until they time out. Up to today I have had no problems creating clusters ...

How do I make approxSimilarityJoin work on 25k 300-d vectors?

I’m trying to use LSH approxSimilarityJoin on a dataset with ~25k 300-d vectors of floats. It gets stuck and eventually fails with ’Slave lost’ error. The size of cluster and memory are likely not a problem, the failure happens even with 16 nodes, 1...