Databricks Community

Apache spark Streaming

When was the last commit done on Spark Streaming

How can we join two pyspark dataframes side by side (without using join,equivalent to pd.concat() in pandas) ? I am trying to join two extremely large dataframes where each is of the order of 50 million.

My two dataframes look like new_df2_record1 and new_df2_record2 and the expected output dataframe I want is like new_df2: The code I have tried is the following: If I print the top 5 rows of new_df2, it gives the output as expected but I cannot pri...

Load an explicit schema from an external metadata.csv file or a json file for reading csv's into dataframe

Hi, I have a metadata csv file which contains column name, and datatype such as Colm1: INT Colm2: String. I can also get the same in a json format as shown: I can store this on ADLS. How can I convert this into a schema like: "Myschema" that I can ...

Not able to fetch data from Simba Spark Jdbc Driver

We are getting below error when we tried to set the date in preparedstatement using Simba Spark Jdbc Driver. Exception: Query execution failed: [Simba][SparkJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: org.apache.h...

PySpark pandas_udf slower than single thread

I used @pandas_udf write a function for speeding up the process(parsing xml file ) and then compare it's speed with single thread , Surprisingly , Using @pandas_udf is two times slower than single-thread code. And the number of xml files I need to p...

Repos file size limit - Is it possible to clone a specific branch into Repos?

We refactored our codebase into another branch of our existing repo and consolidated the files so that they should be useable within the Databricks Repos size/file limitations. However, even though the new branch is smaller, I am still getting an err...

Can we access cool storage with Databricks?

Notebook path can't be in DBFS?

Some of us are working with IDEs and trying to deploy notebooks (.py) files to dbfs. the problem I have noticed is when configuring jobs, those paths are not recognized.notebook_path: If I use this :dbfs:/artifacts/client-state-vector/0.0.0/bootstrap...

Is it possible that only a particular cluster have only access to a s3 bucket or folder in s3

Hi I want to set up a cluster and want to give access to that cluster to some user only those user on that particular cluster should have access to read and write from and to the bucket. that particular bucket is not mounted on the workspace.Is th...

Is the delta schema enforcement flexible?

 In the sense that, is it possible to only check for column names or column data types or will it always be both?

Do we have a way to assign a static IP to a cluster on Databricks

how do i delete files from the DBFS

I can't see where in the databricks UI that I can delete files that have been either uploaded or saved to the DBFS - how do I do this?

Am trying to use SQL, but createOrReplaceTempView("myDataView")​ fails

Am trying to use SQL, but createOrReplaceTempView("myDataView") fails. I can create and display a DataFrame fine... import pandas as pd df = pd.DataFrame(['$3,000,000.00','$3,000.00', '$200.5', '$5.5'], columns = ['Amount']) df I add another cell, ...