- 10342 Views
- 2 replies
- 0 kudos
How to pass column names in selectExpr through one or more string parameters in spark using scala?
I am using script for CDC Merge in spark streaming. I wish to pass column values in selectExpr through a parameter as column names for each table would change. When I pass the columns and struct field through a string variable, I am getting error as...
- 10342 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Swapan Swapandeep Marwaha, Can you pass them as a Seq as in below code, keyCols = Seq("col1", "col2"), structCols = Seq("struct(offset,KAFKA_TS) as otherCols")
- 0 kudos
- 6794 Views
- 1 replies
- 0 kudos
Making HTTP post requests on Spark using foreachPartition
Need some help to understand the behaviour of the below in Spark (using Scala and Databricks) I have some dataframe (reading from S3 if that matters), and would send that data by making HTTP post requests in batches of 1000 (at most). So I reparti...
- 6794 Views
- 1 replies
- 0 kudos
- 0 kudos
Need some help to understand the behaviour of the below in Spark (using Scala and Databricks) I have some dataframe (reading from S3 if that matters), and would send that data by making HTTP post requests in batches of 1000 (at most). So I repa...
- 0 kudos
- 22845 Views
- 2 replies
- 0 kudos
Python spark.read.text Path does not exist
Dear all, I want to read files with python from a storage account. I followed this instruction https://docs.microsoft.com/en-us/azure/azure-databricks/store-secrets-azure-key-vault. This is my python code: dbutils.fs.mount(source = "wasbs://contain...
- 22845 Views
- 2 replies
- 0 kudos
- 0 kudos
@rba76​ Make sure helloworld.txt file exists in the container1 folderI'm able to view the text file using the same commands as follows:Mount Blob Storage:dbutils.fs.mount( source = "wasbs://sampledata@azure.blob.core.windows.net/Azure", mount_po...
- 0 kudos
- 19622 Views
- 6 replies
- 0 kudos
Cannot Convert Column to Bool Error - When Converting dataframe column which is in string to date type in python
Hi All, I am trying to convert a dataframe column which is in the format of string to date type format yyyy-MM-DD? I have written a sql query and stored it in dataframe. df3 = sqlContext.sql(sqlString2) df3.withColumn(df3['CalDay'],pd.to_datetime(df...
- 19622 Views
- 6 replies
- 0 kudos
- 0 kudos
Registered to post this so forgive the formatting nightmare This is a python databricks script function that allows you to convert from string to datetime or date and utilising coalescefrom pyspark.sql.functions import coalesce, to_date def to_dat...
- 0 kudos
- 17672 Views
- 1 replies
- 0 kudos
How can I add jars ("spark.jars") to pyspark notebook?
I want to add a few custom jars to the spark conf. Typically they would be submitted along with the spark-submit command but in Databricks notebook, the spark session is already initialized. So, I want to set the jars in "spark.jars" property in the...
- 17672 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @dbansal, Install the libraries/jars while initialising the cluster.Please go through the documentation on the same below,https://docs.databricks.com/libraries.html#upload-a-jar-python-egg-or-python-wheel
- 0 kudos
- 10605 Views
- 1 replies
- 0 kudos
List all files in a Blob Container
I am trying to find a way to list all files, and related file sizes, in all folders and all sub folders. I guess these are called blobs, in the Databricks world. Anyway, I can easily list all files, and related file sizes, in one single folder, but ...
- 10605 Views
- 1 replies
- 0 kudos
- 0 kudos
from azure.storage.blob import BlockBlobService block_blob_service = BlockBlobService(account_name='your_acct_name', account_key='your_acct_key') mylist = [] generator = block_blob_service.list_blobs('rawdata') for blob in generator: mylist.append(...
- 0 kudos
- 98034 Views
- 11 replies
- 6 kudos
Resolved! How do I get the current cluster id?
I am adding Application Insights telemetry to my Databricks jobs and would like to include the cluster ID of the job run. How can I access the cluster id at run time? The requirement is that my job can programmatically retrieve the cluster id to in...
- 98034 Views
- 11 replies
- 6 kudos
- 8035 Views
- 1 replies
- 0 kudos
Sort within a groupBy with dataframe
Using Spark DataFrame, eg. myDf .filter(col("timestamp").gt(15000)) .groupBy("groupingKey") .agg(collect_list("aDoubleValue")) I want the collect_list to return the result, but ordered according to "timestamp". i.a. I want the GroupBy results...
- 8035 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Laurent Thiebaud,Please use the below format to sort within a groupby, import org.apache.spark.sql.functions._ df.groupBy("columnA").agg(sort_array(collect_list("columnB")))
- 0 kudos
- 20826 Views
- 1 replies
- 0 kudos
Resolved! Length Value of a column in pyspark
Hello, i am using pyspark 2.12 After Creating Dataframe can we measure the length value for each row. For Example: I am measuring length of a value in column 2 Input file |TYCO|1303| |EMC |120989| |VOLVO|102329| |BMW|130157| |FORD|004| Output ...
- 20826 Views
- 1 replies
- 0 kudos
- 0 kudos
You can use the length function for this from pyspark.sql.functions import length mock_data = [('TYCO', '1303'),('EMC', '120989'), ('VOLVO', '102329'),('BMW', '130157'),('FORD', '004')] df = spark.createDataFrame(mock_data, ['col1', 'col2']) df2 = d...
- 0 kudos
- 26461 Views
- 4 replies
- 0 kudos
Removing non-ascii and special character in pyspark
i am running spark 2.4.4 with python 2.7 and IDE is pycharm. The Input file (.csv) contain encoded value in some column like given below. File data looks COL1,COL2,COL3,COL4 CM, 503004, (d$όνυ$F|'.h*Λ!ψμ=(.ξ; ,.ʽ|!3-2-704 The output i am trying ...
- 26461 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Rohini Mathur, use below code on column containing non-ascii and special characters.df['column_name'].str.encode('ascii', 'ignore').str.decode('ascii')
- 0 kudos
- 10712 Views
- 5 replies
- 0 kudos
How to create a dataframe with the files from S3 bucket
I have connected my S3 bucket from databricks. Using the following command : import urllib import urllib.parse ACCESS_KEY = "Test" SECRET_KEY = "Test" ENCODED_SECRET_KEY = urllib.parse.quote(SECRET_KEY, "") AWS_BUCKET_NAME = "Test" MOUNT_NAME = "...
- 10712 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @akj2784,Please go through Databricks documentation on working with files in S3,https://docs.databricks.com/spark/latest/data-sources/aws/amazon-s3.html#mount-s3-buckets-with-dbfs
- 0 kudos
- 10765 Views
- 6 replies
- 0 kudos
Convert String to Timestamp
I have a dataset with one column of string type ('2014/12/31 18:00:36'). How can I convert it to timastamp type with PySpark?
- 10765 Views
- 6 replies
- 0 kudos
- 0 kudos
hope you dont mind if i ask you to elaborate further for a shaper understanding? see my basketball court layout at https://www.recreationtipsy.com/basketball-court/
- 0 kudos
- 17688 Views
- 1 replies
- 0 kudos
ConnectException error
I'm using PySpark on Databricks and trying to pivot a 27753444 X 3 matrix. If I do it in Spark DataFrame: df = df.groupBy("A").pivot("B").avg("C") it takes forever (after 2 hours and I canceled it). If I convert it to pandas dataframe and then pivo...
- 17688 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Raymond_Hu,This means that the driver crashed because of an OOM (Out of memory) exception and after that, it's not able to establish a new connection with the driver. Please try below optionsTry increasing driver-side memory and then retry.You ca...
- 0 kudos
- 16931 Views
- 10 replies
- 1 kudos
Spark Running Really slow - help required
My company urgently needs help, we are having severe performance problems with spark and are having to switch to a different solution if we don't get to the bottom of it. We are on 1.3.1, using spark SQL, ORC Files with partitions and caching in me...
- 16931 Views
- 10 replies
- 1 kudos
- 1 kudos
In my project, following solutions were launched one-by-one to improve performance To store middle-level result, use memory cache instead of HDFS (like: Ignite Cache) Only use spark for complicated data aggregation, to simple result, just do it on d...
- 1 kudos
- 19625 Views
- 15 replies
- 0 kudos
Resolved! Can we pass Databricks output to Azure function body?
Hi, Can anyone help me with Databricks and Azure function. I'm trying to pass databricks json output to azure function body in ADF job, is it possible? If yes, How? If No, what other alternative to do the same?
- 19625 Views
- 15 replies
- 0 kudos
- 0 kudos
You can now pass values back to ADF from a notebook.@@Yogi​ Though there is a size limit, so if you are passing dataset of larger than 2MB then rather write it on storage, and consume it directly with Azure Functions. You can pass the file path/ refe...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
