- 3450 Views
- 4 replies
- 0 kudos
Correlated column is not allowed in non predicate in UDF SQL
Hi Team,I am new to databricks and currently working on creating sql udf 's in databricks .In udf we are calculating min date and that date column using in where clause also.While running udf getting Correlated column is not allowed in non predica...
- 3450 Views
- 4 replies
- 0 kudos
- 0 kudos
Could you please provide your full code? I would also like to know which DBR version you are using in your cluster.
- 0 kudos
- 8189 Views
- 3 replies
- 6 kudos
Bug? Unity Catalog incompatible with Sparklyr in RStudio (on Driver) and as well if used on one cluster from multiple notebooks?
If I start a RStudio Server with in cluster init script as described here in a Unity Catalog Cluster the sparklyr connection fails with an error about a missing Credential Scope.=LI tried it both in 11.3LTS and 12.0 Beta. I tried it only in a Persona...
- 8189 Views
- 3 replies
- 6 kudos
- 6 kudos
Have run into this issue as well. Let me know if there was any resolution
- 6 kudos
- 4163 Views
- 3 replies
- 6 kudos
How can I customize the Notebook Job # while using dbutils.notebook.run method?
When running multiple notebooks parallelly using dbutils.notebook.run from a parent notebook, an url to that running notebook is printed, like belowNotebook job #211371132480519Is there a way I can print the notebook name or some customized string in...
- 4163 Views
- 3 replies
- 6 kudos
- 6 kudos
Hi @Debayan Thank you for your reply.However, the answer I am looking for is : how to print/get a more meaningful name of the jobs when running multiple notebooks parallelly using dbutils.notebook.run from a parent notebook.Now in the parent notebook...
- 6 kudos
- 4460 Views
- 4 replies
- 1 kudos
Resolved! RDD not picking up spark configuration for azure storage account access
I want to open some CSV files as an RDD, do some processing and then load it as a DataFrame. Since the files are stored in an Azure blob storage account I need to configure the access accordingly, which for some reason does not work when using an RDD...
- 4460 Views
- 4 replies
- 1 kudos
- 1 kudos
I decided to load the files into a DataFrame with a single column and then do the processing before splitting it into separate columns and this works just fine.@Hyper Guy​ thanks for the link, I didn't try that but it seems like it would resolve the ...
- 1 kudos
- 3558 Views
- 0 replies
- 0 kudos
I am trying to use SQL join to combine 3 tables but the execution does not go beyond 93 million rows
Hi all,I ran a code to join 3 tables in Azure Databricks using SQL. When I run the code it is indicated "93 million rows read (1GB). It will be showing me " and does not go beyond this. Who knows what the issue could be?
- 3558 Views
- 0 replies
- 0 kudos
- 6700 Views
- 3 replies
- 2 kudos
TypeError : withcolumn() takes 3 positional arguments but 4 were given.
Hi All,Can some one please help me with the error.This is my small python code.binmaster = binmasterdf.withColumnRenamed("tag_number","BinKey")\.withColumn ("Category", when (length("Bin")==4,'SMALL LOT'),(length("Bin")==7,'RACKING'))TypeError : with...
- 6700 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @JohnJustus If you see closely in .withColumn ("Category", when (length("Bin")==4,'SMALL LOT'), when (length("Bin")==7,'RACKING'), otherwise('FLOOR')), withcolumn would take 2 parameters. The first parameter as a string and the second as the colum...
- 2 kudos
- 12962 Views
- 3 replies
- 4 kudos
Liquid clustering with structured streaming pyspark
I would like to try out liquid clustering, but all the examples I see seem to be SQL tables created from selecting from other tables. Our gold tables are pyspark tables written directly to a table, e.g. like this: silver_df.writeStream.partitionBy(["...
- 12962 Views
- 3 replies
- 4 kudos
- 4 kudos
I did not find anything in the docs either. I suppose a pyspark version will come in the future?
- 4 kudos
- 3121 Views
- 0 replies
- 0 kudos
Updating Table Schema: Renaming and Dropping Columns
Are renaming and dropping columns Databricks proprietary methods?How does it work under the hood, and does enabling the feature render lazy loading ineffective?Ref: Do Delta Lake and Parquet Share Partition Strategy?
- 3121 Views
- 0 replies
- 0 kudos
- 5102 Views
- 4 replies
- 2 kudos
Resolved! Call python image function in pyspark
I have a function for rotating images written in python:from PIL import Image def rotate_image(image, rotation_angle): im = Image.open(image) out = im.rotate(rotation_angle, expand = True) return outI now want to use this function as a pyspark ...
- 5102 Views
- 4 replies
- 2 kudos
- 2 kudos
Stock photos, I've come to realize, are the catalysts of imagination. This website's vast reservoir of images new york seal sparks ideas that ripple through my projects. They empower me to envision the previously unimagined, helping me breathe life i...
- 2 kudos
- 1039 Views
- 0 replies
- 0 kudos
How to get job launch type in notebook
I want to get job launched status in notebook if it is launched by scheduler or manuallyI tried using JobTriggerType property of notebook context but it gives only manual and repair but not by scheduleddbutils.notebook.entry_point.getDbutils().notebo...
- 1039 Views
- 0 replies
- 0 kudos
- 4764 Views
- 3 replies
- 0 kudos
What is my subscription plan?
I am working as data engineer I was about checking the subscription plan. I would like to know how I can check the subscription plan. I am "Admin" but I cannot "Manage accounts" on Databricks workspace portal.This subscription information is pretty i...
- 4764 Views
- 3 replies
- 0 kudos
- 0 kudos
Hey, not really. Update you: if you try to add ACL to a secret scope then you will be sure that your subscription is Enterprise or Standard, because either you succeed then you are working with Enterprise or the API respond with the Standard subscrip...
- 0 kudos
- 1892 Views
- 2 replies
- 0 kudos
Trying to vacuum a table that is constantly being "createdOrReplaced"
and it seems that older data (From the "replaced" table) isn't being removed, long after the retention period I'd be glad for clues on how to handle this
- 1892 Views
- 2 replies
- 0 kudos
- 0 kudos
Thanks, I know that, but the table history shows 30 days, but the actual data size and number of files and all other indicators , correlate to 170 days.
- 0 kudos
- 7645 Views
- 0 replies
- 0 kudos
Can't "copy into" new data to delta table after truncate old data
Like title, i have a delta table with data of 22/9 and today i wanna remove old data and add new data of 23/9, i used 'truncate' and 'copy into' query but after 'truncate', nothing is added to table, what's happened with my table, file of old data st...
- 7645 Views
- 0 replies
- 0 kudos
- 8790 Views
- 3 replies
- 2 kudos
Resolved! DLT pipeline
Hi all!I have a question about setting a target schema. How to set different targets for 2 different tables in the same delta live table pipeline. We have 2 target schemas in a database Bronze_chema and silver_schema. The pipeline has a streaming ra...
- 8790 Views
- 3 replies
- 2 kudos
- 20866 Views
- 9 replies
- 4 kudos
Kafka unable to read client.keystore.jks.
Below is the error we have received when trying to read the stream Caused by: kafkashaded.org.apache.kafka.common.KafkaException: Failed to load SSL keystore /dbfs/FileStore/Certs/client.keystore.jksCaused by: java.nio.file.NoSuchFileException: /dbfs...
- 20866 Views
- 9 replies
- 4 kudos
- 4 kudos
Ok, scrub that - the problem in my case was that I was using the 14.0 databricks runtime, which appears to have a bug relating to abfss paths here. Switching back to the 13.3 LTS release resolved it for me. So if you're in the same boat finding abfss...
- 4 kudos
| User | Count |
|---|---|
| 1616 | |
| 790 | |
| 466 | |
| 349 | |
| 287 |
