- 1169 Views
- 1 replies
- 1 kudos
Streaming Data Modeling Normalization with Databricks Delta Live Tables
Streamline Data Modeling Normalization with Databricks Delta Live Tables in Just a Few Steps:- Use the "Apply changes" function to populate tables with slowly changing dimensions using auto-increment IDs.- Register SQL mapping functions to associate ...
- 1169 Views
- 1 replies
- 1 kudos
- 1 kudos
Thank you for sharing this @Hubert-Dudek !!!
- 1 kudos
- 1318 Views
- 0 replies
- 0 kudos
how to connect to aws elasticsearch of another account from databricks
Hi - I have tried my level best to go through both elasticsearch documentation as well as Databricks documentation to get an answer for my question - is it possible to connect to AWS elasticsearch of a different AWS account from Databricks? I did no...
- 1318 Views
- 0 replies
- 0 kudos
- 9791 Views
- 8 replies
- 2 kudos
Invisible empty spaces when reading .csv files
When importing a .csv file with leading and/or trailing empty spaces around the separators, the output results in strings that appear to be trimmed on the output table or when using .display() but are not actually trimmed.It is possible to identify t...
- 9791 Views
- 8 replies
- 2 kudos
- 12198 Views
- 11 replies
- 2 kudos
Resolved! Problems connecting Simba ODBC with a M1 Macbook Pro
Hi,There's a way to make work the Simba ODBC Driver for M1 Macbook Pros?I find myself able to run on an old intel version of Macbook easily, but now every time I even test the connection with the iODBC Manager fails.Definitely, the issue is around no...
- 12198 Views
- 11 replies
- 2 kudos
- 2 kudos
Things seem to be mostly working for me now. I've added a bit more detail on my connection steps and process in case it's helpful for anyone on Stack Overflow: https://stackoverflow.com/questions/76407426/connecting-rstudio-desktop-to-databricks-comm...
- 2 kudos
- 1566 Views
- 0 replies
- 0 kudos
Remove WHERE 1=0
I am hoping someone can help me remove the WHERE 1=0 that is constantly getting added onto the end of my Query (see below). Please let me know if I can provide more info here.This is running a notebook, in Azure Databricks against a cluster that has...
- 1566 Views
- 0 replies
- 0 kudos
- 4227 Views
- 5 replies
- 1 kudos
com.databricks.spark.safespark.UDFException: UNAVAILABLE: Channel shutdownNow invoked
Trying to determine a root cause of UDFException that occurs when returning a variable length ArrayType. If I hardcode the data returned from the UDF to a fixed length, say 19, the error does not occur. Setup codesplit_runs_UDF = udf(split_runs_udf, ...
- 4227 Views
- 5 replies
- 1 kudos
- 1 kudos
After further investigation, It reproduces slightly differently on single user mode.Single user mode: runs foreverShared: gives the above messageI've determined that there was a corner case in the dataset which lead to UDF never returning. I am am as...
- 1 kudos
- 2189 Views
- 0 replies
- 0 kudos
How to retrieve cluster IDs of a deleted All Purpose cluster ?
I need to retrieve the event logs of deleted All Purpose clusters of a certain workspace.databricks list API ({workspace_url}/api/2.0/clusters/list) provides me with the list of all active/terminated clusters but not the clusters that are deleted. I ...
- 2189 Views
- 0 replies
- 0 kudos
- 2491 Views
- 2 replies
- 1 kudos
DLT current channel uses same runtime as the preview channel
Hi,According to the latest release notes, the current channel of DLT should be using Databricks runtime 11.3 and the preview channel should be using 12.2. The current channel was using correct runtime version 11.3 still yesterday morning, but since ...
- 2491 Views
- 2 replies
- 1 kudos
- 1 kudos
I'm seeing the same issue with 12 current / 13 preview. Updating the channel didn't bump the runtime version and even creating a pipeline with the preview channel uses the current version.
- 1 kudos
- 2783 Views
- 4 replies
- 0 kudos
Correlated column is not allowed in non predicate in UDF SQL
Hi Team,I am new to databricks and currently working on creating sql udf 's in databricks .In udf we are calculating min date and that date column using in where clause also.While running udf getting Correlated column is not allowed in non predica...
- 2783 Views
- 4 replies
- 0 kudos
- 0 kudos
Could you please provide your full code? I would also like to know which DBR version you are using in your cluster.
- 0 kudos
- 6973 Views
- 3 replies
- 6 kudos
Bug? Unity Catalog incompatible with Sparklyr in RStudio (on Driver) and as well if used on one cluster from multiple notebooks?
If I start a RStudio Server with in cluster init script as described here in a Unity Catalog Cluster the sparklyr connection fails with an error about a missing Credential Scope.=LI tried it both in 11.3LTS and 12.0 Beta. I tried it only in a Persona...
- 6973 Views
- 3 replies
- 6 kudos
- 6 kudos
Have run into this issue as well. Let me know if there was any resolution
- 6 kudos
- 3671 Views
- 3 replies
- 6 kudos
How can I customize the Notebook Job # while using dbutils.notebook.run method?
When running multiple notebooks parallelly using dbutils.notebook.run from a parent notebook, an url to that running notebook is printed, like belowNotebook job #211371132480519Is there a way I can print the notebook name or some customized string in...
- 3671 Views
- 3 replies
- 6 kudos
- 6 kudos
Hi @Debayan Thank you for your reply.However, the answer I am looking for is : how to print/get a more meaningful name of the jobs when running multiple notebooks parallelly using dbutils.notebook.run from a parent notebook.Now in the parent notebook...
- 6 kudos
- 3653 Views
- 4 replies
- 1 kudos
Resolved! RDD not picking up spark configuration for azure storage account access
I want to open some CSV files as an RDD, do some processing and then load it as a DataFrame. Since the files are stored in an Azure blob storage account I need to configure the access accordingly, which for some reason does not work when using an RDD...
- 3653 Views
- 4 replies
- 1 kudos
- 1 kudos
I decided to load the files into a DataFrame with a single column and then do the processing before splitting it into separate columns and this works just fine.@Hyper Guy​ thanks for the link, I didn't try that but it seems like it would resolve the ...
- 1 kudos
- 3061 Views
- 0 replies
- 0 kudos
I am trying to use SQL join to combine 3 tables but the execution does not go beyond 93 million rows
Hi all,I ran a code to join 3 tables in Azure Databricks using SQL. When I run the code it is indicated "93 million rows read (1GB). It will be showing me " and does not go beyond this. Who knows what the issue could be?
- 3061 Views
- 0 replies
- 0 kudos
- 5834 Views
- 3 replies
- 2 kudos
TypeError : withcolumn() takes 3 positional arguments but 4 were given.
Hi All,Can some one please help me with the error.This is my small python code.binmaster = binmasterdf.withColumnRenamed("tag_number","BinKey")\.withColumn ("Category", when (length("Bin")==4,'SMALL LOT'),(length("Bin")==7,'RACKING'))TypeError : with...
- 5834 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @JohnJustus If you see closely in .withColumn ("Category", when (length("Bin")==4,'SMALL LOT'), when (length("Bin")==7,'RACKING'), otherwise('FLOOR')), withcolumn would take 2 parameters. The first parameter as a string and the second as the colum...
- 2 kudos
- 11964 Views
- 3 replies
- 4 kudos
Liquid clustering with structured streaming pyspark
I would like to try out liquid clustering, but all the examples I see seem to be SQL tables created from selecting from other tables. Our gold tables are pyspark tables written directly to a table, e.g. like this: silver_df.writeStream.partitionBy(["...
- 11964 Views
- 3 replies
- 4 kudos
- 4 kudos
I did not find anything in the docs either. I suppose a pyspark version will come in the future?
- 4 kudos
| User | Count |
|---|---|
| 1611 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
