- 5344 Views
- 3 replies
- 5 kudos
Resolved! Manipulate Column that is an array of objects
I have a column that is an array of objects, let's call it ARRAY, and now I would like to query / manipulate, the elements object without using explode function, this is an example, for each element in that column I would like to create a path. .wit...
- 5344 Views
- 3 replies
- 5 kudos
- 5 kudos
Hello I am working with Scala, and I used somehing similar:def play(col: Column): Column = { concat_ws("", lit(imagePath), lit("/"), col("field1"), lit("/"), col("field2"), lit(".ext"))}val variable = spark.lot_of_stuff. .withColumn("...
- 5 kudos
- 7985 Views
- 4 replies
- 4 kudos
Resolved! I am trying to connect to Databricks SQL using this JDBC driver. com.simba.spark.jdbc.Driver. I can download it and install it on my local laptop , but I need it to be available to maven repository so I can include the maven path in my project.
Please lett me know if the JDB driver is available in maven repository and what needs to be included in pom.xml to download it.
- 7985 Views
- 4 replies
- 4 kudos
- 4 kudos
@Vamsee krishna kanth Arcot​ good news, the driver is up on Maven: https://search.maven.org/artifact/com.databricks/databricks-jdbc
- 4 kudos
- 4803 Views
- 2 replies
- 1 kudos
Resolved! Declare a constant value in a Databricks noteboook to be used in futher SQL code
I'm new to databricks and try to migrate some former SQL code used so far, in a DB notebook.I have some constants declared and I can't find the right way to similarly declare those in my notebook.I tried %sql DECLARE label_language CONSTANT VARCHAR(2...
- 4803 Views
- 2 replies
- 1 kudos
- 1 kudos
Thank you @Hubert Dudek​ , gave it a try and it does the job, both in SQL and in Python btw. When the time will come to industrialize this, I'll have to figure out how to create/use/deal with some configuration files (json, yaml or so). But for explo...
- 1 kudos
- 8579 Views
- 2 replies
- 1 kudos
Resolved! net.ucanaccess.jdbc.UcanaccessSQLException: UCAExc:::5.0.1 user lacks privilege or object not found: full questionnaire in statement [SELECT * FROM "full questionnaire" WHERE 1=0]
I have followed the steps given here to parse .accdb files using ucanaccess on Azure Databricks, however, I receive errors.See below my code:# Connection properties conn_properties = {"driver" : "net.ucanaccess.jdbc.UcanaccessDriver"} # Path url = ...
![net.ucanaccess.jdbc.UcanaccessSQLException_ UCAExc___5.0.1 user lacks privilege or object not found_ full questionnaire in statement [SELECT _ FROM "full questionnaire" WHERE 1=0]](/t5/image/serverpage/image-id/1952iDBC96B99046D53B6/image-size/medium?v=v2&px=400)
- 8579 Views
- 2 replies
- 1 kudos
- 1 kudos
@Valeri Nteleah​ please let me know if you need further help on this.Thanks
- 1 kudos
- 4085 Views
- 2 replies
- 3 kudos
User Initiated Cluster Termination, after 1-2 hours cluster goes offline even while working!
Hi,Wanted to ask, I have been experiencing databricks_error = user initiated cluster terminationdatabricks_error_message: timeout while placing nodesand soetimes it stays stuck on "finding new nodes, instances if neccessary" for hours..and it goes of...
- 4085 Views
- 2 replies
- 3 kudos
- 3 kudos
We have identified that the recent deployment done on the backend service on which Azure Data factory relies upon had prevented the data flow activity from running and this had caused the issue But can you confirm @Ghazanfar Uruj​ if the issue persis...
- 3 kudos
- 26192 Views
- 1 replies
- 10 kudos
Resolved! How do temp views actually work?
So I'm querying data from parquet files that have a couple of billions records (table 1 or t1), and then have to filter and then join with other parquet files with another couple of billions records (t2). This takes quite a long time to run (like 10h...
- 26192 Views
- 1 replies
- 10 kudos
- 10 kudos
Your intuition about views is correct. Views are not materialized, so they are basically just a saved query. Every time you access a view it will have to be recomputed. This is certainly not ideal if it take a long time (like 10hrs) to materialize a ...
- 10 kudos
- 5710 Views
- 3 replies
- 4 kudos
Resolved! VNET injected Databricks cluster not able to mount - 403 error
I'm mounting a Storage Account to a Databricks cluster in Azure. All the resources are included in a VNET and a private and public subnet has been associated to the Databricks resource. Below I've attached the guide we use for mounting the ADLS G2 to...
- 5710 Views
- 3 replies
- 4 kudos
- 4 kudos
Hey there @Derrick Bakhuis​ Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too.Cheers!
- 4 kudos
- 1862 Views
- 0 replies
- 0 kudos
GCP Databricks Cluster Start issue - Free trail Account
I have GCP trial account and took a 14 days databricks free trial from GCP. I created workspace and trying to create cluster and start it but it keeps on rotating/pending state. As a initial step, I tried to increase quotas mentioned on the page but ...
- 1862 Views
- 0 replies
- 0 kudos
- 2417 Views
- 0 replies
- 0 kudos
Recommended Redshift-2-Delta Migration Path
Hello All!My team is previewing Databricks and are contemplating the steps to take to perform one-time migrations of datasets from Redshift to Delta. Based on our understandings of the tool, here are our initial thoughts:Export data from Redshift-2-S...
- 2417 Views
- 0 replies
- 0 kudos
- 11624 Views
- 1 replies
- 2 kudos
"Detected a data update", what changed?
In streaming flows I periodically get a "Detected a data update" error. This error generally seem to indicate that something has changed in the source table schema, but it's not immediately apparent what. In one case yesterday I pulled the source tab...
- 11624 Views
- 1 replies
- 2 kudos
- 2 kudos
@Kaniz Fatma​ , Thanks, that helps. I was assuming this warning indicated a schema evolution, and based on what you say it likely wasn't and I just have to turn on IgnoreChanges any time I have a stream from a table that receives updates/upserts.To b...
- 2 kudos
- 1460 Views
- 1 replies
- 2 kudos
Unable to create account in Databricks community Edition
Unable to login to the community edition as the verification email is not delivered.
- 1460 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi sraj, facing same issue. Is your issue resolved. Please guide me also. That will be of great help.
- 2 kudos
- 5283 Views
- 2 replies
- 2 kudos
Mult task - restart of the failed jobs
Hi Team I am using Multitask and I am trying to restart only the failed task but seems like I have to restart complete workflow again and again , is there any way or workaround
- 5283 Views
- 2 replies
- 2 kudos
- 2 kudos
One way that works is to go to your task definition, click advanced options, and set retry policy. The task will restart per those instructions. Does that work for you?
- 2 kudos
- 10542 Views
- 3 replies
- 0 kudos
Specifying a Managed Resource Group name
Azure Databricks provisions a Managed Resource Group when you create your Workspace. Is there a way to specify the name of the Managed Resource Group and its resources during creation?The defaults created by the Workspace violate my company's standar...
- 10542 Views
- 3 replies
- 0 kudos
- 0 kudos
Hey there @Robert Biddle​ Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too.Cheers!
- 0 kudos
- 6708 Views
- 3 replies
- 3 kudos
Resolved! Dealing with updates to a delta table being used as a streaming source
Hi AllI have a requirement to perform updates on a delta table that is the source for a streaming query.I would like to be able to update the table and have the stream continue to work while also not ending up with duplicates.From my research it se...
- 6708 Views
- 3 replies
- 3 kudos
- 3 kudos
Hey @Mathew Walters​ Hope you are doing great.Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 3 kudos
- 7923 Views
- 4 replies
- 0 kudos
Resolved! ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject
After the new release of numpy 1.22.0 on Dec 31st, Databricks failed with this error for my existing Databricks Notebook Version 10.1 and numpy 1.20.0Qn: Why did the earlier releases after 1.20.0 uptil 1.22.0 did not raise the same exception. ?
- 7923 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Janani Mohan​ Hope you are doing well.Just wanted to check in if you were able to resolve your issue or do you need more help? We'd love to hear from you.Thanks!
- 0 kudos
| User | Count |
|---|---|
| 1615 | |
| 779 | |
| 349 | |
| 287 | |
| 267 |
