topic Re: uuid in Merge in Data Engineering

uuid in Merge

William_Scardua — Thu, 06 Oct 2022 20:58:29 GMT

Hi guys,

I'm trying to use uuid in the merge but I always get an error...

import uuid
 
(
  df_events.alias("events").merge(
    source = df_updates.alias("updates"),
    condition = "events.cod = updates.cod and events.num = updates.num"
  ).whenMatchedUpdate(
    set =
      {
      "events.cod" : "updates.cod",
      "events.num_c" : "updates.num_contrato",
      "events.status": lit("Updated")"
      }
  ).whenNotMatchedInsert(values =
    {
      "events.id" : uuid.uuid4(),
      "events.cod" : "updates.cod",
      "events.num" : "updates.num",
      "events.status" : lit("Inserted")
    }
  )
  .execute()
)

have any tips ?

Re: uuid in Merge

User16752242622 — Fri, 07 Oct 2022 07:25:21 GMT

Hi @William Scardua

Could you please more details? Which DBR version you are currently using?

Please share the error stack trace or snapshot with us.

Is it while importing the UUID module?

Re: uuid in Merge

William_Scardua — Fri, 07 Oct 2022 11:14:49 GMT

Hi @Akash Bhat

My cluster version is: 9.1 LTS (includes Apache Spark 3.1.2, Scala 2.12)

This error occurs when I run the cell

TypeError: Values of dict in 'values' in whenNotMatchedInsert must contain only Spark SQL Columns or strings (expressions in SQL syntax) as values, found '202d282c-045a-402c-895f-832c4c3a5190' of type '<class 'uuid.UUID'>'

Re: uuid in Merge

Sivaprasad1 — Fri, 07 Oct 2022 18:32:05 GMT

@William Scardua :

Could you please refer to https://stackoverflow.com/questions/15859156/python-how-to-convert-a-valid-uuid-from-string-to-uuid

might be helpful

Re: uuid in Merge

William_Scardua — Fri, 07 Oct 2022 21:26:08 GMT

Hi @Sivaprasad C S I tried to convert with uuid4().hex see the error it reported

  ).whenNotMatchedInsert(values =
    {
      "events.id" : uuid.uuid4().hex,
      "events.cod_operadora" : "updates.cod_operadora",
      "events.num_contrato" : "updates.num_contrato",
      "events.qtd_residencia_ok": "updates.qtd_residencia_ok",
      "events.data_atualiz": lit("Inclusao")
    }

AnalysisException: cannot resolve `b6ff50957b0f492ca082e10227f07638` in INSERT clause given columns updates.`id` ...

Re: uuid in Merge

Hubert-Dudek — Fri, 14 Oct 2022 12:48:17 GMT

wrap it in lit 🙂 As it is looking for variable or field equal to uuid

lit(uuid.uuid4().hex)

Re: uuid in Merge

William_Scardua — Fri, 21 Oct 2022 22:10:08 GMT

Hi @Hubert Dudek

in this case UUID are repeted

Re: uuid in Merge

Anonymous — Thu, 17 Nov 2022 03:36:15 GMT

Hi @William Scardua

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!