cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Merchiv
by New Contributor III
  • 7702 Views
  • 8 replies
  • 2 kudos

Resolved! AnalysisException when running SQL queries

When running some SQL queries using spark.sql(...), we sometimes get a variant of the following error:AnalysisException: Undefined function: current_timestamp. This function is neither a built-in/temporary function, nor a persistent function that is ...

  • 7702 Views
  • 8 replies
  • 2 kudos
Latest Reply
ashish1
New Contributor III
  • 2 kudos

This is most likely a conflict in the lib code, you can uninstall some libs on your cluster and try to narrow it down to the problematic one.

  • 2 kudos
7 More Replies
Phani1
by Valued Contributor
  • 1797 Views
  • 2 replies
  • 1 kudos

Integration Dolly with Databricks

Hi Databricks Team,Could you please share any links /docs/Sample notebooks to integrate Dolly with Databricks, our aim is to generate SQL queries based on the free text and execute it via databricks cluster/SQL warehouse.

  • 1797 Views
  • 2 replies
  • 1 kudos
Latest Reply
sean_owen
Honored Contributor II
  • 1 kudos

https://www.dbdemos.ai/demo.html?demoName=llm-dolly-chatbot is a good demonstration of Dolly (or really any LLM) for question answering. LLMs like this are not for SQL generation, but other LLMs are, like starcoderbase

  • 1 kudos
1 More Replies
deficiant_codge
by Contributor II
  • 4367 Views
  • 10 replies
  • 11 kudos

Resolved! Can't able to run following queries

I cannot able to run following queriesALTER TABLE iot_events ADD ATTRIBUTE pii ON emailALTER TABLE users ADD ATTRIBUTE pii ON phoneGRANT SELECT ON DATABASE iot_data HAVING ATTRIBUTE NOT IN (pii) TO product_managersand GRANT SELECT ON iot_events TO ...

  • 4367 Views
  • 10 replies
  • 11 kudos
Latest Reply
karthik_p
Esteemed Contributor
  • 11 kudos

@Kaniz Fatma​ can anyone from data Bircks help on why attribute-based access control function is not working in unity catalog @Rahul Mishra​ below commands ALTER TABLE iot_events ADD ATTRIBUTE pii ON emailALTER TABLE users ADD ATTRIBUTE pii ON phoneG...

  • 11 kudos
9 More Replies
Hubert-Dudek
by Esteemed Contributor III
  • 763 Views
  • 1 replies
  • 5 kudos

Starting from #databricks runtime 12.2 LTS, implicit lateral column aliasing is now supported. This feature enables you to reuse an expression defined...

Starting from #databricks runtime 12.2 LTS, implicit lateral column aliasing is now supported. This feature enables you to reuse an expression defined earlier in the same SELECT list, thus avoiding repetition of the same calculation.For instance, in ...

ezgif-3-d3fac0139c
  • 763 Views
  • 1 replies
  • 5 kudos
Latest Reply
Anonymous
Not applicable
  • 5 kudos

Thanks for sharing this with the Databricks community.

  • 5 kudos
Hubert-Dudek
by Esteemed Contributor III
  • 1424 Views
  • 3 replies
  • 7 kudos

Starting from #databricks runtime 12.2 LTS, implicit lateral column aliasing is now supported. This feature enables you to reuse an expression defined...

Starting from #databricks runtime 12.2 LTS, implicit lateral column aliasing is now supported. This feature enables you to reuse an expression defined earlier in the same SELECT list, thus avoiding repetition of the same calculation.For instance, in ...

ezgif-3-d3fac0139c
  • 1424 Views
  • 3 replies
  • 7 kudos
Latest Reply
Ajay-Pandey
Esteemed Contributor III
  • 7 kudos

Informative Thanks for sharing.

  • 7 kudos
2 More Replies
EmilioGC
by New Contributor III
  • 3626 Views
  • 5 replies
  • 7 kudos

Resolved! Why was SQL formatting removed inside spark.sql functions? Now it looks like a plain string.

Previously we were able to see SQL queries inside spark.sql() like this:But now it just looks like a plain string: I know it's not a big issue, but it's still annoying to have to code in SQL while having it all be blue, it makes debugging more cumber...

old format new format
  • 3626 Views
  • 5 replies
  • 7 kudos
Latest Reply
jose_gonzalez
Moderator
  • 7 kudos

Hi @Emilio Garza​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.

  • 7 kudos
4 More Replies
StanleyTang
by New Contributor III
  • 1672 Views
  • 4 replies
  • 4 kudos

How to run SQL queries from services when data migrated from SQL server to data lake?

Currently our service provides an API to serve the purchase records. The purchase records are stored in SQL database. To simplify, when users want to get their recent purchase records, they make an API call. The API call will run a SQL query on the D...

  • 1672 Views
  • 4 replies
  • 4 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 4 kudos

Hi @Stanley Tang​​​, We haven’t heard from you since the last response from @Debayan Mukherjee​ , and I was checking back to see if his suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful...

  • 4 kudos
3 More Replies
KrishZ
by Contributor
  • 1508 Views
  • 4 replies
  • 0 kudos

How to prevent sql queries in 2 notebooks from reading the same row from a Table ?

I have an SQL query to select and update rows in a table. I do this in batches of 300 rows (select 300 , update the selected 300 , select new 300 and update the newly selected and so on..) I run this query in 2 different notebooks concurrently to spe...

  • 1508 Views
  • 4 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Krishna Zanwar​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 0 kudos
3 More Replies
Ank
by New Contributor II
  • 816 Views
  • 2 replies
  • 2 kudos

Why am I getting a FileNotFoundError after providing the file path?

I used copy file path to get the file path of the notebook I am trying to run from another notebook.file_path = "/Users/ankur.lohiya@workday.com/PAS/Training/Ingest/TrainingQueries-Cloned.py/"ddi = DatabricksDataIngestion(file_path=file_path,        ...

  • 816 Views
  • 2 replies
  • 2 kudos
Latest Reply
Vidula
Honored Contributor
  • 2 kudos

Hello @Ankur Lohiya​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...

  • 2 kudos
1 More Replies
AlexDavies
by Contributor
  • 5246 Views
  • 12 replies
  • 3 kudos

Resolved! Report on SQL queries that are being executed

We have a SQL workspace with a cluster running that services a number of self service reports against a range of datasets. We want to be able to analyse and report on the queries our self service users are executing so we can get better visibility of...

  • 5246 Views
  • 12 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hey there @Alex Davies​ Hope you are doing great. Just checking in if you were able to resolve your issue or do you need more help? We'd love to hear from you.Thanks!

  • 3 kudos
11 More Replies
enichante
by New Contributor
  • 2029 Views
  • 4 replies
  • 5 kudos

Resolved! Databricks: Report on SQL queries that are being executed

We have a SQL workspace with a cluster running that services a number of self service reports against a range of datasets. We want to be able to analyse and report on the queries our self service users are executing so we can get better visibility of...

  • 2029 Views
  • 4 replies
  • 5 kudos
Latest Reply
Anonymous
Not applicable
  • 5 kudos

Looks like the people have spoken: API is your best option! (thanks @Werner Stinckens​  @Chris Grabiel​  and @Bilal Aslam​ !) @eni chante​ Let us know if you have questions about the API! If not, please mark one of the replies above as the "best answ...

  • 5 kudos
3 More Replies
Labels