cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
AlexDavies
by • Contributor
  • 8386 Views
  • 9 replies
  • 2 kudos

Report on SQL queries that are being executed

We have a SQL workspace with a cluster running that services a number of self service reports against a range of datasets. We want to be able to analyse and report on the queries our self service users are executing so we can get better visibility of...

Data Engineering
Reply
  • 8386 Views
  • 9 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hey there @Alex Davies​ Hope you are doing great. Just checking in if you were able to resolve your issue or do you need more help? We'd love to hear from you.Thanks!

  • 2 kudos
8 More Replies
Merchiv
by • New Contributor III
  • 19795 Views
  • 8 replies
  • 2 kudos

Resolved! AnalysisException when running SQL queries

When running some SQL queries using spark.sql(...), we sometimes get a variant of the following error:AnalysisException: Undefined function: current_timestamp. This function is neither a built-in/temporary function, nor a persistent function that is ...

Data Engineering
Reply
  • 19795 Views
  • 8 replies
  • 2 kudos
View Replies
Latest Reply
ashish1
New Contributor III
  • 2 kudos

This is most likely a conflict in the lib code, you can uninstall some libs on your cluster and try to narrow it down to the problematic one.

  • 2 kudos
7 More Replies
Phani1
by • Valued Contributor II
  • 3051 Views
  • 2 replies
  • 1 kudos

Integration Dolly with Databricks

Hi Databricks Team,Could you please share any links /docs/Sample notebooks to integrate Dolly with Databricks, our aim is to generate SQL queries based on the free text and execute it via databricks cluster/SQL warehouse.

Data Engineering
Reply
  • 3051 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
sean_owen
Databricks Employee
  • 1 kudos

https://www.dbdemos.ai/demo.html?demoName=llm-dolly-chatbot is a good demonstration of Dolly (or really any LLM) for question answering. LLMs like this are not for SQL generation, but other LLMs are, like starcoderbase

  • 1 kudos
1 More Replies
deficiant_codge
by • Contributor II
  • 6525 Views
  • 7 replies
  • 9 kudos

Can't able to run following queries

I cannot able to run following queriesALTER TABLE iot_events ADD ATTRIBUTE pii ON emailALTER TABLE users ADD ATTRIBUTE pii ON phoneGRANT SELECT ON DATABASE iot_data HAVING ATTRIBUTE NOT IN (pii) TO product_managersand GRANT SELECT ON iot_events TO ...

Data Engineering
Reply
  • 6525 Views
  • 7 replies
  • 9 kudos
View Replies
Latest Reply
karthik_p
Esteemed Contributor
  • 9 kudos

@Kaniz Fatma​ can anyone from data Bircks help on why attribute-based access control function is not working in unity catalog @Rahul Mishra​ below commands ALTER TABLE iot_events ADD ATTRIBUTE pii ON emailALTER TABLE users ADD ATTRIBUTE pii ON phoneG...

  • 9 kudos
6 More Replies
Hubert-Dudek
by • Esteemed Contributor III
  • 1181 Views
  • 1 replies
  • 5 kudos

Starting from #databricks runtime 12.2 LTS, implicit lateral column aliasing is now supported. This feature enables you to reuse an expression defined...

Starting from #databricks runtime 12.2 LTS, implicit lateral column aliasing is now supported. This feature enables you to reuse an expression defined earlier in the same SELECT list, thus avoiding repetition of the same calculation.For instance, in ...

ezgif-3-d3fac0139c
Data Engineering
Reply
  • 1181 Views
  • 1 replies
  • 5 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 5 kudos

Thanks for sharing this with the Databricks community.

  • 5 kudos
Hubert-Dudek
by • Esteemed Contributor III
  • 2231 Views
  • 3 replies
  • 7 kudos

Starting from #databricks runtime 12.2 LTS, implicit lateral column aliasing is now supported. This feature enables you to reuse an expression defined...

Starting from #databricks runtime 12.2 LTS, implicit lateral column aliasing is now supported. This feature enables you to reuse an expression defined earlier in the same SELECT list, thus avoiding repetition of the same calculation.For instance, in ...

ezgif-3-d3fac0139c
Data Engineering
Reply
  • 2231 Views
  • 3 replies
  • 7 kudos
View Replies
Latest Reply
Ajay-Pandey
Esteemed Contributor III
  • 7 kudos

Informative Thanks for sharing.

  • 7 kudos
2 More Replies
EmilioGC
by • New Contributor III
  • 6011 Views
  • 5 replies
  • 7 kudos

Resolved! Why was SQL formatting removed inside spark.sql functions? Now it looks like a plain string.

Previously we were able to see SQL queries inside spark.sql() like this:But now it just looks like a plain string: I know it's not a big issue, but it's still annoying to have to code in SQL while having it all be blue, it makes debugging more cumber...

old format new format
Data Engineering
Reply
  • 6011 Views
  • 5 replies
  • 7 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 7 kudos

Hi @Emilio Garza​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.

  • 7 kudos
4 More Replies
StanleyTang
by • New Contributor III
  • 2657 Views
  • 3 replies
  • 4 kudos

How to run SQL queries from services when data migrated from SQL server to data lake?

Currently our service provides an API to serve the purchase records. The purchase records are stored in SQL database. To simplify, when users want to get their recent purchase records, they make an API call. The API call will run a SQL query on the D...

Data Engineering
Reply
  • 2657 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
Debayan
Databricks Employee
  • 4 kudos

Hi @Stanley Tang​ , There are several rest API resources managed by Databricks. You can refer https://docs.databricks.com/dev-tools/api/latest/index.html. In this scenario, SQL Warehouses API can be used: https://docs.databricks.com/sql/api/sql-endpo...

  • 4 kudos
2 More Replies
KrishZ
by • Contributor
  • 2372 Views
  • 4 replies
  • 0 kudos

How to prevent sql queries in 2 notebooks from reading the same row from a Table ?

I have an SQL query to select and update rows in a table. I do this in batches of 300 rows (select 300 , update the selected 300 , select new 300 and update the newly selected and so on..) I run this query in 2 different notebooks concurrently to spe...

Data Engineering
Reply
  • 2372 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Krishna Zanwar​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 0 kudos
3 More Replies
Ank
by • New Contributor II
  • 1328 Views
  • 1 replies
  • 2 kudos

Why am I getting a FileNotFoundError after providing the file path?

I used copy file path to get the file path of the notebook I am trying to run from another notebook.file_path = "/Users/ankur.lohiya@workday.com/PAS/Training/Ingest/TrainingQueries-Cloned.py/"ddi = DatabricksDataIngestion(file_path=file_path,        ...

Data Engineering
Reply
  • 1328 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
Vidula
Honored Contributor
  • 2 kudos

Hello @Ankur Lohiya​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...

  • 2 kudos
enichante
by • New Contributor
  • 3562 Views
  • 4 replies
  • 5 kudos

Resolved! Databricks: Report on SQL queries that are being executed

We have a SQL workspace with a cluster running that services a number of self service reports against a range of datasets. We want to be able to analyse and report on the queries our self service users are executing so we can get better visibility of...

Data Engineering
Reply
  • 3562 Views
  • 4 replies
  • 5 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 5 kudos

Looks like the people have spoken: API is your best option! (thanks @Werner Stinckens​  @Chris Grabiel​  and @Bilal Aslam​ !) @eni chante​ Let us know if you have questions about the API! If not, please mark one of the replies above as the "best answ...

  • 5 kudos
3 More Replies
Labels