cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

AhsanKhawaja
by New Contributor
  • 2826 Views
  • 4 replies
  • 0 kudos

using databricks sql warehouse as web app backend

Hi,I wanted to ask if anyone is using Databricks SQL Warehouse as backend for small to large scale web application? What are your thoughts about it, specially what Databricks team thinks of it ?Kind Regards,A

  • 2826 Views
  • 4 replies
  • 0 kudos
Latest Reply
Robert-Scott
New Contributor II
  • 0 kudos

Using Databricks SQL Warehouse as a backend for a web application involves integrating Databricks with your web app to handle data processing, querying, and analytics. Here are the steps to achieve this:1. Set Up Databricks SQL WarehouseCreate a Data...

  • 0 kudos
3 More Replies
pgruetter
by Contributor
  • 9256 Views
  • 7 replies
  • 1 kudos

Resolved! How to use Service Principal to connect PowerBI to Databrick SQL Warehouse

Hi allI'm struggling to connect PowerBI service to a Databricks SQL Warehouse using a service principal. I'm following mostly this guide.I created a new app registration in the AAD and created a client secret for it.Now I'm particularly struggling wi...

  • 9256 Views
  • 7 replies
  • 1 kudos
Latest Reply
pgruetter
Contributor
  • 1 kudos

In the end, once the Service Principal is properly authorized on the Databricks side, I had to create a Personal Access Token for the Service Principal using the Databricks API. On the Power BI service side I then had to use username = 'token' and as...

  • 1 kudos
6 More Replies
rendorHaevyn
by New Contributor III
  • 4038 Views
  • 3 replies
  • 0 kudos

Databricks SQL Warehouse did not auto stop after specified 90 minute interval - why not?

In this specific case, we're running a 2XSmall SQL Warehouse on Databricks SQL.In looking at the SQL Warehouse monitoring log for this cluster, we noticed:final query executed by user at 10:26 on 2023-06-20no activity for some time, yet cluster remai...

  • 4038 Views
  • 3 replies
  • 0 kudos
Latest Reply
Emil_Kaminski
Contributor
  • 0 kudos

@Michael42 sounds like some sort of horror story. Let us know how it goes. It happens to me as well, but I was lucky enough to have this situation on very small compute cluster for just couple of days.

  • 0 kudos
2 More Replies
Fz1
by New Contributor III
  • 5461 Views
  • 5 replies
  • 3 kudos

Resolved! SQL Warehouse Serverless - Not able to access the external tables in the hive_metastore

I have DLT tables created under the hive_metastore with external data stored in ADL gen2.The ADL blob storage is mounted into /mnt/<storage-account>The tables are successfully created and accessible from my notebooks, as well the ADL storage.I have c...

  • 5461 Views
  • 5 replies
  • 3 kudos
Latest Reply
TjommeV-Vlaio
New Contributor II
  • 3 kudos

Can this be done using Terraform as well?

  • 3 kudos
4 More Replies
Rajesh_P
by New Contributor II
  • 761 Views
  • 1 replies
  • 1 kudos

Does JDBC and ODBC connection to Databricks Cluster or SQL Warehouse encrypted?

Hello, our consumers (Dell Boomi and other apps) needs data from Databricks. Databricks provides JDBC and ODBC drivers. Does JDBC and ODBC connection to Databricks Cluster or SQL Warehouse encrypted? I am talking about the data in-transit between Dat...

  • 761 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Rajesh Paul​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 1 kudos
Mado
by Valued Contributor II
  • 1572 Views
  • 2 replies
  • 1 kudos

Resolved! How to set timezone for SQL Warehouse?

Hi, I want to change the default time zone for SQL Warehoue in the SQL Persona. When I try to Edit the SQL warehouse settings in the "SQL Warehouses" section, I am not able to find any setting where I can set the time zone. I am aware that I can set ...

  • 1572 Views
  • 2 replies
  • 1 kudos
Latest Reply
Mado
Valued Contributor II
  • 1 kudos

Thanks. I am aware of the SET TIME ZONE command but I need to run this command every time I start the SQL warehouse. I am looking for a way to change the default time zone of the SQL warehouse. Something like "spark.sql.session.timeZone GMT+10" that ...

  • 1 kudos
1 More Replies
lasribeiro_univ
by New Contributor III
  • 3915 Views
  • 3 replies
  • 3 kudos

Resolved! Cluster SQL Warehouse fails to Start.

I'm a new user of Databricks and I'm taking the Academy course, but I'm having difficulty starting a SQL Warehouse cluster. I've tried several different configurations, but I always get the same error:Clusters are failing to launch. Cluster launch wi...

SQL Warehouse Starter 2 SQL Warehouse Starter 1
  • 3915 Views
  • 3 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Luiz Ribeiro​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...

  • 3 kudos
2 More Replies
Marvin_T
by New Contributor III
  • 6701 Views
  • 2 replies
  • 2 kudos

Resolved! Disabling query caching for SQL Warehouse

Hello everybody,I am currently trying to run some performance tests on queries in Databricks on Azure. For my tests, I am using a Classic SQL Warehouse in the SQL Editor. I have created two views that contain the same data but have different structur...

  • 6701 Views
  • 2 replies
  • 2 kudos
Latest Reply
Marvin_T
New Contributor III
  • 2 kudos

They are probably executing the same query plan now that you say it. And yes, restarting the warehouse does theoretically works but it isnt a nice solution.I guess I will do some restarting and build averages to have a good comparison for now

  • 2 kudos
1 More Replies
Phani1
by Valued Contributor
  • 1690 Views
  • 2 replies
  • 1 kudos

Integration Dolly with Databricks

Hi Databricks Team,Could you please share any links /docs/Sample notebooks to integrate Dolly with Databricks, our aim is to generate SQL queries based on the free text and execute it via databricks cluster/SQL warehouse.

  • 1690 Views
  • 2 replies
  • 1 kudos
Latest Reply
sean_owen
Honored Contributor II
  • 1 kudos

https://www.dbdemos.ai/demo.html?demoName=llm-dolly-chatbot is a good demonstration of Dolly (or really any LLM) for question answering. LLMs like this are not for SQL generation, but other LLMs are, like starcoderbase

  • 1 kudos
1 More Replies
jlgr
by New Contributor II
  • 2850 Views
  • 4 replies
  • 2 kudos

Resolved! How disable disk cache in SQL Warehouse (Azure Databricks)?

Hi! I want to disable disk cache for SQL Warehouse in Azure Databricks, but it seems that is not possible. Is it correct?You can't use this configuration for SQL Warehouse (https://learn.microsoft.com/en-US/azure/databricks/optimizations/disk-cache#-...

  • 2850 Views
  • 4 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @jlgr jlgr​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we ...

  • 2 kudos
3 More Replies
fuselessmatt
by Contributor
  • 2610 Views
  • 3 replies
  • 0 kudos

Omitting columns in an INSERT statement does not seem to work despite meeting the requirements

We want to use the INSERT INTO command with specific columns as specified in the official documentation. The only requirements for this are​️ Databricks SQL warehouse version 2022.35 or higher️ Databricks Runtime 11.2 and above​and the behaviour shou...

  • 2610 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Fusselmanwog​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...

  • 0 kudos
2 More Replies
RafaelGomez61
by New Contributor
  • 2276 Views
  • 2 replies
  • 0 kudos

Can't access delta tables under SQL Warehouse cluster. Getting Error while using path .../_delta_log/000000000.checkpoint

In our Databricks workspace, we have several delta tables available in the hive_metastore catalog. we are able to access and query the data via Data Science & Engineering persona clusters with no issues. The cluster have the credential passthrough en...

  • 2276 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Rafael Gomez​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so ...

  • 0 kudos
1 More Replies
najmead
by Contributor
  • 3426 Views
  • 2 replies
  • 1 kudos

Spark Settings in SQL Warehouse

I'm running a query, trying to parse a string into a map, and I get the following error;org.apache.spark.SparkRuntimeException: Duplicate map key was found, please check the input data. If you want to remove the duplicated keys, you can set "spark.s...

  • 3426 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Nicholas Mead​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedbac...

  • 1 kudos
1 More Replies
Mado
by Valued Contributor II
  • 818 Views
  • 2 replies
  • 2 kudos

Can I use a cluster created in Data Science & Engineering persona to run SQL commands in the SQL persona?

Hi,I have created a single-node cluster in Data Science & Engineering persona (Standard_DS3_v2). I don't have enough vCPU to create a SQL warehouse. Is there any way I can use the cluster to run a query in SQL persona?

  • 818 Views
  • 2 replies
  • 2 kudos
Latest Reply
Rajeev45
New Contributor III
  • 2 kudos

Hi MadoYes, you can use cluster and run sql query in the notebook, please refer the following page for more details. https://docs.databricks.com/getting-started/quick-start.html#tutorial-query-data-with-notebookshttps://docs.databricks.com/getting-st...

  • 2 kudos
1 More Replies
Sujitha
by Community Manager
  • 737 Views
  • 2 replies
  • 2 kudos

Weekly Release Notes RecapHere’s a quick recap of the latest release notes updates from the past one week. Databricks platform release notesMarch 13 -...

Weekly Release Notes RecapHere’s a quick recap of the latest release notes updates from the past one week.Databricks platform release notesMarch 13 - 17, 2023Execute SQL cells in the notebook in parallelYou can now run SQL cells in Databricks noteboo...

  • 737 Views
  • 2 replies
  • 2 kudos
Latest Reply
Kaniz
Community Manager
  • 2 kudos

Hi @Sujitha Ramamoorthy​, Thank you for the valuable content you've contributed to our community. Your well-researched insights and thought-provoking discussions have greatly benefited our team, inspiring us to grow and learn together.Your dedication...

  • 2 kudos
1 More Replies
Labels