Here's your Data + AI Summit 2024 - Warehousing & Analytics recap as you use intelligent data warehousing to improve performance and increase your organization’s productivity with analytics, dashboards and insights. Keynote: Data Warehouse presente...
I'm trying to use Databricks Apps to host a Streamlit app to serve some interactive applicaton. I face two limitations:1. In environment for App I see it using certain python version, but how to update it to use any other version?It is already set to...
Hi @atikiwala , Good Day! Python 3.11 is currently the only version we support. We are thinking of adding additional options in the future. Would love to hear your feedback on this - https://docs.databricks.com/en/resources/ideas.html#submit-product-...
From the DLT documentation it seems that the LIVE TABLE is conceptually the same as MATERIALIZED VIEW. When should I use one over another?
@ImranA and @igorstar I repost my response here again:to create materialized views, you could use CREATE OR REFRESH LIVE TABLE however according to the official docs: The CREATE OR REFRESH LIVE TABLE syntax to create a materialized view is deprecat...
Hi, I'd appreciate it if anyone could help!We are using offical ODBC driver (Simba Spark ODBC Driver 64-bit 2.08.02.1013) in our application. C++ APIs.All the following SQL statements are passed through ODBC API to Databricks:successully executing: C...
@AnnaP Hey,Did you try below:To disable the SQL Connector feature, select the Use Native Query check box.Important:l When this option is enabled, the connector cannot executeparameterized queries.l By default, the connector applies transformations to...
I am working on a use case where I have streaming data that needs to be displayed in real-time on a live dashboard. The goal is for any new data arriving in the stream to instantly reflect on the dashboard. Is this possible on Databricks? If yes, how...
@Akshay_Petkar wrote:I am working on a use case where I have streaming data that needs to be displayed in real-time on a live dashboard. The goal is for any new data arriving in the stream to instantly reflect on the dashboard. Is this possible on Da...
While using cx_oracle python library, it returns the below error: error message: Cannot locate a 64-bit Oracle Client library: "libclntsh.so: cannot open shared object file: No such file or directory The cx_oracle library is dependent on native...
Hi @AshvinManoj I used your script and still get same error. sudo echo 'LD_LIBRARY_PATH="/dbfs/databricks/instantclient_23_6"' >> /databricks/spark/conf/spark-env.shsudo echo 'ORACLE_HOME="/dbfs/databricks/instantclient_23_6"' >> /databricks/spark...
We're creating a report with Power BI using data from our AWS Databricks workspace. Currently, I can view the report on Power BI (service) after publishing. Is there a way to change the data source connection, e.g. if I want to change the data source...
In the Power BI transform data view, you should be able to access the M-Query code and actually change the server and the host directly. My recommendation is to create a couple of parameters to keep this info for all your queries. Then you can just c...
What is the optimal number of cluster/nodes in a warehouse? Depends on your workload. Our DBSQL guide suggests a size range on two main things: time to execute your query and bytes spilled from it. This link can help you understand better optimizatio...
I have an on-premises Power BI Report Server that uses the Simba Spark ODBC Driver (2.8) to connect to Databricks. It can connect to a serverless warehouse successfully and run its queries, but it never seems to disconnect the session, and so the war...
Hi @bradleyjamrozik ,The problem is that the current designed behavior of Power BI can lead to the connection not being closed under certain conditions.In short, since Databricks is Async, if an error occurs while fetching data, the connection may no...
I am preparing to migrate from EMR to Databricks and would like to know the best practices for this process. Is there a direct connector, such as a JDBC connector, available to facilitate the migration? Alternatively, would it be more effective to ex...
@Akshay_Petkar if you're planning to migrate from your current technology to Databricks, Travinto Technologies' Code Converter Tool is here to make the process seamless. This powerful tool enables you to migrate data, ETL workflows, and reports acros...
I cancel my subscription plan (account created via accounts.databricks.com), without noticing that by doing this, all of my workspaces will be deleted. Is this possible to recover those workspace, and if possible, what should I do to recover them?
Hello San2. To recover a workspace, could you please raise a case with support? They will be helping you to recover these workspaces, otherwise you can reach your account team, who will help you for further steps, for more details you see disaster ...
Hi all, working on this project, my team plans to migrate some data from some databases to Databricks. We plan to run this migration by submitting queries to a warehouse through python on a local machine.Now I was wondering what would be the best app...
@Ismail1 if you're planning to migrate from your current technology to Databricks, Travinto Technologies' Code Converter Tool is here to make the process seamless. This powerful tool enables you to migrate data, ETL workflows, and reports across pla...
We are using dbt core and running our transformations in a dbt task. Since a few months ago we have started to see a warning message in our runs. This started happening after an update of dbt core and it seems that Databricks needs to update the def...
Both of you missed the sentence "Databricks needs to update the definition of the generated profiles file."The warning is regarding the structure of the profiles.yml which is generated by Databrick. This is not anything I as a user can change accept ...
Hello,I am new to Databricks and was asked to evaluate the new Apps function as a way to publish dashboards. We do not use Databricks today. We have an Azure SQL database we use as a home grown warehouse. The question is can I connect directly to ...
To connect your Azure SQL database to Databricks Apps, you will need to pull the data from the existing database into Databricks. Databricks Apps are designed to leverage the data and features within the Databricks platform, which means that the data...
Hi ,We are looking for a option to copy tables more than 50 TB to be copied from syanpse to databricks on weekly basis , please suggest of there are any feasible ways for samewe are using connector but it is taking too long to copyhttps://learn.micro...
There is no databricks documentation on this as it is only involved for a very tiny bit:"CREATE TABLE catalog.schema.table USING PARQUET LOCATION 'url_to_the_parquet_files'.All the rest is done in Azure Data Factory, or you can even use the built-in ...
So I did U2M authentication for account-level operations -databricks auth login --host <account-login-url> --account-id <account-id>then I tried to run code-workspaces = account_client.workspaces.list()workspace_obj=account_client.get_workspace_clien...
I am doing using a service principal so my .databrickscfg has -[databricks-demo]client_id = -----client_secret = ----host = https://accounts.cloud.databricks.com/account_id = ----
