Here's your Data + AI Summit 2024 - Warehousing & Analytics recap as you use intelligent data warehousing to improve performance and increase your organization’s productivity with analytics, dashboards and insights. Keynote: Data Warehouse presente...
We are having some issues with getting data from some tables with the character `-` in their tablename.We are connected to the database with an SQL server connection, and the database is (as far as we know) a Microsoft Azure SQL Database. We do not ...
Had the same issue when querying a table with the - character in the table name. Got the error: Incorrect syntax near '-'.Got the error on Databricks runtime version: 13.2 (includes Apache Spark 3.4.0, Scala 2.12)No error when using Databricks runtim...
Hi,is there a way to import tables with UC disallowed characters when creating a foreign catalog? The database we are dealing with contains table names with backspaces and UC seems to completely ignore them when creating the catalog.Thanks for any he...
@ssequ I assume that you're refering to lakehouse federation? Unfortunately that's a limitation, tables with disallowed characters will be ignored.
Cannot find a way to delete this post, but edited
we are trying to run a select * from one of our catalog tables. we are seeing the error : org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized r...
Hello @vishva-fivetran how are you? To set the spark.driver.maxResultSize property, you can do so in the cluster Spark config. The property spark.driver.maxResultSize can be set to a value higher than the value reported in the exception message. For ...
We set up the SQL warehouse IAM role in the settings option. This is applied to all warehouses. How do I create sql warehouses with multiple IAM roles to maintain access control.
When creating a cluster in Databricks, I notice an option for enabling Photon. However, this option does not appear when configuring a warehouse. Is Photon enabled by default for warehouses in Databricks, or is there a different procedure to enable i...
@Akshay_Petkar good day! Photon is enabled by default for Databricks SQL warehouses. There is no need for a separate procedure to enable it for warehouses. For clusters, you have the option to manually enable or disable Photon by selecting the "Use P...
Hey everybody.When creating a SQL Warehouse and trying to start it, I get the following error message:Clusters are failing to launch. Cluster launch will be retried.Request to create a cluster failed with an exception: INVALID_PARAMETER_VALUE: Cannot...
One suggestion would be if you could please have the workspace admin check the Data Access Configuration properties in the Workspace Settings for any secrets that may be held which the warehouse is trying to access on start up? These data access prop...
I would like to know how the cost for a warehouse in Databricks is calculated. Specifically, if the cost for a 2x small warehouse is 4 DBUs per hour, how is the cost determined if I use the warehouse for only 30 minutes and then terminate it? Will it...
@Akshay_Petkar As it states in Databricks documentation:Databricks offers you a pay-as-you-go approach with no up-front costs. Only pay for the products you use at per second granularity.It will charge 2 DBUs.
I'm encountering data truncation in my Databricks dashboards. I'm working with a large dataset, and the dashboard only displays a limited number of (truncated) rows.let's take a dataset containing 1 million sales records. The dashboard currently only...
Hi @Akshay_Petkar , This is simple in Databricks SQL, just uncheck LIMIT 1000 in the drop down. https://docs.databricks.com/en/sql/get-started/visualize-data-tutorial.html Limit 1000 is selected by default for all queries to ensure that the query ret...
We're creating a report with Power BI using data from our AWS Databricks workspace. Currently, I can view the report on Power BI (service) after publishing. Is there a way to change the data source connection, e.g. if I want to change the data source...
Hello,I have an Azure sql warehouse serverless instance that I can connect to using databricks-sql-connector. But, when I try to use pyspark and jdbc driver url, I can't read or write.See my code belowdef get_jdbc_url(): # Define your Databricks p...
According to the documentation the WHERE predicate in a DELETE statement should supports subqueries, including IN, NOT IN, EXISTS, NOT EXISTS, and scalar subqueries.if I try to run a query like: DELETE FROM dev.gold.table AS trg WHERE EXISTS ( ...
@diego_poggioli Can you try selecting a 'year_month_version' column from the view instead of select * DELETE FROM dev.gold.table AS trg WHERE year_month_version IN (select year_month_version FROM v_distinct_year_month_version)
When we create a Lakeview dashboard, the visuals show truncated data. I want to create a dashboard using the entire dataset because the charts do not display the exact values needed for accurate analysis.
Hello @Akshay_Petkar , The FrontEnd (FE) rendering is a limit that determines what data we render on the FE that has been returned by the BackEnd (BE). Render all button allows you to render all the data on the FE that has been returned by the BE. Re...
Hi Guys,I have a JSON as the below structure where the key is as decimal.{ "5.0": { "a": "15.92", "b": 0.0, "c": "15.92", "d": "637.14" }, "0.0": { "a": "15.92", "b": 0.0, "c": "15.92", "d": "637.14" } }schema_of_json returns the following:STRUCT<`0....
Hello @data_guy , I've performed a reproduction of your scenario and could successfully select all data. Please check screenshot below: This is the source code: import json data_values = { "5.0": { "a": "15.92", "b": 0.0, "c":...
Need help in Azure Databricks VNet architecture with Power BI. Our Databricks setup is based on No Public IP. Does anyone has a architecture that can support the Databricks setup in VNet Injection and Power BI tool accessing the Databricks.
happy to help further. email me or call me - takutz.hoshyam@gmail.com, +15123726322.
