cancel
Showing results for 
Search instead for 
Did you mean: 
Warehousing & Analytics
Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Community. Share insights, tips, and best practices for leveraging data for informed decision-making.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Warehousing, Analytics, and BI

Forum Posts

MadelynM
by Databricks Employee
  • 939 Views
  • 0 replies
  • 0 kudos

[Recap] Data + AI Summit 2024 - Warehousing & Analytics | Improve performance and increase insights

Here's your Data + AI Summit 2024 - Warehousing & Analytics recap as you use intelligent data warehousing to improve performance and increase your organization’s productivity with analytics, dashboards and insights.  Keynote: Data Warehouse presente...

Screenshot 2024-07-03 at 10.15.26 AM.png
Warehousing & Analytics
AI BI Dashboards
AI BI Genie
Databricks SQL
  • 939 Views
  • 0 replies
  • 0 kudos
vishva-fivetran
by New Contributor II
  • 1472 Views
  • 2 replies
  • 0 kudos

Databricks query on a warehouse failing with ` spark.driver.maxResultSize` error

we are trying to run a select * from one of our catalog tables. we are seeing the error : org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized r...

  • 1472 Views
  • 2 replies
  • 0 kudos
Latest Reply
anardinelli
Databricks Employee
  • 0 kudos

Hello @vishva-fivetran how are you? To set the spark.driver.maxResultSize property, you can do so in the cluster Spark config. The property spark.driver.maxResultSize can be set to a value higher than the value reported in the exception message. For ...

  • 0 kudos
1 More Replies
Akshay_Petkar
by Contributor
  • 1422 Views
  • 1 replies
  • 1 kudos

Resolved! Is Photon Enabled by Default for Warehouses in Databricks?

When creating a cluster in Databricks, I notice an option for enabling Photon. However, this option does not appear when configuring a warehouse. Is Photon enabled by default for warehouses in Databricks, or is there a different procedure to enable i...

  • 1422 Views
  • 1 replies
  • 1 kudos
Latest Reply
Yeshwanth
Databricks Employee
  • 1 kudos

@Akshay_Petkar good day! Photon is enabled by default for Databricks SQL warehouses. There is no need for a separate procedure to enable it for warehouses. For clusters, you have the option to manually enable or disable Photon by selecting the "Use P...

  • 1 kudos
DominikBraun
by New Contributor II
  • 1467 Views
  • 2 replies
  • 1 kudos

Resolved! SQL Warehouse: INVALID_PARAMETER_VALUE when starting

Hey everybody.When creating a SQL Warehouse and trying to start it, I get the following error message:Clusters are failing to launch. Cluster launch will be retried.Request to create a cluster failed with an exception: INVALID_PARAMETER_VALUE: Cannot...

  • 1467 Views
  • 2 replies
  • 1 kudos
Latest Reply
Ismael-K
Databricks Employee
  • 1 kudos

One suggestion would be if you could please have the workspace admin check the Data Access Configuration properties in the Workspace Settings for any secrets that may be held which the warehouse is trying to access on start up? These data access prop...

  • 1 kudos
1 More Replies
Akshay_Petkar
by Contributor
  • 1328 Views
  • 1 replies
  • 0 kudos

Resolved! Databricks warehouse cost calculation

I would like to know how the cost for a warehouse in Databricks is calculated. Specifically, if the cost for a 2x small warehouse is 4 DBUs per hour, how is the cost determined if I use the warehouse for only 30 minutes and then terminate it? Will it...

  • 1328 Views
  • 1 replies
  • 0 kudos
Latest Reply
daniel_sahal
Esteemed Contributor
  • 0 kudos

@Akshay_Petkar As it states in Databricks documentation:Databricks offers you a pay-as-you-go approach with no up-front costs. Only pay for the products you use at per second granularity.It will charge 2 DBUs.

  • 0 kudos
Akshay_Petkar
by Contributor
  • 950 Views
  • 1 replies
  • 0 kudos

Facing Data Truncation Issues in Databricks Dashboards

I'm encountering data truncation in my Databricks dashboards. I'm working with a large dataset, and the dashboard only displays a limited number of (truncated) rows.let's take a dataset containing 1 million sales records. The dashboard currently only...

  • 950 Views
  • 1 replies
  • 0 kudos
Latest Reply
NandiniN
Databricks Employee
  • 0 kudos

Hi @Akshay_Petkar , This is simple in Databricks SQL, just uncheck LIMIT 1000 in the drop down. https://docs.databricks.com/en/sql/get-started/visualize-data-tutorial.html Limit 1000 is selected by default for all queries to ensure that the query ret...

  • 0 kudos
diego_poggioli
by Contributor
  • 5242 Views
  • 5 replies
  • 2 kudos

Resolved! Nested subquery is not supported in the DELETE condition

According to the documentation the WHERE predicate in a DELETE statement should supports subqueries, including IN, NOT IN, EXISTS, NOT EXISTS, and scalar subqueries.if I try to run a query like:   DELETE FROM dev.gold.table AS trg WHERE EXISTS ( ...

  • 5242 Views
  • 5 replies
  • 2 kudos
Latest Reply
Tejas2022
New Contributor II
  • 2 kudos

@diego_poggioli Can you try selecting a 'year_month_version' column from the view instead of select * DELETE FROM dev.gold.table AS trg WHERE year_month_version IN (select year_month_version FROM v_distinct_year_month_version)  

  • 2 kudos
4 More Replies
Akshay_Petkar
by Contributor
  • 2368 Views
  • 2 replies
  • 2 kudos

Resolved! Truncated Data on Lakeview Dashboard

When we create a Lakeview dashboard, the visuals show truncated data. I want to create a dashboard using the entire dataset because the charts do not display the exact values needed for accurate analysis.

  • 2368 Views
  • 2 replies
  • 2 kudos
Latest Reply
raphaelblg
Databricks Employee
  • 2 kudos

Hello @Akshay_Petkar , The FrontEnd (FE) rendering is a limit that determines what data we render on the FE that has been returned by the BackEnd (BE). Render all button allows you to render all the data on the FE that has been returned by the BE. Re...

  • 2 kudos
1 More Replies
data_guy
by New Contributor
  • 1599 Views
  • 1 replies
  • 1 kudos

Resolved! Issue while extracting value From Decimal Key is Json

Hi Guys,I have a JSON as the below structure where the key is as decimal.{ "5.0": { "a": "15.92", "b": 0.0, "c": "15.92", "d": "637.14" }, "0.0": { "a": "15.92", "b": 0.0, "c": "15.92", "d": "637.14" } }schema_of_json returns the following:STRUCT<`0....

  • 1599 Views
  • 1 replies
  • 1 kudos
Latest Reply
raphaelblg
Databricks Employee
  • 1 kudos

Hello @data_guy , I've performed a reproduction of your scenario and could successfully select all data. Please check screenshot below:      This is the source code: import json data_values = { "5.0": { "a": "15.92", "b": 0.0, "c":...

  • 1 kudos
dalion
by New Contributor III
  • 5248 Views
  • 5 replies
  • 2 kudos

Azure VNet Injection and Power BI

Need help in Azure Databricks VNet architecture with Power BI. Our Databricks setup is based on No Public IP. Does anyone has a architecture that can support the Databricks setup in VNet Injection and Power BI tool accessing the Databricks.

  • 5248 Views
  • 5 replies
  • 2 kudos
Latest Reply
databricks_noob
New Contributor II
  • 2 kudos

happy to help further. email me or call me - takutz.hoshyam@gmail.com, +15123726322.

  • 2 kudos
4 More Replies
Billycheng
by New Contributor
  • 1765 Views
  • 4 replies
  • 0 kudos

Error Message when Creating Catalog

I am very new to DB. Can someone show me how to resolve the error below please? Assistant  The error message you're encountering indicates that when creating a catalog, you need to specify a managed storage location for it. This is a requirement in o...

  • 1765 Views
  • 4 replies
  • 0 kudos
Latest Reply
dkushari
Databricks Employee
  • 0 kudos

If your root metastore bucket is not configured then you need to provide a managed location when creating a catalog in UC. https://docs.databricks.com/en/data-governance/unity-catalog/create-catalogs.html#create-a-catalog    

  • 0 kudos
3 More Replies
Eric_Kieft
by New Contributor III
  • 1045 Views
  • 1 replies
  • 1 kudos

Materialized Views GA and Azure Region Availability

Materialized views are currently public preview (as of May 2024).  Is there a planned date for GA?Also the limitations section for Azure notes: Databricks SQL materialized views are not supported in the South Central US and West US 2 regions.Will thi...

  • 1045 Views
  • 1 replies
  • 1 kudos
Latest Reply
Eric_Kieft
New Contributor III
  • 1 kudos

Thanks, Kaniz. Will this feature be available for these regions in the future?

  • 1 kudos
Mado
by Valued Contributor II
  • 12085 Views
  • 9 replies
  • 4 kudos

Resolved! How to get usage statistics from Databricks or SQL Databricks?

Hi, I am looking for a way to get usage statistics from Databricks (Data Science & Engineering and SQL persona). For example: I created a table. I want to know how many times a specific user queried that table.How many times a pipeline was triggered?...

  • 12085 Views
  • 9 replies
  • 4 kudos
Latest Reply
twole
Databricks Employee
  • 4 kudos

You can use System Tables, now available in Unity Catalog metastore, to create the views you described. https://docs.databricks.com/en/admin/system-tables/index.html

  • 4 kudos
8 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels
Top Kudoed Authors