cancel
Showing results for 
Search instead for 
Did you mean: 
Warehousing & Analytics
Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Community. Share insights, tips, and best practices for leveraging data for informed decision-making.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Warehousing, Analytics, and BI

Forum Posts

MadelynM
by Databricks Employee
  • 932 Views
  • 0 replies
  • 0 kudos

[Recap] Data + AI Summit 2024 - Warehousing & Analytics | Improve performance and increase insights

Here's your Data + AI Summit 2024 - Warehousing & Analytics recap as you use intelligent data warehousing to improve performance and increase your organization’s productivity with analytics, dashboards and insights.  Keynote: Data Warehouse presente...

Screenshot 2024-07-03 at 10.15.26 AM.png
Warehousing & Analytics
AI BI Dashboards
AI BI Genie
Databricks SQL
  • 932 Views
  • 0 replies
  • 0 kudos
Akshay_Petkar
by Contributor
  • 599 Views
  • 1 replies
  • 1 kudos

Impact of Overwriting Databricks SQL Tables on Versioning

When changes are made to a Databricks SQL table, a new version is created. If changes to the table are made using Spark or Python in a notebook and the table is overwritten, will a new version be created, or will it remain as version number 0?

  • 599 Views
  • 1 replies
  • 1 kudos
Latest Reply
Rishabh-Pandey
Esteemed Contributor
  • 1 kudos

When changes are made to a Databricks SQL table (Delta table) using Spark or Python in a notebook, and the table is overwritten, a new version will indeed be created. It will not remain as version number 0. Each overwrite operation increments the ver...

  • 1 kudos
MadelynM
by Databricks Employee
  • 932 Views
  • 0 replies
  • 0 kudos

[Recap] Data + AI Summit 2024 - Warehousing & Analytics | Improve performance and increase insights

Here's your Data + AI Summit 2024 - Warehousing & Analytics recap as you use intelligent data warehousing to improve performance and increase your organization’s productivity with analytics, dashboards and insights.  Keynote: Data Warehouse presente...

Screenshot 2024-07-03 at 10.15.26 AM.png
Warehousing & Analytics
AI BI Dashboards
AI BI Genie
Databricks SQL
  • 932 Views
  • 0 replies
  • 0 kudos
Akshay_Petkar
by Contributor
  • 1272 Views
  • 1 replies
  • 0 kudos

Databrics offers auto tuning?

is databricks offers auto tunning?

  • 1272 Views
  • 1 replies
  • 0 kudos
Latest Reply
raphaelblg
Databricks Employee
  • 0 kudos

Yes, First of all, open source spark already has a set of auto-tuning features denominated Adaptive Query Execution (AQE). Here are more details: https://spark.apache.org/docs/latest/sql-performance-tuning.html#adaptive-query-execution. For even bett...

  • 0 kudos
Shaimaa
by New Contributor II
  • 556 Views
  • 1 replies
  • 0 kudos

running a query against multiple parquet files from a folder

I am runninng a query against multiple parquet files:SELECT SUM(CASE WHEN match_result.year_incorporated IS NOT NULL AND match_result.year_incorporated != '' THEN 1 ELSE 0 END) FROM parquet.`s3://folder_path/*`for some files, the field `year_incorpo...

  • 556 Views
  • 1 replies
  • 0 kudos
Latest Reply
daniel_sahal
Esteemed Contributor
  • 0 kudos

@Shaimaa The column type mismatch between the files could be an issue here.For example: if in one file column 'xyz' is a type of INTEGER and in another one the same column is a type of STRING, Spark will give you a schema conversion error.Below is a ...

  • 0 kudos
Shaimaa
by New Contributor II
  • 1196 Views
  • 3 replies
  • 0 kudos

null object while running a query against parquet

I am running this query against parquet:SELECT SUM(CASE WHEN match_result.ecommerce.has_online_payments THEN 1 ELSE 0 END) FROM parquet.`s3://folder_path/*`when all the values of the object `match_result.ecommerce` are null, I get the following erro...

  • 1196 Views
  • 3 replies
  • 0 kudos
Latest Reply
Shaimaa
New Contributor II
  • 0 kudos

None of these solutions with coalesce work because it's "match_result.ecommerce" that is null not "match_result.ecommerce.has_online_payments". So it's still trying to extract a value from a null. Help me modify the query accordingly please. 

  • 0 kudos
2 More Replies
dbph
by New Contributor II
  • 996 Views
  • 1 replies
  • 0 kudos

How to use Serverless as DBT-CLI compute?

Hi,we'd like to use serverless as the compute for DBT-CLI (of course we already used Serverless SQL before) in a DBT workflow.I configured a normal DBT-task and tried to run a dbt-run command, which i previously tested sucessfully on my local machine...

  • 996 Views
  • 1 replies
  • 0 kudos
Latest Reply
dbph
New Contributor II
  • 0 kudos

Hi, thanks for your help!Unfortunately our problem is not the Databricks-CLI we're using on local machines. The problem is the DBT-CLI which we are trying to run on serverless compute inside a Databricks-Workflow.I already tried adding the code you p...

  • 0 kudos
Akshay_Petkar
by Contributor
  • 962 Views
  • 2 replies
  • 0 kudos

Databricks External Table (ADLS) Access with Power BI?

Can we to connect Power BI directly to an Azure Databricks external table located in ADLS using cluster credentials?

  • 962 Views
  • 2 replies
  • 0 kudos
Latest Reply
Wojciech_BUK
Valued Contributor III
  • 0 kudos

It depends- if you want to connect Power BI to table (wither external or managed) you can do it  BUT it will be through cluster (each time you run something on Power BI desktop - it will wake up cluster or SQL endpoint and you will access data in thi...

  • 0 kudos
1 More Replies
jakubk
by Contributor
  • 2794 Views
  • 2 replies
  • 3 kudos

Resolved! unity catalog information schema columns metadata out of sync with table - cant refresh

I'm using unity catalogI've changed the schema of my table by overwriting it with a newer file df.write \ .format('delta') \ .partitionBy(partitionColumn) \ .mode("overwrite") \ .option("overwriteSchema", "true") \ ...

  • 2794 Views
  • 2 replies
  • 3 kudos
Latest Reply
daniel_sahal
Esteemed Contributor
  • 3 kudos

@jakubk Try runningREPAIR TABLE <table_name> SYNC METADATA 

  • 3 kudos
1 More Replies
EWhitley
by New Contributor III
  • 4147 Views
  • 2 replies
  • 2 kudos

Retrieve task name within workflow task (notebook, python)?

Using workflows, is there a way to obtain the task name from within a task?EX: I have a workflow with a notebook task. From within that notebook task I would like to retrieve the task name so I can use it for a variety of purposes.Currently, we're re...

  • 4147 Views
  • 2 replies
  • 2 kudos
Latest Reply
ttamas
New Contributor III
  • 2 kudos

Hi @EWhitley,Would {{task.name}} help in getting the current task name?https://docs.databricks.com/en/workflows/jobs/parameter-value-references.htmlPass context about job runs into job t 

  • 2 kudos
1 More Replies
Shaimaa
by New Contributor II
  • 3849 Views
  • 2 replies
  • 0 kudos

Running SQL queries against a parquet folder in S3

I need to run sql queries against a parquet folder in S3. I am trying to use "read_files" but sometimes my queries fail due to errors while inferring the schema and sometimes without a specified reason. Sample query:  SELECT SUM(CASE WHEN match_resu...

  • 3849 Views
  • 2 replies
  • 0 kudos
Latest Reply
holly
Databricks Employee
  • 0 kudos

There's a few alternatives for you. 1. a switch in syntax - I doubt this will make much difference, but worth a shot SELECT ... FROM parquet.`s3://folder_path` 2. Create a view first then query against it. You should get better errors this way.  CRE...

  • 0 kudos
1 More Replies
acurus
by New Contributor II
  • 2667 Views
  • 2 replies
  • 2 kudos

Not able to escape `-` in external connected tablename

We are having some issues with getting data from some tables with the character `-` in their tablename.We are connected to the database with an SQL server connection, and the database is (as far as we know)  a Microsoft Azure SQL Database. We do not ...

  • 2667 Views
  • 2 replies
  • 2 kudos
Latest Reply
martinschou
New Contributor II
  • 2 kudos

Had the same issue when querying a table with the - character in the table name. Got the error: Incorrect syntax near '-'.Got the error on Databricks runtime version: 13.2 (includes Apache Spark 3.4.0, Scala 2.12)No error when using Databricks runtim...

  • 2 kudos
1 More Replies
ssequ
by New Contributor II
  • 2164 Views
  • 3 replies
  • 2 kudos

Resolved! Backspaces in Foreign Catalog Table Names

Hi,is there a way to import tables with UC disallowed characters when creating a foreign catalog? The database we are dealing with contains table names with backspaces and UC seems to completely ignore them when creating the catalog.Thanks for any he...

  • 2164 Views
  • 3 replies
  • 2 kudos
Latest Reply
daniel_sahal
Esteemed Contributor
  • 2 kudos

@ssequ I assume that you're refering to lakehouse federation? Unfortunately that's a limitation, tables with disallowed characters will be ignored.

  • 2 kudos
2 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels
Top Kudoed Authors