Here's your Data + AI Summit 2024 - Warehousing & Analytics recap as you use intelligent data warehousing to improve performance and increase your organization’s productivity with analytics, dashboards and insights. Keynote: Data Warehouse presente...
If I create an external table on AWS Databricks, will it be a Delta table? If not, is there a way to make it a Delta table, or is there no Delta capability for external tables?
is databricks offers auto tunning?
Yes, First of all, open source spark already has a set of auto-tuning features denominated Adaptive Query Execution (AQE). Here are more details: https://spark.apache.org/docs/latest/sql-performance-tuning.html#adaptive-query-execution. For even bett...
I am runninng a query against multiple parquet files:SELECT SUM(CASE WHEN match_result.year_incorporated IS NOT NULL AND match_result.year_incorporated != '' THEN 1 ELSE 0 END) FROM parquet.`s3://folder_path/*`for some files, the field `year_incorpo...
@Shaimaa The column type mismatch between the files could be an issue here.For example: if in one file column 'xyz' is a type of INTEGER and in another one the same column is a type of STRING, Spark will give you a schema conversion error.Below is a ...
I am running this query against parquet:SELECT SUM(CASE WHEN match_result.ecommerce.has_online_payments THEN 1 ELSE 0 END) FROM parquet.`s3://folder_path/*`when all the values of the object `match_result.ecommerce` are null, I get the following erro...
None of these solutions with coalesce work because it's "match_result.ecommerce" that is null not "match_result.ecommerce.has_online_payments". So it's still trying to extract a value from a null. Help me modify the query accordingly please.
Hi,we'd like to use serverless as the compute for DBT-CLI (of course we already used Serverless SQL before) in a DBT workflow.I configured a normal DBT-task and tried to run a dbt-run command, which i previously tested sucessfully on my local machine...
Hi, thanks for your help!Unfortunately our problem is not the Databricks-CLI we're using on local machines. The problem is the DBT-CLI which we are trying to run on serverless compute inside a Databricks-Workflow.I already tried adding the code you p...
Can we to connect Power BI directly to an Azure Databricks external table located in ADLS using cluster credentials?
It depends- if you want to connect Power BI to table (wither external or managed) you can do it BUT it will be through cluster (each time you run something on Power BI desktop - it will wake up cluster or SQL endpoint and you will access data in thi...
I'm using unity catalogI've changed the schema of my table by overwriting it with a newer file df.write \ .format('delta') \ .partitionBy(partitionColumn) \ .mode("overwrite") \ .option("overwriteSchema", "true") \ ...
Using workflows, is there a way to obtain the task name from within a task?EX: I have a workflow with a notebook task. From within that notebook task I would like to retrieve the task name so I can use it for a variety of purposes.Currently, we're re...
Hi @EWhitley,Would {{task.name}} help in getting the current task name?https://docs.databricks.com/en/workflows/jobs/parameter-value-references.htmlPass context about job runs into job t
I need to run sql queries against a parquet folder in S3. I am trying to use "read_files" but sometimes my queries fail due to errors while inferring the schema and sometimes without a specified reason. Sample query: SELECT SUM(CASE WHEN match_resu...
There's a few alternatives for you. 1. a switch in syntax - I doubt this will make much difference, but worth a shot SELECT ... FROM parquet.`s3://folder_path` 2. Create a view first then query against it. You should get better errors this way. CRE...
We're currently using a private cloud implementation of Databricks on AWS (not SaaS version), and the Databricks/Simba ODBC driver v2.6.29 with personal access token to fetch data into desktop tools. Can I enable Okta web/mobile authentication with t...
We are having some issues with getting data from some tables with the character `-` in their tablename.We are connected to the database with an SQL server connection, and the database is (as far as we know) a Microsoft Azure SQL Database. We do not ...
Had the same issue when querying a table with the - character in the table name. Got the error: Incorrect syntax near '-'.Got the error on Databricks runtime version: 13.2 (includes Apache Spark 3.4.0, Scala 2.12)No error when using Databricks runtim...
Hi,is there a way to import tables with UC disallowed characters when creating a foreign catalog? The database we are dealing with contains table names with backspaces and UC seems to completely ignore them when creating the catalog.Thanks for any he...
@ssequ I assume that you're refering to lakehouse federation? Unfortunately that's a limitation, tables with disallowed characters will be ignored.
Cannot find a way to delete this post, but edited
we are trying to run a select * from one of our catalog tables. we are seeing the error : org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized r...
Hello @vishva-fivetran how are you? To set the spark.driver.maxResultSize property, you can do so in the cluster Spark config. The property spark.driver.maxResultSize can be set to a value higher than the value reported in the exception message. For ...
We set up the SQL warehouse IAM role in the settings option. This is applied to all warehouses. How do I create sql warehouses with multiple IAM roles to maintain access control.
