cancel
Showing results for 
Search instead for 
Did you mean: 
Warehousing & Analytics
Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Community. Share insights, tips, and best practices for leveraging data for informed decision-making.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Shaimaa
by Visitor
  • 32 Views
  • 1 replies
  • 0 kudos

Running SQL queries against a parquet folder in S3

I need to run sql queries against a parquet folder in S3. I am trying to use "read_files" but sometimes my queries fail due to errors while inferring the schema and sometimes without a specified reason. Sample query:  SELECT SUM(CASE WHEN match_resu...

  • 32 Views
  • 1 replies
  • 0 kudos
Latest Reply
shan_chandra
Esteemed Contributor
  • 0 kudos

 @Shaimaa  - you can divide the query into a nested query to first select all the fields from the s3 by enforcing the schema and build a nested query on top of the below example query (not syntax verified) SELECT * FROM STREAM read_files( 's3...

  • 0 kudos
VKe
by New Contributor
  • 28 Views
  • 0 replies
  • 0 kudos

install Oracle Instant Client

Hello all,I want to install Oracle Instant Client to be able to use python-oracledb in Thick-Mode because one of our databases is old and cannot be reached in Thin-Mode. I have tried the solution from this post, but it doesn't help me. It seems that ...

  • 28 Views
  • 0 replies
  • 0 kudos
acurus
by New Contributor II
  • 1526 Views
  • 3 replies
  • 2 kudos

Not able to escape `-` in external connected tablename

We are having some issues with getting data from some tables with the character `-` in their tablename.We are connected to the database with an SQL server connection, and the database is (as far as we know)  a Microsoft Azure SQL Database. We do not ...

  • 1526 Views
  • 3 replies
  • 2 kudos
Latest Reply
martinschou
New Contributor
  • 2 kudos

Had the same issue when querying a table with the - character in the table name. Got the error: Incorrect syntax near '-'.Got the error on Databricks runtime version: 13.2 (includes Apache Spark 3.4.0, Scala 2.12)No error when using Databricks runtim...

  • 2 kudos
2 More Replies
ssequ
by New Contributor
  • 283 Views
  • 3 replies
  • 1 kudos

Resolved! Backspaces in Foreign Catalog Table Names

Hi,is there a way to import tables with UC disallowed characters when creating a foreign catalog? The database we are dealing with contains table names with backspaces and UC seems to completely ignore them when creating the catalog.Thanks for any he...

  • 283 Views
  • 3 replies
  • 1 kudos
Latest Reply
daniel_sahal
Esteemed Contributor
  • 1 kudos

@ssequ I assume that you're refering to lakehouse federation? Unfortunately that's a limitation, tables with disallowed characters will be ignored.

  • 1 kudos
2 More Replies
vishva-fivetran
by New Contributor
  • 89 Views
  • 2 replies
  • 0 kudos

Databricks query on a warehouse failing with ` spark.driver.maxResultSize` error

we are trying to run a select * from one of our catalog tables. we are seeing the error : org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized r...

  • 89 Views
  • 2 replies
  • 0 kudos
Latest Reply
anardinelli
New Contributor III
  • 0 kudos

Hello @vishva-fivetran how are you? To set the spark.driver.maxResultSize property, you can do so in the cluster Spark config. The property spark.driver.maxResultSize can be set to a value higher than the value reported in the exception message. For ...

  • 0 kudos
1 More Replies
Brammer88
by New Contributor III
  • 97 Views
  • 2 replies
  • 1 kudos

How do I setup a connection to get data from unity catalog into MS Access?

Dear community,Im trying to setup a connection to get data from a catalog / underlying tables from that catalog and load it directly into MS Access 365 (latest version). Is it possible to do this? What would be the best practice? Thanks,Bram

  • 97 Views
  • 2 replies
  • 1 kudos
Latest Reply
Brammer88
New Contributor III
  • 1 kudos

Thanks, yes we are using powerbi too, but we have a specific requirement for which we should be able to connect directly to our legacy MS access databases too. 

  • 1 kudos
1 More Replies
Akshay_Petkar
by New Contributor III
  • 126 Views
  • 1 replies
  • 1 kudos

Resolved! Is Photon Enabled by Default for Warehouses in Databricks?

When creating a cluster in Databricks, I notice an option for enabling Photon. However, this option does not appear when configuring a warehouse. Is Photon enabled by default for warehouses in Databricks, or is there a different procedure to enable i...

  • 126 Views
  • 1 replies
  • 1 kudos
Latest Reply
Yeshwanth
Honored Contributor
  • 1 kudos

@Akshay_Petkar good day! Photon is enabled by default for Databricks SQL warehouses. There is no need for a separate procedure to enable it for warehouses. For clusters, you have the option to manually enable or disable Photon by selecting the "Use P...

  • 1 kudos
DominikBraun
by New Contributor II
  • 190 Views
  • 2 replies
  • 1 kudos

Resolved! SQL Warehouse: INVALID_PARAMETER_VALUE when starting

Hey everybody.When creating a SQL Warehouse and trying to start it, I get the following error message:Clusters are failing to launch. Cluster launch will be retried.Request to create a cluster failed with an exception: INVALID_PARAMETER_VALUE: Cannot...

  • 190 Views
  • 2 replies
  • 1 kudos
Latest Reply
Ismael-K
New Contributor III
  • 1 kudos

One suggestion would be if you could please have the workspace admin check the Data Access Configuration properties in the Workspace Settings for any secrets that may be held which the warehouse is trying to access on start up? These data access prop...

  • 1 kudos
1 More Replies
Hubert-Dudek
by Esteemed Contributor III
  • 3037 Views
  • 2 replies
  • 0 kudos

Optimizing SQL Databricks Warehouse Timeout Settings

Did you know the default timeout setting for SQL #databricks Warehouse is two days?The default timeout can be too long for most use cases. You can easily change this for your session or in the general SQL warehouse configuration.

ezgif-1-a917957651.gif
  • 3037 Views
  • 2 replies
  • 0 kudos
Latest Reply
Hertz
New Contributor II
  • 0 kudos

If the default is set to 2 days why would my tableau extract refresh jobs that utilize and sql warehouse time out due to inactivity after 10 or 20 mins.

  • 0 kudos
1 More Replies
Akshay_Petkar
by New Contributor III
  • 334 Views
  • 1 replies
  • 0 kudos

Resolved! Databricks warehouse cost calculation

I would like to know how the cost for a warehouse in Databricks is calculated. Specifically, if the cost for a 2x small warehouse is 4 DBUs per hour, how is the cost determined if I use the warehouse for only 30 minutes and then terminate it? Will it...

  • 334 Views
  • 1 replies
  • 0 kudos
Latest Reply
daniel_sahal
Esteemed Contributor
  • 0 kudos

@Akshay_Petkar As it states in Databricks documentation:Databricks offers you a pay-as-you-go approach with no up-front costs. Only pay for the products you use at per second granularity.It will charge 2 DBUs.

  • 0 kudos
Akshay_Petkar
by New Contributor III
  • 196 Views
  • 1 replies
  • 0 kudos

Facing Data Truncation Issues in Databricks Dashboards

I'm encountering data truncation in my Databricks dashboards. I'm working with a large dataset, and the dashboard only displays a limited number of (truncated) rows.let's take a dataset containing 1 million sales records. The dashboard currently only...

  • 196 Views
  • 1 replies
  • 0 kudos
Latest Reply
NandiniN
Honored Contributor
  • 0 kudos

Hi @Akshay_Petkar , This is simple in Databricks SQL, just uncheck LIMIT 1000 in the drop down. https://docs.databricks.com/en/sql/get-started/visualize-data-tutorial.html Limit 1000 is selected by default for all queries to ensure that the query ret...

  • 0 kudos
andre_rizzatti
by New Contributor II
  • 614 Views
  • 1 replies
  • 1 kudos

Resolved! SQLWarehouse Case INsensitive

Good morning, is there any parameter or configuration that causes all my data to be consulted without case distinction? insensitive?

  • 614 Views
  • 1 replies
  • 1 kudos
Latest Reply
raphaelblg
Contributor III
  • 1 kudos

Hello @andre_rizzatti,  At the current moment SQL Warehouses don't have this option. This feature is coming soon.  

  • 1 kudos
amelia1
by New Contributor II
  • 299 Views
  • 1 replies
  • 0 kudos

Local pyspark read data using jdbc driver returns column names only

Hello,I have an Azure sql warehouse serverless instance that I can connect to using databricks-sql-connector. But, when I try to use pyspark and jdbc driver url, I can't read or write.See my code belowdef get_jdbc_url(): # Define your Databricks p...

  • 299 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @amelia1, The error messages you provided indicate that there might be a problem with the log4j configuration and formatting. Additionally, the repeated column names suggest that there might be an issue with how the data is being retrieved. Her...

  • 0 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels
Top Kudoed Authors