Hi all! The Databricks Looker Studio connector has now been available for a few weeks. Tested the connector but running into several issues: I am used to working with dynamic queries, so I am able to use date parameters (similar to BigQuery Looker St...
Hi @Kaniz Hope you're doing well! I am very curious about the following thing: However, there might be workarounds or alternative approaches to achieve similar functionality. You could explore using Looker’s native features for dynamic filtering or c...
Im trying to set up a connection to Iceberg on S3 via Snowflake as described https://medium.com/snowflake/how-to-integrate-databricks-with-snowflake-managed-iceberg-tables-7a8895c2c724 and https://docs.snowflake.com/en/user-guide/tables-iceberg-catal...
Hi @Kaniz ,We've been working on setting up Glue as catalog, which is working fine so far. However, Glue takes place of the hive_metastore, which appears to be a legacy way of setting this up. Is the way proposed here the recommended way to set it up...
I'm just curious whether, in the future, Databricks will offer a certification for AI, GenAI, or any other AI-related fields. I'm very interested and looking forward to it.
Hi @jensen22, Thank you for posting your concern on Community!
To expedite your request, please list your concerns on our ticketing portal. Our support staff would be able to act faster on the resolution (our standard resolution time is 24-48 hours).
Hi,I want to remove duplicate rows from my managed delta table in my unity catalog. I use a query on a SQL warehouse similar to this: WITH cte AS (
SELECT
id, ROW_NUMBER() OVER (PARTITION BY id,##,##,## ORDER BY ts) AS row_num
FROM
catalog.sch...
I have first tried to use _metadata.row_index to delete the correct rows but also this resulted in an error. My solution was now to use spark and overwrite the table.table_name = "catalog.schema.table"
df = spark.read.table(table_name)
count_df = df....
Recently, it seems that there has been an intermittent issue where the output of a notebook cell doesn't display, even though the code within the cell executes successfully. For instance, there are times when simply printing a dataframe yields no out...
select {{user_defined_variable}} as my_var, count(*) as cntfrom my_tablewhere {{user_defined_variable}} = {{value}} for user_defined_variable, I use query based dropdown list to get a column_name I'd like ...
Hi @Linglin, It seems you’re dealing with user-defined variables in your SQL query, and you want to dynamically set both the column name and the value in your WHERE clause.
Let’s break down the solution:
Setting User-Defined Variables: You can s...
Hey,I've managed to add my SQL Warehouse as a data source in Pycharm using the JDBC driver and can query the warehouse from an SQL console within Pycharm. This is great, however, what I'm struggling with is getting the catalogs and schemas to show in...
You need to explicitly tell your JetBrains tool to introspect the database using JDBC metadata.I think the reason it (sometimes) works in Datagrip but not PyCharm, IntelliJ, etc is because the default settings can be different across tools and even v...
I am currently trying to write a dataframe to s3 likedf.write.partitionBy("col1","col2").mode("overwrite").format("json").save("s3a://my_bucket/")The path becomes `s3a://my_bucket/col1=abc/col2=opq/`But I want to path to be `s3a://my_bucket/abc/opq/`...
Hi @Jennifer ,
The default behavior of the .partitionBy() function in Spark is to create a directory structure with partition column names. This is similar to Hive's partitioning scheme and is done for optimization purposes. Hence, you cannot directl...
I am in the process of connecting Looker to one of my Databricks databases. To reduce startup time on my SQL warehouse cluster I would like to change the type from "Pro" to "Serverless". I cannot find a way to do that and "Serverless" is not an optio...
Hello everyone,I would like to inquire about the possibility of increasing the default limit of concurrent queries on the cluster which is set to 10.While researching this topic, I noticed that there is no official documentation available regarding t...
Hi @as5, Let’s delve into the details of increasing the concurrent queries limit for SQL warehouse clusters in Databricks.
Default Concurrent Queries Limit: By default, Databricks limits the number of concurrent queries per cluster assigned to a ...
Hello Databricks Community,I have a hard time understanding how is Databricks SQL different from microsoft SQL ? Also, why does databricks provide spark SQL ? If you direct me to a well-written webpage or document its of immense help!Thanks,
Hi @chari,
Certainly! Let’s delve into the differences between Databricks SQL and Microsoft SQL Server, as well as the rationale behind Spark SQL in Databricks.
Databricks SQL vs. Microsoft SQL Server:
Databricks SQL is an integral part of the ...
Hi,I am trying to use pm4py library to visualize my data. my code executes perfectly but there is no visualization how can i solve this problem ? There is no error message#processmining # databricks #pm4py
Hi @rushank29, Here are a few steps you can take to address the problem:
Check Dependencies:
Ensure that you have all the necessary dependencies installed. Sometimes missing or outdated packages can cause issues with visualization libraries.Verif...
Hi,I am trying to connect to databricks from tableau server and facing this error OAuth error response, generally means someone clicked cancel: access_denied (errorCode=180002)I have added it in "app connections" under account console. Any pointers w...
Hi @557879, The OAuth error response you’re encountering—specifically the “access_denied” error with errorCode=180002—typically indicates that the user cancelled the authorization process or denied requested permissions.
Let’s troubleshoot this issu...
