Here's your Data + AI Summit 2024 - Warehousing & Analytics recap as you use intelligent data warehousing to improve performance and increase your organization’s productivity with analytics, dashboards and insights. Keynote: Data Warehouse presente...
I am currently trying to write a dataframe to s3 likedf.write.partitionBy("col1","col2").mode("overwrite").format("json").save("s3a://my_bucket/")The path becomes `s3a://my_bucket/col1=abc/col2=opq/`But I want to path to be `s3a://my_bucket/abc/opq/`...
Hi @Jennifer , The default behavior of the .partitionBy() function in Spark is to create a directory structure with partition column names. This is similar to Hive's partitioning scheme and is done for optimization purposes. Hence, you cannot directl...
I am in the process of connecting Looker to one of my Databricks databases. To reduce startup time on my SQL warehouse cluster I would like to change the type from "Pro" to "Serverless". I cannot find a way to do that and "Serverless" is not an optio...
Echoing glawry - I'd be fascinated to know if this "Ephemeral clusters" are a thing.
Hi Khishore,Please help me how you raised ticket for the certificate issue.Thanks,Anuroop
Please submit a ticket to our Training Team here: https://help.databricks.com/s/contact-us?ReqType=training
Hi,I am trying to connect to databricks from tableau server and facing this error OAuth error response, generally means someone clicked cancel: access_denied (errorCode=180002)I have added it in "app connections" under account console. Any pointers w...
Hello, We have a workspace in West Europe, which we were using with multiple Serverless SQL Warehouses. Recently it suddenly stopped working and presented with the following error when trying to restart an existing serverless warehouse. I am an adm...
Raised a ticket with Azure support but they haven't been very helpful so far to be honest. No uncleared bills and serverless seems disabled for all workspaces in the subscription. Someone mentioned accepting terms and conditions for serverless comp...
Hi All!Has anyone encountered a situation where we need to setup data access for Unity Catalog tables for read access such as external data marts, dashboard tools and etc.We are currently using Databricks to serve data to people in our organisation t...
I see there’s a “test” capability within a DAB, but I’d like to know more about how this should/could be used. Does anyone know of any documentation or examples which might provide insights into its intended use?
Hi @EWhitley You can check and validate whether the Asset Bundle configuration is valid or not by using the below command databricks bundle validate If a JSON representation of the bundle configuration is returned, then the validation succeed...
While setting up metastore in GCP Databricks, I added the bucket name and then service account permissons as well. Still my catalog dont have base root location. This deters me from creating table in my catalog. Root storage credential for metastore...
Hi @manish05485 , Good Day! Error:Root storage credential for metastore XXXXXX does not exist. Please contact your Databricks representative or consider updating the metastore with a valid storage Error states that the data access configuration for ...
Hi,I cannot see the query execution time in the response to the "api/2.0/sql/history/queries" request.Basically, I get only the following fields:{"next_page_token":...,"has_next_page":...,"res":[ { "query_id":..., "status":.., "query_tex...
Spot on @feiyun0112 So this confirms that the API is working as expected right?
Watch the youtube video : https://www.youtube.com/watch?v=MO7Dk035654
I am attempting to recreate a legacy dashboard in Lakeview. The bar graph in no way resembles what I created in the SQL visualization editor. Lakeview has far fewer formatting options for one thing.How do I recreate the graph so that it resembles the...
Hi Team,I have a delta table in databricks which contains a encrypted column. For encrypting I am using databricks "aes_encrypt" function. For reference: https://docs.databricks.com/en/sql/language-manual/functions/aes_encrypt.html#aes_encrypt-functi...
you can create two cloumns , and display base on userDisplay different columns in Power BI based on logged in user | Paige Liu’s Posts (liupeirong.github.io)
I completed the Data Engineering Lakehouse course and I am familiar with different objects and concepts of databricks and lakehouse but I cant tie them together in my mind.Where do you typically use managed and non-managed tables? Bronze layer? Or no...
Hi @sjmb , In the Medallion architecture, the usage of managed and non-managed tables, autoloader/cloud files, and the apply changes into function can vary depending on the layer (Bronze, Silver, Gold) and the specific use case. As a general rule, us...
We found a post on LinkedIn that revealed if "this table is deprecated" is added to a table comment, the table will appear with a strikethrough in notebooks and SQL editor windows. Is this functionality GA? Is there any documentation on the use of ...
Snowflake vs Databricks SQL Endpoint for Datawarehousing which is more persistent
Databricks and Snowflake are both powerful platforms designed to address different aspects of data processing and analytics. Databricks shines in big data processing, machine learning, and AI workloads, while Snowflake excels in data warehousing, sto...
Hi, am sure am missing something as this should be something trivial but am struggling to find how to add a suffix with a date to a table name.Does anyone have a way to do this?Thanks
Hi @BobDobalina - Dynamic naming of table name is not allowed in DBSQL. However, you can try something similar %python from datetime import datetime date_suffix = datetime.now().strftime("%Y%m%d") table_name = f"students{date_suffix}" spark.sql(f"CR...
