cancel
Showing results for 
Search instead for 
Did you mean: 
Warehousing & Analytics
Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Community. Share insights, tips, and best practices for leveraging data for informed decision-making.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Warehousing, Analytics, and BI

Forum Posts

MadelynM
by Databricks Employee
  • 932 Views
  • 0 replies
  • 0 kudos

[Recap] Data + AI Summit 2024 - Warehousing & Analytics | Improve performance and increase insights

Here's your Data + AI Summit 2024 - Warehousing & Analytics recap as you use intelligent data warehousing to improve performance and increase your organization’s productivity with analytics, dashboards and insights.  Keynote: Data Warehouse presente...

Screenshot 2024-07-03 at 10.15.26 AM.png
Warehousing & Analytics
AI BI Dashboards
AI BI Genie
Databricks SQL
  • 932 Views
  • 0 replies
  • 0 kudos
youssefmrini
by Databricks Employee
  • 9421 Views
  • 1 replies
  • 1 kudos

Dataricks asset bundles is now Public preview

Databricks asset bundles, now in Public Preview, enable end-to-end data, analytics, and ML projects to be expressed as a collection of source files. This makes it simpler to apply data engineering best practices such as source control, code review, t...

Warehousing & Analytics
CICD
MachineLearning
  • 9421 Views
  • 1 replies
  • 1 kudos
Latest Reply
avrm91
Contributor
  • 1 kudos

Will a Databricks Asset Bunldes (DAB) implementation take place in the cloud (Browser - Workspace)?Personally, I work on a developer VM in a closed network where it is very difficult to install software (Databricks CLI) because it has to be requested...

  • 1 kudos
Linglin
by New Contributor III
  • 3070 Views
  • 3 replies
  • 0 kudos

Resolved! Add Visualization in Notebook to Dashboard, how to set default add to Dashboard Bottom

I'm creating dashboard with multiple visualizations from a notebook.Whenever I add a new visualization, the default position in dashboard is top left which mess up all the format I did for previous graph. Is there a way to default add to the bottom o...

  • 3070 Views
  • 3 replies
  • 0 kudos
Latest Reply
Linglin
New Contributor III
  • 0 kudos

@shan_chandra I'm using Lakeview dashboard. In dbx notebook, there is Add to dashboard > button on the right of each visualization. It's super handy. Actually I have this issue solved. 

  • 0 kudos
2 More Replies
Sudheer2
by New Contributor III
  • 1658 Views
  • 2 replies
  • 2 kudos

Updating SQL Warehouse using Terraform

Manual ApproachWe can Update SQL Warehouse manually in Databricks.Click SQL Warehouses in the sidebarIn Advanced optionsWe can find Unity Catalog toggle button there! While Updating Existing SQL Warehouse in Azure to enable unity catalog using terraf...

warehouse error.png
  • 1658 Views
  • 2 replies
  • 2 kudos
Latest Reply
Sudheer2
New Contributor III
  • 2 kudos

Hello Raphael,Thank you for the update and for looking into the feature request. I appreciate your efforts in following up on this matter.If possible, could you please provide me with any updates or insights you receive from the Terraform team regard...

  • 2 kudos
1 More Replies
harripy
by New Contributor III
  • 1259 Views
  • 1 replies
  • 0 kudos

AWS Databricks SQL to support TABLE rename

We have been evaluating Databricks SQL and its capability to be used as DW. We are using Unity catalog in our implementation.There seems to be a functionality mismatch between Azure and AWS versions as where table rename is supported on Azure side, i...

  • 1259 Views
  • 1 replies
  • 0 kudos
Latest Reply
harripy
New Contributor III
  • 0 kudos

This seems to be fixed now

  • 0 kudos
Billycheng
by New Contributor
  • 2141 Views
  • 1 replies
  • 0 kudos

QuotaExceeded

Hi, I am a newbie. Can someone please help me with the below? Details for the latest failure: Error: Error code: QuotaExceeded, error message: Operation could not be completed as it results in exceeding approved standardEDSv4Family Cores quota. Addit...

  • 2141 Views
  • 1 replies
  • 0 kudos
Latest Reply
Elenorelen
New Contributor II
  • 0 kudos

Thanks for the information

  • 0 kudos
AWRIGHT
by New Contributor III
  • 4669 Views
  • 5 replies
  • 2 kudos

Silver layer when tracking SCD2

HiI’m new to data modelling so could use some help.I’m building a personal project using a fairly standard 3NF sales database as the source data. So far I have a pipeline that incrementally extracts data from the source system each day into a Raw sto...

  • 4669 Views
  • 5 replies
  • 2 kudos
Latest Reply
AWRIGHT
New Contributor III
  • 2 kudos

Thank you.From a high level it makes sense. It's the details I'm a bit unclear on. So when a single pipeline runs (let's say today - 10/05/2024) for your architecture it might: Extract records from source system tables that have changed since last ti...

  • 2 kudos
4 More Replies
RobertAnderson6
by New Contributor
  • 946 Views
  • 1 replies
  • 0 kudos

Support for JDBC tables in SQL endpoint.

Hello,I'm wondering if there's a method or workaround to execute JDBC table queries in a similar manner to other cluster types. Currently, attempting to do so results in an error stating that only text-based files (such as JSON, Parquet, Delta, etc.)...

  • 946 Views
  • 1 replies
  • 0 kudos
Latest Reply
adurand-accure
New Contributor III
  • 0 kudos

Could you explain more what you are trying to achieve ? 

  • 0 kudos
shanebo425
by New Contributor III
  • 4177 Views
  • 4 replies
  • 0 kudos

Resolved! Grant Unity Catalog Access without Workspace Access

We have created a Unity Catalog instance on top of our Lakehouse (built entirely with Azure Databricks). We are using Power BI to develop and serve our analytics and reporting needs. I've granted the "Account Users" group the appropriate privileges f...

Warehousing & Analytics
Databricks
permissions
Unity Catalog
  • 4177 Views
  • 4 replies
  • 0 kudos
Latest Reply
shanebo425
New Contributor III
  • 0 kudos

Thanks for explaining this! This doesn't do exactly what I was hoping—it doesn't block all access to the workspace. Users can still login and access their own workspace and run SQL queries, explore the catalog, etc. But they ARE blocked from accessin...

  • 0 kudos
3 More Replies
DataFarmer
by New Contributor II
  • 1155 Views
  • 2 replies
  • 1 kudos

Data Warehouse in Databricks Date values as date or int: what is recommended?

In  relational data warehouse systems it was best practise to represent date values as YYYYMMDD integer type values in tables. Date comparison could be done easily without using date-functions and with low performance impact.Is this still the recomme...

  • 1155 Views
  • 2 replies
  • 1 kudos
Latest Reply
Ajay-Pandey
Esteemed Contributor III
  • 1 kudos

Hi @DataFarmer I Databricks I will advise you to use date type instead of int, this will make your life much simpler while working on the date type data.

  • 1 kudos
1 More Replies
JustinM
by New Contributor II
  • 4731 Views
  • 3 replies
  • 1 kudos

Cannot connect to SQL Warehouse using JDBC connector in Spark

When trying to connect to a SQL warehouse using the JDBC connector with Spark the below error is thrown. Note that connecting directly to a cluster with similar connection parameters works without issue, the error only occurs with SQL Warehouses.py4j...

  • 4731 Views
  • 3 replies
  • 1 kudos
Latest Reply
jmms
New Contributor II
  • 1 kudos

Same error here, I am trying to save spark dataframe to Delta lake using JDBC driver and pyspark using this code:#Spark session spark_session = SparkSession.builder \ .appName("RCT-API") \ .config("spark.metrics.namespace", "rct-a...

  • 1 kudos
2 More Replies
Kroy
by Contributor
  • 1554 Views
  • 3 replies
  • 2 kudos

Not able to create SQL warehouse cluster in free subscription

I have taken a free subscription to azure databricks, but when try to create 2x small warehouse clusture, getting below error, help appreciated. 

Kroy_0-1702694045718.png
  • 1554 Views
  • 3 replies
  • 2 kudos
Latest Reply
TimJB
New Contributor II
  • 2 kudos

Can somebody please answer this? I'm having the same issue. 

  • 2 kudos
2 More Replies
florent
by New Contributor III
  • 4035 Views
  • 6 replies
  • 6 kudos

Resolved! it's possible to deliver a sql dashboard created in a Dev workspace to a Prod workspace?

In order to create a ci/cd pipeline to deliver dashboards (here monitoring), how to export / import a dashboard created in databricks sql dashboard from one workspace to another?Thanks

  • 4035 Views
  • 6 replies
  • 6 kudos
Latest Reply
miranda_luna_db
Databricks Employee
  • 6 kudos

Recommendation is to update your legacy dashboard to Lakeview and then leverage built in export/import support.

  • 6 kudos
5 More Replies
Kaizen
by Valued Contributor
  • 951 Views
  • 1 replies
  • 1 kudos

Command to display all computes available in your workspace

Hi Is there a command you could use to list all computes configured in your workspace (active and non-active).  This would be really helpful for anyone managing the platfrom to pull all the meta data (tags ,etc) and quickly evaluate all the configura...

  • 951 Views
  • 1 replies
  • 1 kudos
Latest Reply
daniel_sahal
Esteemed Contributor
  • 1 kudos

@Kaizen You've got three ways of doing this:- Using REST API (https://docs.databricks.com/api/workspace/clusters/list),- Using CLI (https://github.com/databricks/cli/blob/main/docs/commands.md#databricks-clusters-list---list-all-clusters)- Using Pyth...

  • 1 kudos
pankaj2264
by New Contributor II
  • 2132 Views
  • 2 replies
  • 1 kudos

Using profile_metrics and drift_metrics

Is there any business use-case where profile_metrics and drift_metrics are used by Databricks customers.If so,kindly provide the scenario where to leverage this feature e.g data lineage,table metadata updates.

  • 2132 Views
  • 2 replies
  • 1 kudos
Latest Reply
MohsenJ
Contributor
  • 1 kudos

hey @pankaj2264. both profile metric and drift metric tables are created and used by Lakehouse monitoring to assess the performance of your model and data over time or relative to a baseline table. you can find all the relevant information here Intro...

  • 1 kudos
1 More Replies
rocky5
by New Contributor III
  • 2901 Views
  • 1 replies
  • 0 kudos

Resolved! Incorrect results of row_number() function

I wrote simple code:from pyspark.sql import SparkSession from pyspark.sql.window import Window from pyspark.sql.functions import row_number, max import pyspark.sql.functions as F streaming_data = spark.read.table("x") window = Window.partitionBy("BK...

  • 2901 Views
  • 1 replies
  • 0 kudos
Latest Reply
ThomazRossito
Contributor
  • 0 kudos

Hi,In my opinion the result is correctWhat needs to be noted in the result is that it is sorted by the "Onboarding_External_LakehouseId" column so if there is "BK_AccountApplicationId" with the same code, it will be partitioned into 2 row_numbersJust...

  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels
Top Kudoed Authors