cancel
Showing results for 
Search instead for 
Did you mean: 
Warehousing & Analytics
Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Community. Share insights, tips, and best practices for leveraging data for informed decision-making.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

MadelynM
by Databricks Employee
  • 1104 Views
  • 0 replies
  • 0 kudos

[Recap] Data + AI Summit 2024 - Warehousing & Analytics | Improve performance and increase insights

Here's your Data + AI Summit 2024 - Warehousing & Analytics recap as you use intelligent data warehousing to improve performance and increase your organization’s productivity with analytics, dashboards and insights.  Keynote: Data Warehouse presente...

Screenshot 2024-07-03 at 10.15.26 AM.png
Warehousing & Analytics
AI BI Dashboards
AI BI Genie
Databricks SQL
  • 1104 Views
  • 0 replies
  • 0 kudos
Akshay_Petkar
by Contributor II
  • 2624 Views
  • 5 replies
  • 1 kudos

Resolved! Migrate Azure Synapse Analytics data to Databricks

I have to migrate the data from Azure Synapse Analytics to Databricks. Could anyone share the different approaches to migrate data, and from those, which is the best approach to use?

  • 2624 Views
  • 5 replies
  • 1 kudos
Latest Reply
raushan_kumar
New Contributor II
  • 1 kudos

Can we mount the Syanpse storage account ADLS Gen2 directly to Databricks and then we can only  migrate pipeline/code to databricks

  • 1 kudos
4 More Replies
VKe
by New Contributor III
  • 3179 Views
  • 4 replies
  • 0 kudos

install Oracle Instant Client

Hello all,I want to install Oracle Instant Client to be able to use python-oracledb in Thick-Mode because one of our databases is old and cannot be reached in Thin-Mode. I have tried the solution from this post, but it doesn't help me. It seems that ...

  • 3179 Views
  • 4 replies
  • 0 kudos
Latest Reply
NelsonE
New Contributor III
  • 0 kudos

What about shared compute, is it possible? Neither of those scripts is working for me really...

  • 0 kudos
3 More Replies
KosmaS
by New Contributor III
  • 163 Views
  • 1 replies
  • 2 kudos

SQL Warehouse - Turning off caching

Hi All,I'd like to do some benchmarking and I need to turn off caching on my SQL Warehouse Server. However, whatever I try I receive quite high level of caching after running my queries (>60%). I tried to turn off my server, but it automatically wake...

  • 163 Views
  • 1 replies
  • 2 kudos
Latest Reply
Walter_C
Databricks Employee
  • 2 kudos

To disable query result caching, you can run SET use_cached_result = false in the SQL editor.

  • 2 kudos
jay99
by New Contributor II
  • 542 Views
  • 4 replies
  • 1 kudos

Create multiple SQL warehouse with custom access control

We set up the SQL warehouse IAM role in the settings option. This is applied to all warehouses. How do I create sql warehouses with multiple IAM roles to maintain access control.

  • 542 Views
  • 4 replies
  • 1 kudos
Latest Reply
Walter_C
Databricks Employee
  • 1 kudos

Unfortunately there is no way to restrict the access the compute has, the restrictions are being performed via the users permissions. Only option here will be to submit a feature request through https://docs.databricks.com/en/resources/ideas.html#ide...

  • 1 kudos
3 More Replies
Akshay_Petkar
by Contributor II
  • 1481 Views
  • 7 replies
  • 5 kudos

Need a Sample MERGE INTO Query for SCD Type 2 Implementation

Can anyone provide a sample MERGE INTO SQL query for implementing SCD Type 2 in Databricks using Delta Tables?

  • 1481 Views
  • 7 replies
  • 5 kudos
Latest Reply
bhanu_gautam
Contributor III
  • 5 kudos

@JissMathew  and @David_Torrejon , Thanks for sharing the example

  • 5 kudos
6 More Replies
Akshay_Petkar
by Contributor II
  • 420 Views
  • 6 replies
  • 7 kudos

How to Complete CDW Migration Best Practices Training Labs on Databricks Learning Portal?

I am going through the labs for the CDW Migration Best Practices training on the Databricks Learning Portal and scored 75%. However, it still shows as "In Progress."Can anyone explain the criteria for marking the labs as complete? Do I need a higher ...

  • 420 Views
  • 6 replies
  • 7 kudos
Latest Reply
vikram2406
New Contributor III
  • 7 kudos

Yes, it typically takes 10-15 minutes to generate a grade after checking the labs.

  • 7 kudos
5 More Replies
chari
by Contributor
  • 6662 Views
  • 3 replies
  • 2 kudos

What is databricks SQL, spark SQL and how are they different from MS SQL ?

Hello Databricks Community,I have a hard time understanding how is Databricks SQL different from microsoft SQL ? Also, why does databricks provide spark SQL ? If you direct me to a well-written webpage or document its of immense help!Thanks,

  • 6662 Views
  • 3 replies
  • 2 kudos
Latest Reply
Rahul_Saini
New Contributor II
  • 2 kudos

Databricks SQL and Spark SQL are built for distributed big data analytic. Databricks SQL is great for business intelligence tools and uses Delta Lake for efficient data storage. Spark SQL works with Spark's programming features for data processing. U...

  • 2 kudos
2 More Replies
nomnomnom543
by New Contributor II
  • 639 Views
  • 3 replies
  • 2 kudos

Databricks SQL Wildcard Operator Not Parsed Correctly

Hello there,Wasn't sure if this was just an error on my part, but I'm using a Databricks Pro SQL warehouse and unity catalogue to pull some data from my tables. I'm having this issue where whenever I try and use a wildcard operator with my LIKE claus...

  • 639 Views
  • 3 replies
  • 2 kudos
Latest Reply
Rahul_Saini
New Contributor II
  • 2 kudos

Hi @nomnomnom543 ,Try this SELECT * FROM table_name WHERE LEFT(column_name, LENGTH('string')) = 'string';

  • 2 kudos
2 More Replies
amelia1
by New Contributor II
  • 1501 Views
  • 1 replies
  • 0 kudos

Local pyspark read data using jdbc driver returns column names only

Hello,I have an Azure sql warehouse serverless instance that I can connect to using databricks-sql-connector. But, when I try to use pyspark and jdbc driver url, I can't read or write.See my code belowdef get_jdbc_url(): # Define your Databricks p...

  • 1501 Views
  • 1 replies
  • 0 kudos
Latest Reply
NandiniN
Databricks Employee
  • 0 kudos

The error does not look specific to the warehouse that you are connecting to. The error message "Unrecognized conversion specifier [msg] starting at position 54 in conversion pattern" indicates that there is an issue with the logging configuration in...

  • 0 kudos
harveychun
by New Contributor II
  • 1350 Views
  • 3 replies
  • 0 kudos

Measures or KPI

With Databricks BI, Is there anyway to create a KPI or Measures that can be used in a visual? If so, how to achieved that one?

  • 1350 Views
  • 3 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Can you please provide some more context on your current usecase?

  • 0 kudos
2 More Replies
MightyMasdo
by New Contributor II
  • 936 Views
  • 4 replies
  • 2 kudos

writing into delta table using databricks connect

Hello we are currently facing a challenge with writing data from a local machine to Delta tables. While small data loads (e.g., 100 Rows) work without any issues, attempting to write larger batches (around 1,000 Rows) results in an exception. the iss...

  • 936 Views
  • 4 replies
  • 2 kudos
Latest Reply
vr
Contributor
  • 2 kudos

Hi,Can you provide a sample code to reproduce the issue?

  • 2 kudos
3 More Replies
OfirM
by New Contributor
  • 308 Views
  • 1 replies
  • 0 kudos

spark.databricks.optimizer.replaceWindowsWithAggregates.enabled

I have seen in the release notes of 15.3 that this was introduced and couldn't wrap my head around it.Does someone has an example of a plan before and after?Quote:Performance improvement for some window functionsThis release includes a change that im...

  • 308 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Before Optimization: Consider a query that calculates the sum of a column value partitioned by category without an ORDER BY clause or a window_frame parameter:   SELECT category, SUM(value) OVER (PARTITION BY category) AS total_value FROM sales;  ...

  • 0 kudos
rk2511
by New Contributor
  • 469 Views
  • 1 replies
  • 0 kudos

Access Each Input Item of a For Each Task

I have two tasks. The first task (Sample_Notebook) returns a JSON array (Input_List). Sample data in Input_List['key1':value1, 'key2':value2, 'key3':value3]The second task is a "For Each" task that executes a notebook for each entry in the Input_List...

  • 469 Views
  • 1 replies
  • 0 kudos
Latest Reply
BigRoux
Databricks Employee
  • 0 kudos

To access each item of the iteration within the notebook of the second task in your Databricks workflow, you need to utilize the parameterization feature of the For Each task. Instead of trying to retrieve the entire list using dbutils.jobs.taskValue...

  • 0 kudos
AnaMocanu
by New Contributor III
  • 478 Views
  • 2 replies
  • 1 kudos

Streamlit app on Databricks doesn't recognise the DATABRICKS_WAREHOUSE_ID in the yaml file

Hey everyone,So I managed to create a Streamlit app on Databricks, works fine deployed in the cloud.However, when I try to run it locally, it complains about "assert os.getenv('DATABRICKS_WAREHOUSE_ID'), "DATABRICKS_WAREHOUSE_ID must be set in app.ya...

  • 478 Views
  • 2 replies
  • 1 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 1 kudos

Hi @AnaMocanu, How are you setting up: DATABRICKS_WAREHOUSE_ID? When running the app locally, the environment variables set in the app.yaml file might not be picked up automatically. You can manually set the environment variable in your local environ...

  • 1 kudos
1 More Replies
hank12345
by New Contributor
  • 601 Views
  • 2 replies
  • 0 kudos

Resolved! Lakehouse federation support for Oracle DB

https://docs.databricks.com/en/query-federation/index.htmlAre there plans to provide Oracle support for Databricks on AWS lakehouse federation? Not sure if that's possible or not. Thanks!

  • 601 Views
  • 2 replies
  • 0 kudos
Latest Reply
PiotrU
Contributor II
  • 0 kudos

Federation with Oracle is available https://learn.microsoft.com/en-us/azure/databricks/query-federation/oracle

  • 0 kudos
1 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels
Top Kudoed Authors