cancel
Showing results for 
Search instead for 
Did you mean: 
Community Discussions
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

alano
by New Contributor
  • 130 Views
  • 1 replies
  • 0 kudos

Handling large volumes of streamed transactional data using DLT

We have a data stream from event hub with approximately 10 million rows per day (into one table) - these records are insert only (no update). We are trying to find a solution to aggregate / group by the data based on multiple data points and our requ...

  • 130 Views
  • 1 replies
  • 0 kudos
Latest Reply
artsheiko
Valued Contributor III
  • 0 kudos

Hi, please find below a set of resources I believe relevant for you. Success stories You can find the success stories of companies leveraging the streaming on Databricks here. Videos Introduction to Data Streaming on the Lakehouse : Structured Stream...

  • 0 kudos
chemajar
by New Contributor III
  • 440 Views
  • 3 replies
  • 1 kudos

Resolved! Rearrange tasks in databricks workflow

Hello,There is anyway to rearrange tasks in databricks workflow?.I would like that line that join the two marked tasks doesn't pass behind the other tasks.  It is posible that this line by one side?Thanks. 

image.png
  • 440 Views
  • 3 replies
  • 1 kudos
Latest Reply
artsheiko
Valued Contributor III
  • 1 kudos

Hi @chemajar, Take a look at Databricks Asset Bundles. It allows you to streamline the development of complex workflows using a yaml definition. In case you need to change the task dependencies, you can rearrange the flow as you need just change the ...

  • 1 kudos
2 More Replies
RahulChaubey
by New Contributor III
  • 401 Views
  • 3 replies
  • 0 kudos

Do we pay just for qurery run duration while using databricks serverless sql ?

While using databricks serverless sql to run queries does we only pay for the compute resources during the run duration of the query ?

  • 401 Views
  • 3 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @RahulChaubey ,    When using Databricks Serverless SQL, the pricing model is designed to be pay-as-you-go and is based on Databricks Units (DBUs).   Let me break it down for you: Serverless SQL allows you to run SQL queries for BI reporting,...

  • 0 kudos
2 More Replies
prasha123
by New Contributor II
  • 250 Views
  • 3 replies
  • 0 kudos

Unity Catalog view access in Azure Storage account

Hi,I have my unity catalog in Azure Storage account and I can able to access table objects but I couldn't find my views that were created on top of those table. 1. I can can access Delta tables & related views via Databricks SQL and also find the tab...

Community Discussions
Azure Storage Account
Delta views
Unity Catalog
  • 250 Views
  • 3 replies
  • 0 kudos
Latest Reply
artsheiko
Valued Contributor III
  • 0 kudos

Hi, Couple of options are possible :  Use Databricks to do the complex SQL queries (joins, unions, etc) and write to a staging Delta Table. Then use DataFlow to read from that staged table. Orchestrate all of this using ADF or even Databricks Workflo...

  • 0 kudos
2 More Replies
Carpender
by New Contributor II
  • 240 Views
  • 2 replies
  • 1 kudos

PowerBI Tips

Does anyone have any tips for using PowerBI on top of databricks? Any best practices you know of or roadblocks you have run into that should be avoided?Thanks.

  • 240 Views
  • 2 replies
  • 1 kudos
Latest Reply
artsheiko
Valued Contributor III
  • 1 kudos

Hey, Use Partner Connect to establish a connection to PBI Consider to use Databricks SQL Serverless warehouses for the best user experience and performance (see Intelligent Workload Management aka auto-scaling and query queuing, remote result cache, ...

  • 1 kudos
1 More Replies
Databricks_info
by New Contributor II
  • 270 Views
  • 4 replies
  • 0 kudos

Concurrent Update to Delta - Throws error

Team,I get a ConcurrentAppendException: Files were added to the root of the table by a concurrent update when trying to update a table which executes via jobs with for each activity in ADF,I tried with Databricks run time 14.x and set the delete vect...

  • 270 Views
  • 4 replies
  • 0 kudos
Latest Reply
artsheiko
Valued Contributor III
  • 0 kudos

Hey, This issue happens whenever two or more jobs try to write to the same partition for a table. This exception is often thrown during concurrent DELETE, UPDATE, or MERGE operations. While the concurrent operations may be physically updating differe...

  • 0 kudos
3 More Replies
RahulChaubey
by New Contributor III
  • 151 Views
  • 1 replies
  • 1 kudos

Resolved! Can we get SQL Serverless warehouses monitoring data using APIs or logs ?

I am looking for a possible way to get the autoscaling history data for SQL Serverless Warehouses using API or logs.I want something like what we see in monitoring UI.   

Screenshot 2024-04-15 at 6.52.29 PM.png
  • 151 Views
  • 1 replies
  • 1 kudos
Latest Reply
artsheiko
Valued Contributor III
  • 1 kudos

Hi Rahul, you need to perform two actions :  Enable system tables schema named "compute" (how-to, take a look on the page, it's highly possible that you'll find other schemas useful too)Explore system.compute.warehouse_events table Hope this helps. B...

  • 1 kudos
kankotan
by New Contributor II
  • 342 Views
  • 1 replies
  • 3 kudos

Regional Group Request for Istanbul

Hello, I kindly request the formation of a regional group for Istanbul/Turkey. I would appreciate your assistance in this matter.Thank you,Can

Community Discussions
Istanbul
Request
Turkey
  • 342 Views
  • 1 replies
  • 3 kudos
Latest Reply
Sujitha
Community Manager
  • 3 kudos

@kankotan Happy to help set it up for you. I have dropped an email for more information! 

  • 3 kudos
jensen22
by Contributor
  • 250 Views
  • 2 replies
  • 0 kudos

Method public void org.apache.spark.sql.internal.CatalogImpl.clearCache() is not whitelisted on clas

I'm executing a notebook and failed with this error: Sometime, when i execute some function in spark and also failed with the error 'this class is not whitelist'. Could everyone help me check on this?Thanks for your help!

Screenshot 2024-04-03 at 21.49.46.png
  • 250 Views
  • 2 replies
  • 0 kudos
Latest Reply
jensen22
Contributor
  • 0 kudos

Thanks for your feedback, actually my cluster is shared cluster, and then I change to single cluster then can run that method/

  • 0 kudos
1 More Replies
egndz
by New Contributor
  • 302 Views
  • 0 replies
  • 0 kudos

Cluster Memory Issue (Termination)

Hi,I have a single-node personal cluster with 56GB memory(Node type: Standard_DS5_v2, runtime: 14.3 LTS ML). The same configuration is done for the job cluster as well and the following problem applies to both clusters:To start with: once I start my ...

egndz_2-1712845742934.png egndz_1-1712845616736.png
  • 302 Views
  • 0 replies
  • 0 kudos