Here's your Data + AI Summit 2024 - Warehousing & Analytics recap as you use intelligent data warehousing to improve performance and increase your organization’s productivity with analytics, dashboards and insights. Keynote: Data Warehouse presente...
I see that cluster sizes are mentioned here https://docs.databricks.com/sql/admin/sql-endpoints.html#cluster-size but I would like to know when to pick what type of cluster (data size/ users/ concurrency) without having to do too much trial and error...
I have two jars with the same class name. It works fine on yarn. When trying to run these jars on the Databricks cluster, I run into issues. Why Databricks is having this limitation?
When you run the jobs in Yarn, those are 2 different applications getting submitted on Yarn. Hence each application will have a separate Spark driver JVM's. In Databricks, a cluster has one JVM for the Spark driver. When applications with the same na...
There is not a default timeout to prevent long running queries.
Does Koalas support Structured Streaming
No, Koalas does not support Structured Streaming officially.As a workaround, you can use Koalas APIs with foreachBatch in Structured Streaming which allows batch APIs:>>> def func(batch_df, batch_id): ... koalas_df = ks.DataFrame(batch_df) .....
So if you've got an S3 bucket with your data in it, the first thing you'll need to do is connect it to a Databricks workspace to grant access. Then you can start querying the contents of the bucket from notebooks (or running jobs) by using clusters (...
Frequency of logs in stdout / stderr would be a function of what code you run on the databricks clusters .The default log level for log4j is INFO - you could change it following the instructions here
How do i connect DBSQL to other BI tools?
Generally, you can connect to SQL endpoint using a ODBC or JDBC driver. More information can be found here. https://docs.databricks.com/integrations/bi/index-sqla.html
When sizing, this is the recommendation. Data set Cluster SizeITB / rows X-Large+500GB / 1B rows X-LargeSOGB / IOOM+ rows LargeIOOGB / rows MediumIOGB / -M rows SmallThis table maps SQL endpoint cluster sizes to Databricks cluster driver sizes and wo...
Second question. Yes. Just don't grant CAN_RUN to a user/grouphttps://docs.databricks.com/sql/user/security/access-control/dashboard-acl.html#dashboard-permissions
I'd like to be able to limit the rows users see when querying tables in Databricks SQL based on what access level each user is supposed to be granted. Is this possible in the SQL environment?
Using dynamic views you can specify permissions down to the row or field level e.g. CREATE VIEW sales_redacted AS SELECT user_id, country, product, total FROM sales_raw WHERE CASE WHEN is_member('managers') THEN TRUE ELSE total <= 1...
I see the option to enable Photon when creating a new SQL Endpoint. The description says that enabling it helps speed up up queries, which sounds good, but are there any downsides I need to be aware of?
Generally, yes you should enable photon. The majority of functionality is available and will perform extremely well. There are some limitations with it that can be found here. Limitations: Works on Delta and Parquet tables only for both read and writ...
If I want to get more information about how an individual query is performing in the Databricks SQL environment, is there anywhere I can see that?
You could see details on different queries that ran against an endpoint under the query history section
Need to know if I can use job clusters to start and run streaming jobs or if it has to be interactive
Yes. Here is a doc containing some info on running Structured Streaming in production using Databricks jobs
How to troubleshoot SSL certificate errors while connecting Business Intelligence (BI) tools to Databricks in a Private Cloud (PVC) environment?
See attached for steps to inspect the certificate chain using openssl
