- 608 Views
- 3 replies
- 1 kudos
Incremental refresh - non serverless compute
Hey,I read the page about incremental refresh. Will you make it available on more than just serverless compute?If so, when?ThanksReference - https://docs.databricks.com/en/optimizations/incremental-refresh.html
- 608 Views
- 3 replies
- 1 kudos
- 1186 Views
- 2 replies
- 1 kudos
Resolved! Is dbutils.notebook.run() supported from a local Spark Connect environment (VS Code)?
Hi everyone,I’m experimenting with the Databricks VS Code extension, using Spark Connect to run code locally in my Python environment while connecting to a Databricks cluster. I’m trying to call one notebook from another via: notebook_params = { ...
- 1186 Views
- 2 replies
- 1 kudos
- 1 kudos
@filipniziol just curious to know if getting the context and setting it manually would help, have you tried this approach? Example: ctx = dbutils.notebook.entry_point.getDbutils().notebook().getContext() dbutils.notebook.setContext(ctx) Or from pyspa...
- 1 kudos
- 335 Views
- 1 replies
- 0 kudos
Rest API version 1
How long rest api in version 1 will be support ?
- 335 Views
- 1 replies
- 0 kudos
- 0 kudos
There is no mentioned of support duration for Databricks REST API version 1. I can check internally. Do you have any API in specific?
- 0 kudos
- 996 Views
- 3 replies
- 1 kudos
Databricks+DBT best practices
Hi All,could you provide the best practices for building and optimizing DBT models in databricks.Regards,Phani
- 996 Views
- 3 replies
- 1 kudos
- 1 kudos
@Phani1 Please let us know if after going through @szymon_dybczak references, you still need some guidance on more specific aspects that we can help with.
- 1 kudos
- 1430 Views
- 11 replies
- 0 kudos
Resolved! how to read the CDF logs in DLT Pipeline?
Hi Community,How to read the CDF logs in materialized views created by DLT Pipeline?Thanks for you time,
- 1430 Views
- 11 replies
- 0 kudos
- 0 kudos
@jeremy98 correct, If permissions management is complex, consider using standard Delta tables with CDF enabled and orchestrate changes through Databricks Workflows. This approach simplifies collaboration and avoids issues with restricted internal sch...
- 0 kudos
- 2256 Views
- 3 replies
- 0 kudos
Optimizing a complex pyspark join
I have a complex join that I'm trying to optimize df1 has cols id,main_key,col1,col1_isnull,col2,col2_isnull...col30 df2 has cols id,main_key,col1,col2..col_30I'm trying to run this sql query on Pysparkselect df1.id, df2.id from df1 join df2 on df1.m...
- 2256 Views
- 3 replies
- 0 kudos
- 0 kudos
@Omri thanks for your question! To help optimize your complex join further, we need clarification on a few details: Data Characteristics: Approximate size of df1 and df2 (in rows and/or size).Distribution of main_key in both dataframes—are the top...
- 0 kudos
- 3324 Views
- 5 replies
- 1 kudos
Dashboard Usage
Hi there,My team is developing some SQL Dashboards. I would like to know how many people view that dashboard/or at least click to it and then queries triggered.I found out that there is one endpoint provided by Databricks: List Queries | Query Histor...
- 3324 Views
- 5 replies
- 1 kudos
- 1 kudos
When I click to the dashboard, there are 6 statements in my dashboard -> I receive 6 records in `system.access.audit`.But the event_time is different, I expect event_time should be the same across records. So with the differences in event time, how c...
- 1 kudos
- 1067 Views
- 7 replies
- 0 kudos
Shouldn't the global statement_timeout parameter prevail over the session parameter?
How can I block the use of statement_timeout at the session level in Databricks? I want the global parameter to be enforced even if a SET statement_timeout has been executed in Databricks notebooks."
- 1067 Views
- 7 replies
- 0 kudos
- 0 kudos
Yes, this global config only applies to Serverless SQL Warehouses, but wont apply if you use serverless compute for notebooks
- 0 kudos
- 2788 Views
- 3 replies
- 5 kudos
Resolved! Help with Integration Testing for SQL Notebooks in Databricks
Hi everyone,I’m looking for the best way to implement integration tests for SQL notebooks in an environment that uses Unity Catalog and workflows to execute these notebooks.For unit tests on SQL functions, I’ve reviewed the https://docs.databricks.co...
- 2788 Views
- 3 replies
- 5 kudos
- 5 kudos
Hi @jorperort ,I see the question is already answered, but your question motivated me to create an article in medium and also to create a sample repo with the integration test written for SQL notebook.I hope it will be of useful for you:https://filip...
- 5 kudos
- 1044 Views
- 2 replies
- 3 kudos
Lakehouse Fundamentals Course
I have an issue I encountered while attempting to access the Lakehouse Fundamentals badge quiz. Every time I try to access the quiz, I receive the following error message: "Access denied. You do not have permission to access this page. Please contact...
- 1044 Views
- 2 replies
- 3 kudos
- 3 kudos
If this does not help you can reach out to training-ops@databricks.com
- 3 kudos
- 1143 Views
- 2 replies
- 2 kudos
Resolved! Wap pattern unity catalog
Good afternoon,I am looking for documentation to implement the WAP pattern using Unity Catalog, workflows, SQL notebooks, and any other services necessary to use this pattern. Could you share information on how to approach the problem with documentat...
- 1143 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @jorperort ,Apart from nice step by step instruction that @VZLA has provided, you can also take a look at short presentation of WAP pattern at the official databricks YT channel:https://youtu.be/4K3zAmUgViE?t=492
- 2 kudos
- 1481 Views
- 8 replies
- 1 kudos
Is it possible to load data only using Databricks SDK?
Is it possible to load data only using Databricks SDK?I have custom library that has to load data to a table, and I know about other features like autoloader, COPY INTO, notebook with spark dataframe... but I wonder if it is possible to load data dir...
- 1481 Views
- 8 replies
- 1 kudos
- 1 kudos
Got it, the reason of the cluster is because with shared access cluster the access to local system is more restricted that with single user cluster due to security constraints. As you are using serverless it acts as a shared cluster, on this case you...
- 1 kudos
- 610 Views
- 2 replies
- 0 kudos
Submit new records from gold layer to postgres db
Hi community,I want to ask you which is the best practice in your opinion to fill the data from gold layer to postgres db that is used to provide "real-time" data to an UI interface?Thanks for any help!
- 610 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @hari-prasad, Thanks you for your answer, considers that we are using DLT Pipelines, is it a good choice in this case? Because actually I don't see with DLT Pipelines the metadata for these materialized tables in this case these CDF statements
- 0 kudos
- 3320 Views
- 1 replies
- 0 kudos
Refresh options on PBI from Databricks workflow using Azure Databricks
Hi!I have a workflow that includes my medallion architecture and DLT. Currently, I have a separate notebook for refreshing my Power BI semantic model, which works based on the method described in Refresh a PowerBI dataset from Azure Databricks. Howe...
- 3320 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @mkEngineer, Have you reviewed this documentation: https://learn.microsoft.com/en-us/azure/databricks/partners/bi/power-bi Also I don't think Serverless compute for Notebook will work for your connection with Power BI. You might need to setup a Se...
- 0 kudos
- 55250 Views
- 7 replies
- 13 kudos
How to connect and extract data from sharepoint using Databricks (AWS) ?
We are using Databricks (on AWS). We need to connect to SharePoint and extract & load data to Databricks Delta table. Any possible solution on this ?
- 55250 Views
- 7 replies
- 13 kudos
- 13 kudos
Wondering the same.. Can we use Sharepoint REST API to download the file and save to dbfs/external location and read it?
- 13 kudos
| User | Count |
|---|---|
| 1615 | |
| 779 | |
| 349 | |
| 287 | |
| 266 |
