Databricks Community

Hi @jeremy98, To read the Change Data Feed (CDF) logs in materialized views created by a Delta Live Tables (DLT) pipeline, you can follow these steps:   Enable Change Data Feed: Ensure that the change data feed is enabled on the base tables of the ma...

Hi @mkEngineer, Have you reviewed this documentation: https://learn.microsoft.com/en-us/azure/databricks/partners/bi/power-bi Also I don't think Serverless compute for Notebook will work for your connection with Power BI. You might need to setup a Se...

Checked the config.yml file and it looks like this: agent_prompt: "Use functions to interact with questions about eggs / henhouse."llm_endpoint: "databricks-meta-llama-3-3-70b-instruct"warehouse_id: "1c7bf12e78b673de"uc_functions: "main.egg_shop.*"St...

Hi @kasiviss42, Are you using any Scala code in your notebook? The error [UNITY_CREDENTIAL_SCOPE_MISSING_SCOPE] Missing Credential Scope. Unity Credential Scope id not found in thread locals that you are encountering when using dbutils.fs.ls and whil...

Hi @soumiknow, To resolve the 'connection refused' error when using the google-cloud-bigquery library in your Databricks notebook, you need to ensure that your Databricks cluster is properly configured to authenticate with Google Cloud Platform (GCP)...

Hi @Phani1 ,You can use delta sharing. In that way you can create share that will allow you to access data stored in GCS and it's govern by UC permissions model.What is Delta Sharing? | Databricks on AWSYou can also use legacy approach, but it doesn'...

It's not feasible, in my opinion. It is feasible to load a table from DataBricks into a PowerBI dataset, but not the other way around. 

Hi @svm_varma ,You can try to create Standard_DS3_v2 cluster. It has 4 cores and your current subscription limit for given region is 6 cores. The one you're trying to create needs 8 cores and hence you're getting quota exceeded exception.You can also...

Hi @guiferviz, To determine the type of refresh used, you can query the Delta Live Tables event log. Look for the event_type called planning_information to see the technique used for the refresh. The techniques include:   FULL_RECOMPUTE: Indicates a ...

For further details on managing init scripts and optimizing the bootstrap process, you can refer to the Databricks documentation on init scripts. This documentation provides recommendations for using built-in platform features instead of init scripts...

Continuation of my comments:  Shuffle Hash Join: Prefer shuffle hash join over sort-merge join if applicable. This can be more efficient in certain scenarios: spark.conf.set("spark.sql.join.preferSortMergeJoin", "false")   Data Skew Remediation: Iden...

Hi @singhanuj2803, It is correct that Spark SQL does not natively support recursive Common Table Expressions (CTEs). However, there are some workarounds and alternative methods you can use to achieve similar results.   Using DataFrame API with Loops:...

To execute a PostgreSQL stored procedure from an Azure Databricks notebook, you need to follow these steps: Required Libraries:You need to install the psycopg2 library, which is a PostgreSQL adapter for Python. This can be done using the %pip install...