Does anyone have any tips for using PowerBI on top of databricks? Any best practices you know of or roadblocks you have run into that should be avoided?Thanks.
Hey,
Use Partner Connect to establish a connection to PBI
Consider to use Databricks SQL Serverless warehouses for the best user experience and performance (see Intelligent Workload Management aka auto-scaling and query queuing, remote result cache, ...
Team,I get a ConcurrentAppendException: Files were added to the root of the table by a concurrent update when trying to update a table which executes via jobs with for each activity in ADF,I tried with Databricks run time 14.x and set the delete vect...
Hey,
This issue happens whenever two or more jobs try to write to the same partition for a table.
This exception is often thrown during concurrent DELETE, UPDATE, or MERGE operations. While the concurrent operations may be physically updating differe...
I am looking for a possible way to get the autoscaling history data for SQL Serverless Warehouses using API or logs.I want something like what we see in monitoring UI.
Hi Rahul, you need to perform two actions :
Enable system tables schema named "compute" (how-to, take a look on the page, it's highly possible that you'll find other schemas useful too)Explore system.compute.warehouse_events table
Hope this helps.
B...
I'm executing a notebook and failed with this error: Sometime, when i execute some function in spark and also failed with the error 'this class is not whitelist'. Could everyone help me check on this?Thanks for your help!
We know that Databricks with VNET injection (our own VNET) allows is to connect to blob storage/ ADLS Gen2 over private endpoints and peering. This is what we typically do.We have a client who created Databricks with EnableNoPublicIP=No (secure clust...
Hi,We are trying to ingest zip files into Azure Databricks delta lake using COPY INTO command. There are 100+ zip files with average size of ~300MB each.Cluster configuration:1 driver: 56GB, 16 cores2-8 workers: 32GB, 8 cores (each). Autoscaling enab...
Although we were able to copy the zip files onto the DB volume, we were not able to share them with any system outside of the Databricks environment. Guess delta sharing does not support sharing files that are on UC volumes.
I setup a model serving endpoint and created a monitoring dashboard to monitor its performance. The problem is my inference table doesn't get updated by model serving endpoints. To test the endpoint I use the following codeimport random
import time
...
Hi @MohsenJ,
The log shows several reconfiguration errors related to the logger configuration. These errors are likely due to missing or incorrect configuration settings. Here are some steps to troubleshoot:
Check Log Configuration: Verify that the...
I have a very strange thing happening. I'm importing a csv file and nulls and blanks are being interpreted correctly. What is strange is that a column that regularly has a single space character value is having the single space converted to null.I'...
Hi @Chrispy,
Handling Empty Cells as NULLs: When importing data from a CSV file, you want to treat empty cells as NULL values. This is a common requirement, especially when dealing with databases. Let’s explore a couple of approaches to achieve ...
For my AWS databricks cluster, i configured shared computer with 1min worker node and 3 max worker node, initailly only one worker node and driver node instance is created in the AWS console. Is there any rule set by databricks for scale up the next ...
Hi @Nandhini_Kumar,
Cluster Configuration:
When you create a Databricks cluster, you have several options for compute configuration. These choices impact performance, cost, and scalability.Two primary types of computing are available:
All-purp...
Hi,In a particular Workflows Job, I am trying to add some data checks in between each task by using If else statement. I used following statement in a notebook to call parameter in if else condition to check logic.{"job_id": XXXXX,"notebook_params": ...
Hi @Azsdc, In Databricks Jobs, you can use conditional logic to control task execution.
Let’s break down how you can achieve this:
Using Parameters in If/Else Conditions:
To define a parameter for use in an If/Else condition within a job, follow...
After downloading a file using `wget`, I'm attempting to read it by spark.read.json.I am getting error: PATH_NOT_FOUND - Path does not exist: dbfs:/tmp/data.json. SQLSTATE: 42K03File <command-3327713714752929>, line 2 I have checked the file do exist...
Hi @sharma_kamal , Good Day!
Could you please try the below code suggested by @ThomazRossito , it will help you.
Also please refer to the below document to work with the files on Databricks:
https://docs.databricks.com/en/files/index.html
Please l...
Hello,Previously I was able to run the folowing command in databricks to see a list of the mount points but it seems the system does not accept this anymore as I get the following error.Any thoughts on how to get a list of the mount points?Thank youd...
Hi @arkiboys, To retrieve a list of mount points in Azure Databricks, you can use the following methods:
Using Databricks Utilities (dbutils):
In a Python Notebook, execute the command dbutils.fs.mounts(). This will display all the mount points w...
Currently in unity catalog enabled workspace users with "Workspace access" can create workflows/jobs, there is no access control available to restrict users from creating jobs/workflows.Use case: In production there is no need for users, data enginee...
We have the "allow unrestricted cluster creation" box deselected for all groups and have users creating jobs in production so we are looking for a way to disable this. I cannot believe this isn't an option. Did anyone find a solution for this?
