Hello Everyone,I am thrilled to announce that we have our 6th winner for the raffle contest - @Yogita Chavan . Please join me in congratulating her on this remarkable achievement!Your dedication and hard work have paid off, and we are delighted to ...
Hello all,I am suddenly getting an HTTP 502 and DRIVER_LIBRARY_INSTALLATION_FAILURE error during the Python library installation when the cluster gets initialized. I have around 10 Python packages out of which 2-3, packages always failed to install a...
Hi @Priyadarshini G Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
from the docs I read : Create a storage container where the metastore’s managed table data will be stored.This storage container must be in a Premium performance Azure Data Lake Storage Gen2 account in the same region as the workspaces you want to us...
Hi @Carlos Restrepo We haven't heard from you since the last response from @Kaniz Fatma , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to o...
We have a Databricks account on AWS and storage is on Azure Cloud (ADLS), because of cross-platform do we face any security and performance-related issues also is this feasible and supported? could you please provide your suggestions on this?
Hi! When I use `databricks jobs list --version=2.0` I get all jobs deployed using 2.0 and 2.1 API, however, when I use `databricks jobs list --version=2.1` I only get jobs deployed using 2.1 API. This is a behaviour that we've only experienced recent...
Hi @Guillermo Sanchez Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
I am planning to introduce a custom logging to the databricks workload. To achieve this I am using a python logging module. I am storing logs in driver memory "file:/tmp/" directory before I move those logs to blob storage. In my personal databricks ...
Cloudformation runs into error during 'assign Metastore' step.Following is the error from Cloudwatch:HTTP content: b'{ "error_code": "PERMISSION_DENIED", "message": "Cannot assign metastore to STANDARD tier workspace xyz" }What is the solution for th...
Hi @Abhilash Mhaisne Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
I need to process some transformation on incoming data as a batch and want to know if there is way to use foreachbatch option in deltalivetable. I am using autoloader to load json files and then I need to apply foreachbatch and store results into ano...
Not sure if this will apply to you or not...I was looking at the foreachbatch tool to reduce the workload of getting distinct data from a history table of 20million + records because the df.dropDuplicates() function was intermittently running out of ...
Is there a way to get the directory size in ADLS(gen2) using dbutils in databricks?
If I run this
dbutils.fs.ls("/mnt/abc/xyz")
I get the file sizes inside the xyz folder( there are about 5000 files), I want to get the size of the XYZ folder
how ca...
File size is only specified for files. So, if you specify a directory as your source, you have to iterate through the directory. The below snippet should work (and should be faster than the other solutions).import glob
def get_directory_size_in_byt...
I also got that error. What worked for me was downloading the chrome driver and ensuring its the latest version with shell scripts in the same notebook I used for web scraping. I could not use the web driver manager. You can see all the details here...
What worked for me was downloading the chrome driver and ensuring its the latest version with shell scripts in the same notebook I used for web scraping. You can see all the details here: https://stackoverflow.com/questions/69192050/using-selenium-...
