- 9509 Views
- 17 replies
- 7 kudos
Resolved! unzip twice the same file not executing
Hi, I need to unzip some files that are ingested but when I unzip twice the same zipped file, the unzip command does not execute :As suggesgted in the documentation I did :import urllib urllib.request.urlretrieve("https://resources.lendingclub.com/L...
- 9509 Views
- 17 replies
- 7 kudos
- 7 kudos
Another problem is that dbfs storage doesn't support random writes (used by zip):Does not support random writes. For workloads that require random writes, perform the operations on local disk first and then copy the result to/dbfssource: https://docs...
- 7 kudos
- 9985 Views
- 8 replies
- 7 kudos
The log in function and password reset function in the community edition do not work
I want to register a databrick account. I already set up my account. I also receive the email to set my password. However, I cannot use my password to log in the community account. I can use it to log in my standard account. I also click the reset th...
- 9985 Views
- 8 replies
- 7 kudos
- 7 kudos
Hello, @lawrance Zhang​ - I wanted you to know that this isn't the first time we've heard of this recently. Thank you for opening a ticket. We've also escalated this to the team. We'll get there.
- 7 kudos
- 2164 Views
- 0 replies
- 19 kudos
www.lunasec.io
Log4J: part 2Apparently log4j 2.15 is still vulnerable:https://www.lunasec.io/docs/blog/log4j-zero-day-update-on-cve-2021-45046/So beter use version 2.16.But as mentioned in several topics: Databricks does not use an impacted version.
- 2164 Views
- 0 replies
- 19 kudos
- 6685 Views
- 3 replies
- 4 kudos
Resolved! ADLSGen2: Cannot display Delta table summary from Data>Table tab on Databricks
Hello, I have set up my account storage on Azure with an ADLSGen2 and I have succeeded to save the delta table on my ADLSGen2, from there I have created my delta table on Databricks.From there I am unable to display the summary of my delta table unde...
- 6685 Views
- 3 replies
- 4 kudos
- 4 kudos
Hello,Following Hubert comment, in order to create a delta table on Databricks from Azure, I had to use CLONE argument in order to copy the data plus the metadata of my delta table on Azure. In order to set up the connection between Databricks and A...
- 4 kudos
- 3376 Views
- 7 replies
- 2 kudos
Create table for non-admins (Table Access Control cluster/workspace)
Is there a way for non admin (at workspace level) or users without having (SELECT, MODIFY on ANY File) to create tables (unmanaged/external) even though they are owner of the database in which they want to create tables in a Table Access Controlled c...
- 3376 Views
- 7 replies
- 2 kudos
- 2 kudos
Grant privileges on all the explain tables to non admin user as ... where BIADMIN is the non admin user who wants to generate explain plans. AdvancedMD Login
- 2 kudos
- 5946 Views
- 3 replies
- 4 kudos
Resolved! Gstreamer installation in databricks
Hi All,i am trying to install the Gstreamer using the instructions in the below link.https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/how-to-use-codec-compressed-audio-input-streams?tabs=debian&pivots=programming-language-pyth...
- 5946 Views
- 3 replies
- 4 kudos
- 4 kudos
Thanks you all.i have resolved this issue by running the below command.Update Packagessudo apt-get update sudo apt-get upgradesudo apt-get build-essentialsudo apt-get pkg-configsudo apt install libgstreamer1.0-0 \gstreamer1.0-plugins-base \gstreamer1...
- 4 kudos
- 5163 Views
- 2 replies
- 0 kudos
How to process images and video through structured streaming using Delta Lake?
Can we scan though videos and identify and alert in real time if something goes wrong? what are best practices for this kind of use case?
- 5163 Views
- 2 replies
- 0 kudos
- 0 kudos
Maybe I'm a little off topic, but can you recommend companies that are engaged in video production? I want to make an explanatory video for my site.
- 0 kudos
- 8767 Views
- 10 replies
- 3 kudos
Apache Log4J Vulnerability
Hi Community, We got an email from our IT Team regarding Apache Log4J Vulnerability. Just wanted to understand if our implementation will be affected by this or not. We are using the following library or package in our notebooksimport org.apache.log4...
- 8767 Views
- 10 replies
- 3 kudos
- 7883 Views
- 3 replies
- 11 kudos
When the billing time starts for the cluster?
Hi All, I'm just wondering when exactly the billing time starts for the DataBricks cluster? Is starting time included? If cluster creation time takes 3 minutes and query execution only 2, will I pay for 2 or 5?​Thanks in advance! MC
- 7883 Views
- 3 replies
- 11 kudos
- 11 kudos
Billing for databricks DBUs starts when Spark Context becomes available. Billing for the cloud provider starts when the request for compute is received and the VMs are starting up.
- 11 kudos
- 3653 Views
- 2 replies
- 1 kudos
Resolved! AutoLoader with Custom Queue
Hi Everyone can someone help with creating custom queue for auto loader as given here as default FlushwithClose event is not getting created when my data is uploaded to blob as given in azure DB docscloudFiles.queueNameThe name of the Azure queue. If...
- 3653 Views
- 2 replies
- 1 kudos
- 1 kudos
you need to setup notification service for blob/adls like here https://docs.databricks.com/spark/latest/structured-streaming/auto-loader-gen2.html#cloud-resource-managementsetUpNotificationServices will return queue name which later can be used in au...
- 1 kudos
- 6776 Views
- 4 replies
- 3 kudos
what is the difference between weekday and dayofweek function in spark SQL?
dayofweek: https://docs.databricks.com/sql/language-manual/functions/dayofweek.htmlweekday : https://docs.databricks.com/sql/language-manual/functions/weekday.htmlAccording to the documentation , they both are synonym functions. But when I use it I n...
- 6776 Views
- 4 replies
- 3 kudos
- 3 kudos
That's correct for weekday moday=0 for dayofweek Sunday=1.​​You can also look for documentation here https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.functions.dayofweek.html​https://spark.apache.org/docs/latest/api/sql/index...
- 3 kudos
- 4712 Views
- 6 replies
- 4 kudos
CVE-2021-44228
Hi,Any affect of CVE-2021-44228 problem on Databricks platform?Is there any action that needs to be done by Databricks customer related to CVE-2021-44228?
- 4712 Views
- 6 replies
- 4 kudos
- 2948 Views
- 1 replies
- 5 kudos
How to find out the users who accessed Databricks and from which location
How to find out the users who accessed Databricks and from which location
- 2948 Views
- 1 replies
- 5 kudos
- 5 kudos
You can use Audit logs to fetch this dataQuery:%sqlSELECT DISTINCT userIdentity.email, sourceIPAddressFROM audit_logsWHERE serviceName = "accounts" AND actionName LIKE "%login%"Please find below the docs to analyse the Audit logshttps://docs.databric...
- 5 kudos
- 2939 Views
- 3 replies
- 14 kudos
Notebook fails in job but not in interactive mode
I have this notebook which is scheduled by Data Factory on a daily basis.It works fine, up to today. All of a sudden I keep on getting NullpointerException when writing the data.After some searching online, I disabled AQE. But this does not help.Th...
- 2939 Views
- 3 replies
- 14 kudos
- 14 kudos
After some tests it seems that if I run the notebook on an interactive cluster, I only get 80% of load (Ganglia metrics).If I run the same notebook on a job cluster with the same VM types etc (so the only difference is interactive vs job), I get over...
- 14 kudos
- 2166 Views
- 4 replies
- 9 kudos
Databrick Job - Notebook Execution
Question - When you set a reoccuring job to simply update a notebook, does databricks clear the state of the notebook prior to executing the notebook? If not, can I configure it to make sure it clears the state before running?
- 2166 Views
- 4 replies
- 9 kudos
- 9 kudos
@Paras Patel​ - Would you be happy to mark Hubert's answer as best so that other members can find the solution more easily?Thanks!
- 9 kudos