- 3534 Views
- 1 replies
- 0 kudos
Python custom Logging on Databricks
I am planning to introduce a custom logging to the databricks workload. To achieve this I am using a python logging module. I am storing logs in driver memory "file:/tmp/" directory before I move those logs to blob storage. In my personal databricks ...
- 3534 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @MARKSACHIN K​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 0 kudos
- 2343 Views
- 1 replies
- 1 kudos
Error while creating workspace with 'Quickstart' on AWS
Cloudformation runs into error during 'assign Metastore' step.Following is the error from Cloudwatch:HTTP content: b'{ "error_code": "PERMISSION_DENIED", "message": "Cannot assign metastore to STANDARD tier workspace xyz" }What is the solution for th...
- 2343 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Abhilash Mhaisne​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 1 kudos
- 6397 Views
- 3 replies
- 3 kudos
How to use foreachbatch in deltalivetable or DLT?
I need to process some transformation on incoming data as a batch and want to know if there is way to use foreachbatch option in deltalivetable. I am using autoloader to load json files and then I need to apply foreachbatch and store results into ano...
- 6397 Views
- 3 replies
- 3 kudos
- 3 kudos
Not sure if this will apply to you or not...I was looking at the foreachbatch tool to reduce the workload of getting distinct data from a history table of 20million + records because the df.dropDuplicates() function was intermittently running out of ...
- 3 kudos
- 11326 Views
- 3 replies
- 3 kudos
New SQL Dashboard feature never loads
I've created the sample Taxi dashboard, but the charts never load?
- 11326 Views
- 3 replies
- 3 kudos
- 3 kudos
Do you have the full error stack trace? does this happens frequently? whats the size of your cluster?
- 3 kudos
- 19367 Views
- 5 replies
- 8 kudos
How Selenium Webdriver works on Azure Databricks? I am unable to run a simple code.
from selenium import webdriverfrom webdriver_manager.chrome import ChromeDriverManagerfrom selenium.webdriver.common.by import Byfrom selenium.webdriver.chrome.options import Optionsdrivers = webdriver.Chrome(ChromeDriverManager().install())drivers.g...
- 19367 Views
- 5 replies
- 8 kudos
- 8 kudos
I also got that error. What worked for me was downloading the chrome driver and ensuring its the latest version with shell scripts in the same notebook I used for web scraping. I could not use the web driver manager. You can see all the details here...
- 8 kudos
- 16356 Views
- 4 replies
- 6 kudos
Resolved! ChromeDriver installation in Databricks
I am working on a Webscraping logic and need to install Chrome driver. How can I install it in the Databricks workbook ?
- 16356 Views
- 4 replies
- 6 kudos
- 6 kudos
What worked for me was downloading the chrome driver and ensuring its the latest version with shell scripts in the same notebook I used for web scraping. You can see all the details here: https://stackoverflow.com/questions/69192050/using-selenium-...
- 6 kudos
- 2414 Views
- 2 replies
- 2 kudos
Can Lint be implemented in multilingual databricks notebook. ?
I want to implement lint in my project as part of CICD. But as my notebooks has a mix of python and scala . Can Lint be implemented there ?
- 2414 Views
- 2 replies
- 2 kudos
- 2 kudos
@Vidula Khanna​ I didn't hear from anyone . I am looking for this answer to implement in my databricks code of my project.Can you please assist.
- 2 kudos
- 9285 Views
- 4 replies
- 5 kudos
Resolved! Why does spark.read.csv come back with an error: com.databricks.sql.io.FileReadException: Error while reading file dbfs:/mnt/cntnr/demo/circuits.csv ?
I need help understanding why I can't open a file.In a databricks notebook, I use this code:%fs ls /mnt/cntnr/demoI get back dbfs:/mnt/cntnr/demo/circuits.csv as one of the path values.When I use this code, I get an error:circuits_df = spark.read....
- 9285 Views
- 4 replies
- 5 kudos
- 5 kudos
It turns out my spark config was wrong #Set Spark configuration configs = {"fs.azure.account.auth.type": "OAuth", "fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider", "fs.azu...
- 5 kudos
- 12452 Views
- 4 replies
- 0 kudos
Resolved! Cannot configure VS code databricks extension with a non-standard databricks URL: not a databricks host.
Hello,I'm trying to connect to our databricks instance using the vscode extension. However, when following this guide we cannot get the configuration to proceed past the point that it asks for our instance URL. The prompt appears to expect a URL of t...
- 12452 Views
- 4 replies
- 0 kudos
- 0 kudos
Hello,Yes, the databricks team shared a modified version of the vs code plugin which did not include the URL matching logic. It connects successfully. However, our custom URL is as it is because our organisation is hosting its own instance of Databri...
- 0 kudos
- 14542 Views
- 4 replies
- 4 kudos
Unable to use dbutils in Premium
I am unable to use dbutils commands and mkdir, etc also does not work after upgrading my Databricks Workspace from Standard tier to Premium tier.It throws the following error:py4j.security.Py4JSecurityException: Constructor public com.databricks.back...
- 14542 Views
- 4 replies
- 4 kudos
- 4 kudos
Hi @Abhishek Jain​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...
- 4 kudos
- 3204 Views
- 2 replies
- 0 kudos
Running multiple Databricks queries with golang client
I am using github.com/databricks/databricks-sql-go to connect and run queries on my Databricks warehouse. However when I try to run multiple queries:insert into....; insert into....;It fails and complains on syntax errors.[PARSE_SYNTAX_ERROR] Syntax ...
- 3204 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Gil Omer​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your...
- 0 kudos
- 2303 Views
- 2 replies
- 1 kudos
pandas issue
We identify a potential bug in either DBFS or Pandas that when writting a dataframe using Pandas `to_csv`, `to_parquet`, `to_pickle` etc to a mounted ADLS location with read-only service principle didn't throw permission deny exceptions. However, met...
- 2303 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Yung-Hang Chang​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...
- 1 kudos
- 2453 Views
- 1 replies
- 1 kudos
Concurrent Insert on a delta table fails if the table contains Identity Columns. Error message are added below.MetadataChangedException: The metadata ...
Concurrent Insert on a delta table fails if the table contains Identity Columns. Error message are added below.MetadataChangedException: The metadata of the Delta table has been changed by a concurrent update. Please try the operation again.
- 2453 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @SK ASIF ALI​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 1 kudos
- 1793 Views
- 1 replies
- 0 kudos
We have concern with respect to the feasibility of installing the Databricks runtime supported version on-premise. This is because we are planning to...
We have concern with respect to the feasibility of installing the Databricks runtime supported version on-premise. This is because we are planning to migrate the existing Azure Databricks application code to an on-premise environment based on the cl...
- 1793 Views
- 1 replies
- 0 kudos
- 0 kudos
Any update?
- 0 kudos
- 3926 Views
- 2 replies
- 2 kudos
DLT config/setting json support
hi team,There used to be option to provide DLT pipeline settings either via UI or JSON, but I do not see it anymore after switching to new UI. Is this something expected ? am I missing something ? here is screenshot for reference.
- 3926 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Chhaya Vishwakarma​ This option is available, could you please clear the browser cache and try ? or can you try in an incognito window?
- 2 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 567 | |
| 349 | |
| 287 |
