cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
Prabhakar1
by • New Contributor III
  • 13981 Views
  • 5 replies
  • 8 kudos

How Selenium Webdriver works on Azure Databricks? I am unable to run a simple code.

from selenium import webdriverfrom webdriver_manager.chrome import ChromeDriverManagerfrom selenium.webdriver.common.by import Byfrom selenium.webdriver.chrome.options import Optionsdrivers = webdriver.Chrome(ChromeDriverManager().install())drivers.g...

Data Engineering
Reply
  • 13981 Views
  • 5 replies
  • 8 kudos
View Replies
Latest Reply
Evan_MCK
Contributor
  • 8 kudos

I also got that error. What worked for me was downloading the chrome driver and ensuring its the latest version with shell scripts in the same notebook I used for web scraping. I could not use the web driver manager. You can see all the details here...

  • 8 kudos
4 More Replies
AJ270990
by • Contributor II
  • 7835 Views
  • 5 replies
  • 6 kudos

Resolved! ChromeDriver installation in Databricks

I am working on a Webscraping logic and need to install Chrome driver. How can I install it in the Databricks workbook ?

Data Engineering
Reply
  • 7835 Views
  • 5 replies
  • 6 kudos
View Replies
Latest Reply
Evan_MCK
Contributor
  • 6 kudos

What worked for me was downloading the chrome driver and ensuring its the latest version with shell scripts in the same notebook I used for web scraping. You can see all the details here: https://stackoverflow.com/questions/69192050/using-selenium-...

  • 6 kudos
4 More Replies
Suman
by • New Contributor III
  • 959 Views
  • 2 replies
  • 2 kudos

Can Lint be implemented in multilingual databricks notebook. ?

I want to implement lint in my project as part of CICD. But as my notebooks has a mix of python and scala . Can Lint be implemented there ?

Data Engineering
Reply
  • 959 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Suman
New Contributor III
  • 2 kudos

@Vidula Khanna​ I didn't hear from anyone . I am looking for this answer to implement in my databricks code of my project.Can you please assist.

  • 2 kudos
1 More Replies
jch
by • New Contributor III
  • 6789 Views
  • 4 replies
  • 5 kudos

Resolved! Why does spark.read.csv come back with an error: com.databricks.sql.io.FileReadException: Error while reading file dbfs:/mnt/cntnr/demo/circuits.csv ?

I need help understanding why I can't open a file.In a databricks notebook, I use this code:%fs   ls /mnt/cntnr/demoI get back dbfs:/mnt/cntnr/demo/circuits.csv as one of the path values.When I use this code, I get an error:circuits_df = spark.read....

Data Engineering
Reply
  • 6789 Views
  • 4 replies
  • 5 kudos
View Replies
Latest Reply
jch
New Contributor III
  • 5 kudos

It turns out my spark config was wrong    #Set Spark configuration    configs = {"fs.azure.account.auth.type": "OAuth",          "fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",          "fs.azu...

  • 5 kudos
3 More Replies
PrawnballNightm
by • New Contributor III
  • 3903 Views
  • 4 replies
  • 0 kudos

Resolved! Cannot configure VS code databricks extension with a non-standard databricks URL: not a databricks host.

Hello,I'm trying to connect to our databricks instance using the vscode extension. However, when following this guide we cannot get the configuration to proceed past the point that it asks for our instance URL. The prompt appears to expect a URL of t...

databricks_error
Data Engineering
Reply
  • 3903 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
PrawnballNightm
New Contributor III
  • 0 kudos

Hello,Yes, the databricks team shared a modified version of the vs code plugin which did not include the URL matching logic. It connects successfully. However, our custom URL is as it is because our organisation is hosting its own instance of Databri...

  • 0 kudos
3 More Replies
bkbarunkalita64
by • New Contributor II
  • 1143 Views
  • 2 replies
  • 2 kudos

Update query from Azure databricks to Oracle table

I need to update a single row on a on-prem Oracle table via jdbc connection.Please note, I don't want to append, just have to update a row, is it possible ?

Data Engineering
Reply
  • 1143 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 2 kudos

Hi @Tony S​ , This article mentioned all the steps required to establish a successful connection with the database using Python.However, you can try this code :- df.write.format("jdbc") \ .option("url", dbServerJdbcUrl) \ .option("user", username...

  • 2 kudos
1 More Replies
Jain
by • New Contributor III
  • 3548 Views
  • 4 replies
  • 4 kudos

Unable to use dbutils in Premium

I am unable to use dbutils commands and mkdir, etc also does not work after upgrading my Databricks Workspace from Standard tier to Premium tier.It throws the following error:py4j.security.Py4JSecurityException: Constructor public com.databricks.back...

Data Engineering
Reply
  • 3548 Views
  • 4 replies
  • 4 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Abhishek Jain​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...

  • 4 kudos
3 More Replies
gilo12
by • New Contributor III
  • 1216 Views
  • 2 replies
  • 0 kudos

Running multiple Databricks queries with golang client

I am using github.com/databricks/databricks-sql-go to connect and run queries on my Databricks warehouse. However when I try to run multiple queries:insert into....; insert into....;It fails and complains on syntax errors.[PARSE_SYNTAX_ERROR] Syntax ...

Data Engineering
Reply
  • 1216 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Gil Omer​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your...

  • 0 kudos
1 More Replies
gtyhchang
by • New Contributor II
  • 892 Views
  • 2 replies
  • 1 kudos

pandas issue

We identify a potential bug in either DBFS or Pandas that when writting a dataframe using Pandas `to_csv`, `to_parquet`, `to_pickle` etc to a mounted ADLS location with read-only service principle didn't throw permission deny exceptions. However, met...

Data Engineering
Reply
  • 892 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Yung-Hang Chang​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...

  • 1 kudos
1 More Replies
Databricks3
by • Contributor
  • 1139 Views
  • 1 replies
  • 1 kudos

Concurrent Insert on a delta table fails if the table contains Identity Columns. Error message are added below.MetadataChangedException: The metadata ...

Concurrent Insert on a delta table fails if the table contains Identity Columns. Error message are added below.MetadataChangedException: The metadata of the Delta table has been changed by a concurrent update. Please try the operation again.

Data Engineering
Reply
  • 1139 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @SK ASIF ALI​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 1 kudos
Susheel
by • New Contributor II
  • 734 Views
  • 1 replies
  • 0 kudos

We have concern with respect to the feasibility of installing the Databricks runtime supported version on-premise. This is because we are planning to...

We have concern with respect to the feasibility of installing the Databricks runtime supported version on-premise. This is because we are planning to migrate the existing Azure Databricks application code to an on-premise environment based on the cl...

Data Engineering
Reply
  • 734 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Susheel
New Contributor II
  • 0 kudos

Any update?

  • 0 kudos
Chhaya
by • New Contributor III
  • 1383 Views
  • 2 replies
  • 2 kudos

DLT config/setting json support

hi team,There used to be option to provide DLT pipeline settings either via UI or JSON, but I do not see it anymore after switching to new UI. Is this something expected ? am I missing something ? here is screenshot for reference.

image
Data Engineering
Reply
  • 1383 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
User16752245772
Contributor
  • 2 kudos

Hi @Chhaya Vishwakarma​ This option is available, could you please clear the browser cache and try ? or can you try in an incognito window?

  • 2 kudos
1 More Replies
ChrisS
by • New Contributor III
  • 4457 Views
  • 2 replies
  • 1 kudos

Trying to mount Azure Data Lake Storage Gen 2 to Azure Databricks

I have validated many many many times all my credentials and I am still getting the following error (at the very end). ChatGPT said to basically recheck everything and I did. The one thing I didn't do was grant the permissions which I have since done...

Data Engineering
Reply
  • 4457 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
User16752245772
Contributor
  • 1 kudos

Hi @Chris Sarrico​ Could you please specify the container name before the storage account name like this :so the source looks like this. :source = "abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/"https://learn.microsoft.com/en-u...

  • 1 kudos
1 More Replies
kyzers0ze
by • New Contributor
  • 916 Views
  • 1 replies
  • 0 kudos

0 byte files appearing in blob storage

When writing out files to azure blob storage using spark there are some 0 byte files being created on the web portal but not when I view it using storage explorer.I was wondering if there is there any work around for this?

Data Engineering
Reply
  • 916 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Ryan Low​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 0 kudos
ssaloni_bhatia
by • New Contributor
  • 943 Views
  • 1 replies
  • 0 kudos

Spark read not working in parallel

Hi All,Am trying to run a spark jdbc query and the read is running on only 1 worker and is taking a lot of time. Usually, it takes 4 minutes in aws lambda but now it is taking approx 10 minutes while using databricks. There are only 4m records (write...

Data Engineering
Reply
  • 943 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Saloni Bhatia​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 0 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels
Top Kudoed Authors
View all