cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

abhilashmhaisne
by New Contributor II
  • 702 Views
  • 1 replies
  • 1 kudos

Error while creating workspace with 'Quickstart' on AWS

Cloudformation runs into error during 'assign Metastore' step.Following is the error from Cloudwatch:HTTP content: b'{ "error_code": "PERMISSION_DENIED", "message": "Cannot assign metastore to STANDARD tier workspace xyz" }What is the solution for th...

  • 702 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Abhilash Mhaisne​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 1 kudos
rdobbss
by New Contributor II
  • 2892 Views
  • 4 replies
  • 3 kudos

How to use foreachbatch in deltalivetable or DLT?

I need to process some transformation on incoming data as a batch and want to know if there is way to use foreachbatch option in deltalivetable. I am using autoloader to load json files and then I need to apply foreachbatch and store results into ano...

  • 2892 Views
  • 4 replies
  • 3 kudos
Latest Reply
TomRenish
New Contributor III
  • 3 kudos

Not sure if this will apply to you or not...I was looking at the foreachbatch tool to reduce the workload of getting distinct data from a history table of 20million + records because the df.dropDuplicates() function was intermittently running out of ...

  • 3 kudos
3 More Replies
Oliver_Angelil
by Valued Contributor II
  • 2523 Views
  • 3 replies
  • 3 kudos

New SQL Dashboard feature never loads

I've created the sample Taxi dashboard, but the charts never load?

Screenshot 2023-05-11 at 13.39.59
  • 2523 Views
  • 3 replies
  • 3 kudos
Latest Reply
jose_gonzalez
Moderator
  • 3 kudos

Do you have the full error stack trace? does this happens frequently? whats the size of your cluster?

  • 3 kudos
2 More Replies
Prabhakar1
by New Contributor III
  • 13196 Views
  • 5 replies
  • 8 kudos

How Selenium Webdriver works on Azure Databricks? I am unable to run a simple code.

from selenium import webdriverfrom webdriver_manager.chrome import ChromeDriverManagerfrom selenium.webdriver.common.by import Byfrom selenium.webdriver.chrome.options import Optionsdrivers = webdriver.Chrome(ChromeDriverManager().install())drivers.g...

  • 13196 Views
  • 5 replies
  • 8 kudos
Latest Reply
Evan_MCK
Contributor
  • 8 kudos

I also got that error. What worked for me was downloading the chrome driver and ensuring its the latest version with shell scripts in the same notebook I used for web scraping. I could not use the web driver manager. You can see all the details here...

  • 8 kudos
4 More Replies
AJ270990
by Contributor II
  • 7072 Views
  • 5 replies
  • 6 kudos

Resolved! ChromeDriver installation in Databricks

I am working on a Webscraping logic and need to install Chrome driver. How can I install it in the Databricks workbook ?

  • 7072 Views
  • 5 replies
  • 6 kudos
Latest Reply
Evan_MCK
Contributor
  • 6 kudos

What worked for me was downloading the chrome driver and ensuring its the latest version with shell scripts in the same notebook I used for web scraping. You can see all the details here: https://stackoverflow.com/questions/69192050/using-selenium-...

  • 6 kudos
4 More Replies
Suman
by New Contributor III
  • 839 Views
  • 2 replies
  • 2 kudos

Can Lint be implemented in multilingual databricks notebook. ?

I want to implement lint in my project as part of CICD. But as my notebooks has a mix of python and scala . Can Lint be implemented there ?

  • 839 Views
  • 2 replies
  • 2 kudos
Latest Reply
Suman
New Contributor III
  • 2 kudos

@Vidula Khanna​ I didn't hear from anyone . I am looking for this answer to implement in my databricks code of my project.Can you please assist.

  • 2 kudos
1 More Replies
jch
by New Contributor III
  • 2409 Views
  • 4 replies
  • 5 kudos

Resolved! Why does spark.read.csv come back with an error: com.databricks.sql.io.FileReadException: Error while reading file dbfs:/mnt/cntnr/demo/circuits.csv ?

I need help understanding why I can't open a file.In a databricks notebook, I use this code:%fs   ls /mnt/cntnr/demoI get back dbfs:/mnt/cntnr/demo/circuits.csv as one of the path values.When I use this code, I get an error:circuits_df = spark.read....

  • 2409 Views
  • 4 replies
  • 5 kudos
Latest Reply
jch
New Contributor III
  • 5 kudos

It turns out my spark config was wrong    #Set Spark configuration    configs = {"fs.azure.account.auth.type": "OAuth",          "fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",          "fs.azu...

  • 5 kudos
3 More Replies
PrawnballNightm
by New Contributor III
  • 3447 Views
  • 4 replies
  • 0 kudos

Resolved! Cannot configure VS code databricks extension with a non-standard databricks URL: not a databricks host.

Hello,I'm trying to connect to our databricks instance using the vscode extension. However, when following this guide we cannot get the configuration to proceed past the point that it asks for our instance URL. The prompt appears to expect a URL of t...

databricks_error
  • 3447 Views
  • 4 replies
  • 0 kudos
Latest Reply
PrawnballNightm
New Contributor III
  • 0 kudos

Hello,Yes, the databricks team shared a modified version of the vs code plugin which did not include the URL matching logic. It connects successfully. However, our custom URL is as it is because our organisation is hosting its own instance of Databri...

  • 0 kudos
3 More Replies
bkbarunkalita64
by New Contributor II
  • 1007 Views
  • 2 replies
  • 2 kudos

Update query from Azure databricks to Oracle table

I need to update a single row on a on-prem Oracle table via jdbc connection.Please note, I don't want to append, just have to update a row, is it possible ?

  • 1007 Views
  • 2 replies
  • 2 kudos
Latest Reply
Kaniz
Community Manager
  • 2 kudos

Hi @Tony S​ , This article mentioned all the steps required to establish a successful connection with the database using Python.However, you can try this code :- df.write.format("jdbc") \ .option("url", dbServerJdbcUrl) \ .option("user", username...

  • 2 kudos
1 More Replies
Jain
by New Contributor III
  • 3123 Views
  • 4 replies
  • 4 kudos

Unable to use dbutils in Premium

I am unable to use dbutils commands and mkdir, etc also does not work after upgrading my Databricks Workspace from Standard tier to Premium tier.It throws the following error:py4j.security.Py4JSecurityException: Constructor public com.databricks.back...

  • 3123 Views
  • 4 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Abhishek Jain​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...

  • 4 kudos
3 More Replies
gilo12
by New Contributor III
  • 1114 Views
  • 2 replies
  • 0 kudos

Running multiple Databricks queries with golang client

I am using github.com/databricks/databricks-sql-go to connect and run queries on my Databricks warehouse. However when I try to run multiple queries:insert into....; insert into....;It fails and complains on syntax errors.[PARSE_SYNTAX_ERROR] Syntax ...

  • 1114 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Gil Omer​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your...

  • 0 kudos
1 More Replies
gtyhchang
by New Contributor II
  • 791 Views
  • 2 replies
  • 1 kudos

pandas issue

We identify a potential bug in either DBFS or Pandas that when writting a dataframe using Pandas `to_csv`, `to_parquet`, `to_pickle` etc to a mounted ADLS location with read-only service principle didn't throw permission deny exceptions. However, met...

  • 791 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Yung-Hang Chang​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...

  • 1 kudos
1 More Replies
Databricks3
by Contributor
  • 1013 Views
  • 1 replies
  • 1 kudos

Concurrent Insert on a delta table fails if the table contains Identity Columns. Error message are added below.MetadataChangedException: The metadata ...

Concurrent Insert on a delta table fails if the table contains Identity Columns. Error message are added below.MetadataChangedException: The metadata of the Delta table has been changed by a concurrent update. Please try the operation again.

  • 1013 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @SK ASIF ALI​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 1 kudos
Susheel
by New Contributor II
  • 640 Views
  • 1 replies
  • 0 kudos

We have concern with respect to the feasibility of installing the Databricks runtime supported version on-premise. This is because we are planning to...

We have concern with respect to the feasibility of installing the Databricks runtime supported version on-premise. This is because we are planning to migrate the existing Azure Databricks application code to an on-premise environment based on the cl...

  • 640 Views
  • 1 replies
  • 0 kudos
Latest Reply
Susheel
New Contributor II
  • 0 kudos

Any update?

  • 0 kudos
Chhaya
by New Contributor III
  • 1228 Views
  • 2 replies
  • 2 kudos

DLT config/setting json support

hi team,There used to be option to provide DLT pipeline settings either via UI or JSON, but I do not see it anymore after switching to new UI. Is this something expected ? am I missing something ? here is screenshot for reference.

image
  • 1228 Views
  • 2 replies
  • 2 kudos
Latest Reply
User16752245772
Contributor
  • 2 kudos

Hi @Chhaya Vishwakarma​ This option is available, could you please clear the browser cache and try ? or can you try in an incognito window?

  • 2 kudos
1 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels