cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Mumrel
by Contributor
  • 1439 Views
  • 2 replies
  • 2 kudos

Resolved! Error 95 when importing one Notebook into another

When I follow the instructions Modularize your code using files I get the following error:I am on azure, use DBRT 12.2 LTS, use ADLS as storage, I am happy to provide more details if needed. My research suggest that the reason is that the dfbs fuse...

image
  • 1439 Views
  • 2 replies
  • 2 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 2 kudos

import works for .py files..%run is for notebooks.is lib a .py file or a notebook?

  • 2 kudos
1 More Replies
AmineHY
by Contributor
  • 2372 Views
  • 4 replies
  • 6 kudos

Resolved! Error When Starting the Cluster

I am having this error when running my cluster, any idea why?

  • 2372 Views
  • 4 replies
  • 6 kudos
Latest Reply
NandiniN
Honored Contributor
  • 6 kudos

@Werner Stinckens​ , I checked again, you cannot change them after your workspace is deployed. The only way right now is to recreate the workspace and migrate. It’s not possible to update CIDR range right now without migration.

  • 6 kudos
3 More Replies
Krish1
by New Contributor II
  • 3456 Views
  • 4 replies
  • 0 kudos

Error while mounting ADLS in python using AccountKey

I'm using the below code using Account key to mount ADLS in python but running into error:shaded.databricks.org.apache.hadoop.fs.azure.AzureException: java.lang.IllegalArgumentException: The String is not a valid Base64-encoded string. Can you pleas...

  • 3456 Views
  • 4 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Krish Lam​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers you...

  • 0 kudos
3 More Replies
ivanychev
by Contributor
  • 3563 Views
  • 7 replies
  • 5 kudos

DBR 12.2: DeltaOptimizedWriter: Resolved attribute(s) missing from in operator

After upgrading from DBR 11.3 LTS to DBR 12.2 LTS we started to observe the following error during "read from parquet and write to delta" piece of logic.AnalysisException: Resolved attribute(s) group_id#72,display_name#73,parent_id#74,path#75,path_li...

  • 3563 Views
  • 7 replies
  • 5 kudos
Latest Reply
Valtor
New Contributor II
  • 5 kudos

I can confirm that this issue is resolved for us as well in the latest 12.2 release.

  • 5 kudos
6 More Replies
vdp_dlv
by New Contributor III
  • 1578 Views
  • 3 replies
  • 0 kudos

Resolved! when trying to use %run to a notebook, I'm getting an error.

this error is occurring randomly. sometimes it resolves on its own. not sure what is the cause of the error. The notebook I'm sourcing runs flawlessly. I'm only trying to import dates from this notebook

  • 1578 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @viswa p​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we ca...

  • 0 kudos
2 More Replies
JLSy
by New Contributor III
  • 9284 Views
  • 5 replies
  • 6 kudos

cannot convert Parquet type INT64 to Photon type string

I am receiving an error similar to the post in this link: https://community.databricks.com/s/question/0D58Y00009d8h4tSAA/cannot-convert-parquet-type-int64-to-photon-type-doubleHowever, instead of type double the error message states that the type can...

  • 9284 Views
  • 5 replies
  • 6 kudos
Latest Reply
Anonymous
Not applicable
  • 6 kudos

@John Laurence Sy​ :It sounds like you are encountering a schema conversion error when trying to read in a Parquet file that contains an INT64 column that cannot be converted to a string type. This error can occur when the Parquet file has a schema t...

  • 6 kudos
4 More Replies
Anonymous
by Not applicable
  • 1575 Views
  • 1 replies
  • 1 kudos

"[PARSE_SYNTAX_ERROR] Syntax error at or near 'ROW'(line 2, pos 4)".

Alter table <TABLE_NAME> SET ROW FILTER <func_name> on (COLUMN)Got the below error while running the below code "[PARSE_SYNTAX_ERROR] Syntax error at or near 'ROW'(line 2, pos 4)". Please help on this issue. we tried this code as part of access polic...

Image
  • 1575 Views
  • 1 replies
  • 1 kudos
Latest Reply
Rajeev45
New Contributor III
  • 1 kudos

Hello,Please can you confirm which DBR version are you using & do you use unity catalog?

  • 1 kudos
akanksha_gupta
by New Contributor II
  • 1434 Views
  • 2 replies
  • 0 kudos

ERROR : Failure starting repl. Try detaching and re-attaching the notebook. Getting this error when running any python command in 10.4LTS cluster configured with https://github.com/mspnp/spark-monitoring to send databricks spark logs to Log Analytics.

ERROR Description :java.lang.Exception: Cannot run program &quot;/local_disk0/pythonVirtualEnvDirs/virtualEnv-5acc1ea9-d03f-4de3-b76b-203d42614000/bin/python&quot; (in directory &quot;.&quot;): error=2, No such file or directory at java.lang.ProcessB...

  • 1434 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

@Akanksha Gupta​ :The error message suggests that the Python executable file specified in the configuration of the Databricks cluster cannot be found or accessed. Specifically, it seems that the Python executable file at the path "/local_disk0/python...

  • 0 kudos
1 More Replies
ayesharahmat
by New Contributor II
  • 1818 Views
  • 3 replies
  • 2 kudos

AutoLoader issue - java.lang.AssertionError

The below error I am encountering . I am using microbatch for autoloader. please help to rectify this issuejava.lang.AssertionError: assertion failed: Invalid batch: path#36188,modificationTime#36189,length#36190L,content#36191,PROVIDER#36192,LOCATIO...

  • 1818 Views
  • 3 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

@Ayesha Rahmatali​ :The error message you provided suggests that there is an assertion failure due to invalid batch data in your AutoLoader implementation. The error specifically indicates that the schema of the incoming data is not matching with the...

  • 2 kudos
2 More Replies
tototox
by New Contributor III
  • 2470 Views
  • 3 replies
  • 0 kudos

Using dbutils.fs.ls gives overlap error.

I created a schema with that route as a managed location.(abfss://~~@~~.dfs.core.windows.net/dejeong)And an external table named 'first_table' was created in the corresponding path.(abfss://~~@~~.dfs.core.windows.net/dejeong/first_table)​The results ...

  • 2470 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @jin park​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we c...

  • 0 kudos
2 More Replies
kaileena
by New Contributor
  • 1693 Views
  • 2 replies
  • 0 kudos

Error in library(RMySQL): there is no package called ‘RMySQL’

i tried to install RMySQL on databricks like this:install.packages("RMySQL")i got this error:Installing package into ‘/local_disk0/.ephemeral_nfs/envs/rEnv-c677bc4c-e6a3-40df-a5ab-bfd5d277e0c0’ (as ‘lib’ is unspecified) Warning: unable to access inde...

  • 1693 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @miru miro​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers you...

  • 0 kudos
1 More Replies
JLCDA
by New Contributor
  • 1799 Views
  • 2 replies
  • 0 kudos

databricks-connect 9.1 : StreamCorruptedException: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe

Hello, I'm using databricks-connect 9.1 and I started having issues since last week in all functions that have a "collect()". Everything was working before : myList = df1.select("id").rdd.flatMap(lambda x: x).collect()here the error : py4j.protocol.P...

  • 1799 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Julien Larcher​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...

  • 0 kudos
1 More Replies
Punnu
by New Contributor
  • 1181 Views
  • 1 replies
  • 0 kudos

Error while running spark.catalog.listDatabases()

I am running steps mentioned in https://github.com/databrickslabs/splunk-integration/blob/master/notebooks/source/push_to_splunk.pyWhen I am running spark.catalog.listDatabases()getting error py4j.security.Py4JSecurityException: Method public java.l...

  • 1181 Views
  • 1 replies
  • 0 kudos
Latest Reply
pvignesh92
Honored Contributor
  • 0 kudos

Hi @Purnima Bhatia​ , I faced a similar error for a different command when I was using a wrong type of cluster access mode. You can try to create a different cluster with different access mode and check. I might be wrong but try and check this.

  • 0 kudos
PraveenC
by New Contributor II
  • 2088 Views
  • 4 replies
  • 3 kudos

[Databricks][JDBC](10400) Invalid type for data - column: 10, type: Array

Getting below error while mapping an Array Column to String[] entity. Please suggest if Databricks JDBC support entity mapping of Array Values [Worked the same code for below config - H2 DB version - 2.1.214 and org.hibernate.dialect.H2Dialect - ...

  • 2088 Views
  • 4 replies
  • 3 kudos
Latest Reply
Atanu
Esteemed Contributor
  • 3 kudos

Hello @Emmanuel Trindade​  @Praveen C​  This does not look like coming from Databricks end. Look at the error thread.javax.persistence.PersistenceException: org.hibernate.exception.DataException: Could not read entity state from ResultSet : EntityKey...

  • 3 kudos
3 More Replies
Labels