cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Jennifer
by New Contributor III
  • 4361 Views
  • 5 replies
  • 0 kudos

Resolved! Import python file to notebook doesn't work

I followed the documentation here under the section "Import a file into a notebook" to import a shared python file among notebooks used by delta live table. But it sometimes can find the module, sometimes not and returns me exception No module named ...

  • 4361 Views
  • 5 replies
  • 0 kudos
Latest Reply
Vartika
Moderator
  • 0 kudos

Thank you so much for getting back to us @Jennifer MJ​ . It's really great of you to send in the solution. Would you be happy to mark the answer as best so other community members can find the solution quickly and easily? We really appreciate your ti...

  • 0 kudos
4 More Replies
avnish26
by New Contributor III
  • 7514 Views
  • 4 replies
  • 8 kudos

Spark 3.3.0 connect kafka problem

I am trying to connect to my Kafka from spark but getting an error:Kafka Version: 2.4.1Spark Version: 3.3.0I am using jupyter notebook to execute the pyspark code below:```from pyspark.sql.functions import *from pyspark.sql.types import *#import libr...

  • 7514 Views
  • 4 replies
  • 8 kudos
Latest Reply
jose_gonzalez
Moderator
  • 8 kudos

Hi @avnish26, did you added the Jar files to the cluster? do you still have issues? please let us know

  • 8 kudos
3 More Replies
mickniz
by Contributor
  • 15579 Views
  • 7 replies
  • 18 kudos

cannot import name 'sql' from 'databricks'

I am working on Databricks version 10.4 premium cluster and while importing sql from databricks module I am getting below error. cannot import name 'sql' from 'databricks' (/databricks/python/lib/python3.8/site-packages/databricks/__init__.py).Trying...

  • 15579 Views
  • 7 replies
  • 18 kudos
Latest Reply
wallystart
New Contributor II
  • 18 kudos

I resolve the same error installing library from cluster interface (UI)

  • 18 kudos
6 More Replies
tirato
by New Contributor
  • 1603 Views
  • 3 replies
  • 2 kudos

Resolved! Cannot import-dir from AzureDevops, but works fine locally.

Hello,as i'm trying to create a CI/CD for the project, I'm finding myself stuck.Tried to upload the Notebooks from my Azure DevOps Release and I'm getting 403-forbidden access.I used 'cat ~/.databrickscfg file and matched with the local config that I...

  • 1603 Views
  • 3 replies
  • 2 kudos
Latest Reply
valeryuaba
New Contributor III
  • 2 kudos

Hey everyone! I can totally relate to the frustration of encountering authentication issues when setting up a CI/CD pipeline. It's great that you're able to import the notebooks locally, but facing difficulties on Azure DevOps can be quite puzzling.F...

  • 2 kudos
2 More Replies
Arby
by New Contributor II
  • 5852 Views
  • 3 replies
  • 0 kudos

Help With OSError: [Errno 95] Operation not supported: '/Workspace/Repos/Connectors....

Hello,I am experiencing issues with importing from utils repo the schema file I created.this is the logic we use for all ingestion and all other schemas live in this repo utills/schemasI am unable to access the file I created for a new ingestion pipe...

icon
  • 5852 Views
  • 3 replies
  • 0 kudos
Latest Reply
Arby
New Contributor II
  • 0 kudos

@Debayan Mukherjee​ Hello, thank you for your response. please let me know if these are the correct commands to access the file from notebookI can see the files in the repo folderbut I just noticed this. the file I am trying to access the size is 0 b...

  • 0 kudos
2 More Replies
Mikki007
by New Contributor II
  • 3691 Views
  • 2 replies
  • 1 kudos

Resolved! Constructor public org.apache.spark.SparkConf(boolean) is not whitelisted.

My code:from great_expectations.datasource import SparkDFDatasourcefrom pyspark.sql import SparkSessionsession_name = 'mk_spark_session'spark = SparkSession.builder.appName(session_name).getOrCreate()datasource = SparkDFDatasource(spark)query = "SELE...

  • 3691 Views
  • 2 replies
  • 1 kudos
Latest Reply
jose_gonzalez
Moderator
  • 1 kudos

Are you using a high concurrency cluster? if you do, please try to run this code on standard cluster

  • 1 kudos
1 More Replies
Papa_PSL
by New Contributor
  • 924 Views
  • 2 replies
  • 1 kudos

I want to import a notebook

Want to import a note book, facing this following error:Importation fails: could not deserialize: Unkown language htm

  • 924 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @BITAIGOT Guy-Serge​ We haven't heard from you since the last response from @Debayan Mukherjee​ ​, and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be hel...

  • 1 kudos
1 More Replies
sanjay
by Valued Contributor II
  • 3473 Views
  • 2 replies
  • 1 kudos

Resolved! ImportError: cannot import name dataclass_transform

Hi,I am using Standard Runtime 11.3 LTS and trying to utilize spacy-> en_core_web_sm but I am getting following error.ImportError: cannot import name dataclass_transformIt was working last week but stopped working recently.Appreciate any help. Regard...

  • 3473 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Sanjay Jain​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...

  • 1 kudos
1 More Replies
wb
by New Contributor II
  • 825 Views
  • 1 replies
  • 2 kudos

Import paths using repos and installed libraries get confused

We use Azure Devops and Azure Databricks and have custom Python libraries. I placed my notebooks in the same repo and the structure is like this:mylib/ mylib/__init__.pyt mylib/code.py notebooks/ notebooks/job_notebook.py setup.pyAzure pipelines buil...

  • 825 Views
  • 1 replies
  • 2 kudos
Latest Reply
Avinash_94
New Contributor III
  • 2 kudos

It looks for the configs locally i suppose if you can share requirements .txt i can elaborate

  • 2 kudos
Baumeister
by New Contributor II
  • 1734 Views
  • 2 replies
  • 0 kudos

Error when importing .dbc of a complete Workspace

I saved the content of an older Databricks Workspace by clicking on the Dropdown next to Workspace -> Export -> DBC Archive and saved it on my local machine.In a new Databricks Workspace, I now want to import That .DBC archive to restore the previous...

dbcerror
  • 1734 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

@Sebastian K​ :It looks like the error you are facing while importing the DBC archive could be due to the version incompatibility between the Databricks instance where you created the DBC archive and the one where you are trying to import it. Can you...

  • 0 kudos
1 More Replies
guru1
by New Contributor II
  • 2817 Views
  • 2 replies
  • 0 kudos

Resolved! facing issue mentioned in body when connecting event hub with databricks , followed earlier discussion on this but no solution

ERROR: Query termination received for [id=37bada03-131b-4fbb-8992-a427263fef2c, runId=cf3d7c18-780e-43ae-aed0-9daf2939b823], with exception: java.lang.IllegalArgumentException: Input byte array has wrong 4-byte ending unit at java.util.Base64$Decoder...

  • 2817 Views
  • 2 replies
  • 0 kudos
Latest Reply
Annapurna_Hiriy
New Contributor III
  • 0 kudos

The issue could be due to the mismatch in the eventHub jar and the dependencies added. Also, not all the required dependencies may be added.Suggestions:Using the azure_eventhubs_spark_2_12_.jar eventHub spark jar along with the following dependencies...

  • 0 kudos
1 More Replies
db-avengers2rul
by Contributor II
  • 1242 Views
  • 1 replies
  • 0 kudos

Resolved! zip file not able to import in workspace

Dear Team,Using the community edition when i tried to import a zip file it is always throwing some error

  • 1242 Views
  • 1 replies
  • 0 kudos
Latest Reply
db-avengers2rul
Contributor II
  • 0 kudos

Please refer to the error in the attachment my question is this restriction is only for community edition ? or also for premium account ?

  • 0 kudos
sreedata
by New Contributor III
  • 2829 Views
  • 5 replies
  • 12 kudos

Resolved! Date field getting changed when reading from excel file to dataframe

The date field is getting changed while reading data from source .xls file to the dataframe. In the source xl file all columns are strings but i am not sure why date column alone behaves differentlyIn Source file date is 1/24/2022.In dataframe it is ...

  • 2829 Views
  • 5 replies
  • 12 kudos
Latest Reply
Pradeep_Namani
New Contributor III
  • 12 kudos

Hi Team, @Merca Ovnerud​ I am also facing same issue , below is the code snippet which I am using df=spark.read.format("com.crealytics.spark.excel").option("header","true").load("/mnt/dataplatform/Tenant_PK/Results.xlsx")I have a couple of date colum...

  • 12 kudos
4 More Replies
avnerrhh
by New Contributor III
  • 2629 Views
  • 6 replies
  • 4 kudos

Resolved! How do I import class/functions so it work in Databricks and in my IDE

I already saw this postI want my code to work on both platforms (Databricks and PyCharm), is there any way to do it?

  • 2629 Views
  • 6 replies
  • 4 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 4 kudos

yes.one way is to develop everything locally on your pc, so you also need to have spark installed.This is of course not ideal as you will not have some interesting stuff that databricks provides.But it can be done. What you have to do is create a whl...

  • 4 kudos
5 More Replies
swetha
by New Contributor III
  • 1559 Views
  • 2 replies
  • 1 kudos

I am unable to attach a streaming listener to a spark streaming job. Error: no streaming listener attached to the spark application is the error we are observing post accessing streaming statistics API. Please help us with this issue ASAP. Thanks.

Issue:After adding the listener jar file in the cluster init script, the listener is working (From what I see in the stdout/log4j logs)But when I try to hit the 'Content-Type: application/json' http://host:port/api/v1/applications/app-id/streaming/st...

  • 1559 Views
  • 2 replies
  • 1 kudos
Latest Reply
Vidula
Honored Contributor
  • 1 kudos

Hi @swetha kadiyala​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...

  • 1 kudos
1 More Replies
Labels