- 11401 Views
- 5 replies
- 8 kudos
Spark 3.3.0 connect kafka problem
I am trying to connect to my Kafka from spark but getting an error:Kafka Version: 2.4.1Spark Version: 3.3.0I am using jupyter notebook to execute the pyspark code below:```from pyspark.sql.functions import *from pyspark.sql.types import *#import libr...
- 11401 Views
- 5 replies
- 8 kudos
- 8 kudos
Hi @avnish26, did you added the Jar files to the cluster? do you still have issues? please let us know
- 8 kudos
- 2731 Views
- 3 replies
- 1 kudos
I am unable to attach a streaming listener to a spark streaming job. Error: no streaming listener attached to the spark application is the error we are observing post accessing streaming statistics API. Please help us with this issue ASAP. Thanks.
Issue:After adding the listener jar file in the cluster init script, the listener is working (From what I see in the stdout/log4j logs)But when I try to hit the 'Content-Type: application/json' http://host:port/api/v1/applications/app-id/streaming/st...
- 2731 Views
- 3 replies
- 1 kudos
- 25796 Views
- 8 replies
- 18 kudos
cannot import name 'sql' from 'databricks'
I am working on Databricks version 10.4 premium cluster and while importing sql from databricks module I am getting below error. cannot import name 'sql' from 'databricks' (/databricks/python/lib/python3.8/site-packages/databricks/__init__.py).Trying...
- 25796 Views
- 8 replies
- 18 kudos
- 18 kudos
if you ever received this kind of error after installing the correct Python package then try running the below command. dbutils.library.restartPython()
- 18 kudos
- 4857 Views
- 5 replies
- 2 kudos
Graphframes not importing on Databricks 9.1 LTS ML
Is Graphframes for python meant to be installed by default on Databricks 9.1 LTS ML? Previously I was running the attached python command on 7.3 LTS ML with no issue, however now I am getting "no module named graphframes" when trying to import the pa...
- 4857 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @MuthuLakshmi , As per the documentation it was mentioned that graphframes comes preinstalled in databricks runtime for machine learning. but when trying to import the python module of graphframes, getting no module found error.from graphframes i...
- 2 kudos
- 26091 Views
- 5 replies
- 11 kudos
Resolved! How to import a function to another notebook using Repos without %run?
Hi all,I was reading the Repos documentation: https://docs.databricks.com/repos.html#migrate-from-run-commandsIt is explained that, one advantage of Repos is no longer necessary to use %run magic command to make funcions available in one notebook to ...
- 26091 Views
- 5 replies
- 11 kudos
- 11 kudos
Due to new functionalies in Runtime 16.0 regarding autoload i came across this autoload. Performaed a practical test. It works. However had some problems at first.As in solution the key was that definitions are places in a file.py not a notebook.
- 11 kudos
- 12044 Views
- 4 replies
- 0 kudos
Help With OSError: [Errno 95] Operation not supported: '/Workspace/Repos/Connectors....
Hello,I am experiencing issues with importing from utils repo the schema file I created.this is the logic we use for all ingestion and all other schemas live in this repo utills/schemasI am unable to access the file I created for a new ingestion pipe...
- 12044 Views
- 4 replies
- 0 kudos
- 0 kudos
@Debayan Mukherjee Hello, thank you for your response. please let me know if these are the correct commands to access the file from notebookI can see the files in the repo folderbut I just noticed this. the file I am trying to access the size is 0 b...
- 0 kudos
- 6664 Views
- 5 replies
- 1 kudos
Resolved! Import python file to notebook doesn't work
I followed the documentation here under the section "Import a file into a notebook" to import a shared python file among notebooks used by delta live table. But it sometimes can find the module, sometimes not and returns me exception No module named ...
- 6664 Views
- 5 replies
- 1 kudos
- 1 kudos
Thank you so much for getting back to us @Jennifer MJ . It's really great of you to send in the solution. Would you be happy to mark the answer as best so other community members can find the solution quickly and easily? We really appreciate your ti...
- 1 kudos
- 3082 Views
- 3 replies
- 2 kudos
Resolved! Cannot import-dir from AzureDevops, but works fine locally.
Hello,as i'm trying to create a CI/CD for the project, I'm finding myself stuck.Tried to upload the Notebooks from my Azure DevOps Release and I'm getting 403-forbidden access.I used 'cat ~/.databrickscfg file and matched with the local config that I...
- 3082 Views
- 3 replies
- 2 kudos
- 2 kudos
Hey everyone! I can totally relate to the frustration of encountering authentication issues when setting up a CI/CD pipeline. It's great that you're able to import the notebooks locally, but facing difficulties on Azure DevOps can be quite puzzling.F...
- 2 kudos
- 6201 Views
- 2 replies
- 1 kudos
Resolved! Constructor public org.apache.spark.SparkConf(boolean) is not whitelisted.
My code:from great_expectations.datasource import SparkDFDatasourcefrom pyspark.sql import SparkSessionsession_name = 'mk_spark_session'spark = SparkSession.builder.appName(session_name).getOrCreate()datasource = SparkDFDatasource(spark)query = "SELE...
- 6201 Views
- 2 replies
- 1 kudos
- 1 kudos
Are you using a high concurrency cluster? if you do, please try to run this code on standard cluster
- 1 kudos
- 1625 Views
- 2 replies
- 1 kudos
I want to import a notebook
Want to import a note book, facing this following error:Importation fails: could not deserialize: Unkown language htm
- 1625 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @BITAIGOT Guy-Serge We haven't heard from you since the last response from @Debayan Mukherjee , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be hel...
- 1 kudos
- 4666 Views
- 2 replies
- 1 kudos
Resolved! ImportError: cannot import name dataclass_transform
Hi,I am using Standard Runtime 11.3 LTS and trying to utilize spacy-> en_core_web_sm but I am getting following error.ImportError: cannot import name dataclass_transformIt was working last week but stopped working recently.Appreciate any help. Regard...
- 4666 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Sanjay Jain Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...
- 1 kudos
- 1273 Views
- 1 replies
- 2 kudos
Import paths using repos and installed libraries get confused
We use Azure Devops and Azure Databricks and have custom Python libraries. I placed my notebooks in the same repo and the structure is like this:mylib/ mylib/__init__.pyt mylib/code.py notebooks/ notebooks/job_notebook.py setup.pyAzure pipelines buil...
- 1273 Views
- 1 replies
- 2 kudos
- 2 kudos
It looks for the configs locally i suppose if you can share requirements .txt i can elaborate
- 2 kudos
- 5018 Views
- 2 replies
- 0 kudos
Error when importing .dbc of a complete Workspace
I saved the content of an older Databricks Workspace by clicking on the Dropdown next to Workspace -> Export -> DBC Archive and saved it on my local machine.In a new Databricks Workspace, I now want to import That .DBC archive to restore the previous...
- 5018 Views
- 2 replies
- 0 kudos
- 0 kudos
@Sebastian K :It looks like the error you are facing while importing the DBC archive could be due to the version incompatibility between the Databricks instance where you created the DBC archive and the one where you are trying to import it. Can you...
- 0 kudos
- 4066 Views
- 2 replies
- 0 kudos
Resolved! facing issue mentioned in body when connecting event hub with databricks , followed earlier discussion on this but no solution
ERROR: Query termination received for [id=37bada03-131b-4fbb-8992-a427263fef2c, runId=cf3d7c18-780e-43ae-aed0-9daf2939b823], with exception: java.lang.IllegalArgumentException: Input byte array has wrong 4-byte ending unit at java.util.Base64$Decoder...
- 4066 Views
- 2 replies
- 0 kudos
- 0 kudos
The issue could be due to the mismatch in the eventHub jar and the dependencies added. Also, not all the required dependencies may be added.Suggestions:Using the azure_eventhubs_spark_2_12_.jar eventHub spark jar along with the following dependencies...
- 0 kudos
- 2215 Views
- 1 replies
- 0 kudos
Resolved! zip file not able to import in workspace
Dear Team,Using the community edition when i tried to import a zip file it is always throwing some error
- 2215 Views
- 1 replies
- 0 kudos
- 0 kudos
Please refer to the error in the attachment my question is this restriction is only for community edition ? or also for premium account ?
- 0 kudos