- 9322 Views
- 4 replies
- 0 kudos
Help With OSError: [Errno 95] Operation not supported: '/Workspace/Repos/Connectors....
Hello,I am experiencing issues with importing from utils repo the schema file I created.this is the logic we use for all ingestion and all other schemas live in this repo utills/schemasI am unable to access the file I created for a new ingestion pipe...
- 9322 Views
- 4 replies
- 0 kudos
- 0 kudos
@Debayan Mukherjee Hello, thank you for your response. please let me know if these are the correct commands to access the file from notebookI can see the files in the repo folderbut I just noticed this. the file I am trying to access the size is 0 b...
- 0 kudos
- 5806 Views
- 5 replies
- 0 kudos
Resolved! Import python file to notebook doesn't work
I followed the documentation here under the section "Import a file into a notebook" to import a shared python file among notebooks used by delta live table. But it sometimes can find the module, sometimes not and returns me exception No module named ...
- 5806 Views
- 5 replies
- 0 kudos
- 0 kudos
Thank you so much for getting back to us @Jennifer MJ . It's really great of you to send in the solution. Would you be happy to mark the answer as best so other community members can find the solution quickly and easily? We really appreciate your ti...
- 0 kudos
- 9502 Views
- 4 replies
- 8 kudos
Spark 3.3.0 connect kafka problem
I am trying to connect to my Kafka from spark but getting an error:Kafka Version: 2.4.1Spark Version: 3.3.0I am using jupyter notebook to execute the pyspark code below:```from pyspark.sql.functions import *from pyspark.sql.types import *#import libr...
- 9502 Views
- 4 replies
- 8 kudos
- 8 kudos
Hi @avnish26, did you added the Jar files to the cluster? do you still have issues? please let us know
- 8 kudos
- 20412 Views
- 7 replies
- 18 kudos
cannot import name 'sql' from 'databricks'
I am working on Databricks version 10.4 premium cluster and while importing sql from databricks module I am getting below error. cannot import name 'sql' from 'databricks' (/databricks/python/lib/python3.8/site-packages/databricks/__init__.py).Trying...
- 20412 Views
- 7 replies
- 18 kudos
- 18 kudos
I resolve the same error installing library from cluster interface (UI)
- 18 kudos
- 2439 Views
- 3 replies
- 2 kudos
Resolved! Cannot import-dir from AzureDevops, but works fine locally.
Hello,as i'm trying to create a CI/CD for the project, I'm finding myself stuck.Tried to upload the Notebooks from my Azure DevOps Release and I'm getting 403-forbidden access.I used 'cat ~/.databrickscfg file and matched with the local config that I...
- 2439 Views
- 3 replies
- 2 kudos
- 2 kudos
Hey everyone! I can totally relate to the frustration of encountering authentication issues when setting up a CI/CD pipeline. It's great that you're able to import the notebooks locally, but facing difficulties on Azure DevOps can be quite puzzling.F...
- 2 kudos
- 4942 Views
- 2 replies
- 1 kudos
Resolved! Constructor public org.apache.spark.SparkConf(boolean) is not whitelisted.
My code:from great_expectations.datasource import SparkDFDatasourcefrom pyspark.sql import SparkSessionsession_name = 'mk_spark_session'spark = SparkSession.builder.appName(session_name).getOrCreate()datasource = SparkDFDatasource(spark)query = "SELE...
- 4942 Views
- 2 replies
- 1 kudos
- 1 kudos
Are you using a high concurrency cluster? if you do, please try to run this code on standard cluster
- 1 kudos
- 1282 Views
- 2 replies
- 1 kudos
I want to import a notebook
Want to import a note book, facing this following error:Importation fails: could not deserialize: Unkown language htm
- 1282 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @BITAIGOT Guy-Serge We haven't heard from you since the last response from @Debayan Mukherjee , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be hel...
- 1 kudos
- 4089 Views
- 2 replies
- 1 kudos
Resolved! ImportError: cannot import name dataclass_transform
Hi,I am using Standard Runtime 11.3 LTS and trying to utilize spacy-> en_core_web_sm but I am getting following error.ImportError: cannot import name dataclass_transformIt was working last week but stopped working recently.Appreciate any help. Regard...
- 4089 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Sanjay Jain Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...
- 1 kudos
- 1088 Views
- 1 replies
- 2 kudos
Import paths using repos and installed libraries get confused
We use Azure Devops and Azure Databricks and have custom Python libraries. I placed my notebooks in the same repo and the structure is like this:mylib/ mylib/__init__.pyt mylib/code.py notebooks/ notebooks/job_notebook.py setup.pyAzure pipelines buil...
- 1088 Views
- 1 replies
- 2 kudos
- 2 kudos
It looks for the configs locally i suppose if you can share requirements .txt i can elaborate
- 2 kudos
- 4304 Views
- 2 replies
- 0 kudos
Error when importing .dbc of a complete Workspace
I saved the content of an older Databricks Workspace by clicking on the Dropdown next to Workspace -> Export -> DBC Archive and saved it on my local machine.In a new Databricks Workspace, I now want to import That .DBC archive to restore the previous...
- 4304 Views
- 2 replies
- 0 kudos
- 0 kudos
@Sebastian K :It looks like the error you are facing while importing the DBC archive could be due to the version incompatibility between the Databricks instance where you created the DBC archive and the one where you are trying to import it. Can you...
- 0 kudos
- 3437 Views
- 2 replies
- 0 kudos
Resolved! facing issue mentioned in body when connecting event hub with databricks , followed earlier discussion on this but no solution
ERROR: Query termination received for [id=37bada03-131b-4fbb-8992-a427263fef2c, runId=cf3d7c18-780e-43ae-aed0-9daf2939b823], with exception: java.lang.IllegalArgumentException: Input byte array has wrong 4-byte ending unit at java.util.Base64$Decoder...
- 3437 Views
- 2 replies
- 0 kudos
- 0 kudos
The issue could be due to the mismatch in the eventHub jar and the dependencies added. Also, not all the required dependencies may be added.Suggestions:Using the azure_eventhubs_spark_2_12_.jar eventHub spark jar along with the following dependencies...
- 0 kudos
- 1758 Views
- 1 replies
- 0 kudos
Resolved! zip file not able to import in workspace
Dear Team,Using the community edition when i tried to import a zip file it is always throwing some error
- 1758 Views
- 1 replies
- 0 kudos
- 0 kudos
Please refer to the error in the attachment my question is this restriction is only for community edition ? or also for premium account ?
- 0 kudos
- 3963 Views
- 5 replies
- 12 kudos
Resolved! Date field getting changed when reading from excel file to dataframe
The date field is getting changed while reading data from source .xls file to the dataframe. In the source xl file all columns are strings but i am not sure why date column alone behaves differentlyIn Source file date is 1/24/2022.In dataframe it is ...
- 3963 Views
- 5 replies
- 12 kudos
- 12 kudos
Hi Team, @Merca Ovnerud I am also facing same issue , below is the code snippet which I am using df=spark.read.format("com.crealytics.spark.excel").option("header","true").load("/mnt/dataplatform/Tenant_PK/Results.xlsx")I have a couple of date colum...
- 12 kudos
- 4016 Views
- 6 replies
- 4 kudos
Resolved! How do I import class/functions so it work in Databricks and in my IDE
I already saw this postI want my code to work on both platforms (Databricks and PyCharm), is there any way to do it?
- 4016 Views
- 6 replies
- 4 kudos
- 4 kudos
yes.one way is to develop everything locally on your pc, so you also need to have spark installed.This is of course not ideal as you will not have some interesting stuff that databricks provides.But it can be done. What you have to do is create a whl...
- 4 kudos
- 2143 Views
- 2 replies
- 1 kudos
I am unable to attach a streaming listener to a spark streaming job. Error: no streaming listener attached to the spark application is the error we are observing post accessing streaming statistics API. Please help us with this issue ASAP. Thanks.
Issue:After adding the listener jar file in the cluster init script, the listener is working (From what I see in the stdout/log4j logs)But when I try to hit the 'Content-Type: application/json' http://host:port/api/v1/applications/app-id/streaming/st...
- 2143 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @swetha kadiyala Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...
- 1 kudos