- 16247 Views
- 21 replies
- 11 kudos
java.lang.IllegalArgumentException: java.net.URISyntaxException
I am using Databricks Autoloader to load JSON files from ADLS gen2 incrementally in directory listing mode. All source filename has Timestamp on them. The autoloader works perfectly couple of days with the below configuration and breaks the next day ...
- 16247 Views
- 21 replies
- 11 kudos
- 11 kudos
Hi Everyone,I'm seeing this issue as well - same configuration of the previous posts, using autoloader with incremental file listing turned on. The strange part is that it mostly works despite almost all of the files we're loading having colons incl...
- 11 kudos
- 4499 Views
- 4 replies
- 2 kudos
spark-streaming read from specific event hub partition
The azure event hub "my_event_hub" has a total of 5 partitions ("0", "1", "2", "3", "4")The readstream should only read events from partitions "0" and "4"event hub configuration as streaming source:-val name = "my_event_hub" val connectionString = "m...
- 4499 Views
- 4 replies
- 2 kudos
- 2 kudos
I tried using below snippet to receive messages only from partition id=0ehName = "<<EVENT-HUB-NAME>>" # Create event position for partition 0 positionKey1 = { "ehName": ehName, "partitionId": 0 } eventPosition1 = { "offset": "@latest", ...
- 2 kudos
- 3996 Views
- 3 replies
- 0 kudos
Resolved! how to add the background color to excel sheet by python
i just want to add color to excel sheet by python to specific cells, and i done that, but i need to exclude the header column, then if i tried the same method to other sheet it doesn't worked.​​but that bg color addition is reflected in one sheet but...
- 3996 Views
- 3 replies
- 0 kudos
- 0 kudos
Convert your dataframe to pandas on sparkcolor cells using style property https://spark.apache.org/docs/latest/api/python/reference/pyspark.pandas/api/pyspark.pandas.DataFrame.style.htmlexport to excel using pandas to_excel https://spark.apache.org/d...
- 0 kudos
- 1118 Views
- 1 replies
- 0 kudos
Relation between Driver and Executor size
HiI would like to ask for recommendations regarding the size of the driver and the amount of executors managed by that driver. I am aware of the best practices regarding executor size/number but I have doubts about the number of executors a single dr...
- 1118 Views
- 1 replies
- 0 kudos
- 0 kudos
Depends on your use case. The best is to connect Datatog and see driver and workers utilization https://docs.datadoghq.com/integrations/databricks/?tab=driveronlyJust from my experience, Usually, for big datasets, when need autoscale workers between ...
- 0 kudos
- 2250 Views
- 1 replies
- 1 kudos
Resolved! Task time Spark UI
Hello all,I would like to know why task times (among other times in Spark UI) display values like 1h 2h when the task does only really take some seconds or minutes. What is the meaning of these high time values I see all around Spark UI.Thanks in adv...
- 2250 Views
- 1 replies
- 1 kudos
- 1 kudos
that is accumulated time.https://stackoverflow.com/questions/73302982/task-time-and-gc-time-calculation-in-spark-ui-in-executor-section.
- 1 kudos
- 2760 Views
- 3 replies
- 0 kudos
Error when accessing the file from azure blob storage
I am getting the following error when accessing the file in Azure blob storagejava.io.FileNotFoundException: File /10433893690638/mnt/22200/22200Ver1.sps does not exist.Code:ves_blob = dbutils.widgets.get("ves_blob") try: dbutils.fs.ls(ves_blob ) e...
- 2760 Views
- 3 replies
- 0 kudos
- 0 kudos
that is certainly an invalid path, as the error shows.with %fs ls /mnt you can show the directory structure of the /mnt directory, assuming the blob storage is mounted.if not, you need to define the access ( URL etc.)
- 0 kudos
- 1930 Views
- 3 replies
- 0 kudos
Query API Result
Hi, I'm new here.Currently I have to read information from a query in databricks. I've used the query API to get the query definition but so far I'm not able to run the query and get the results.Is it possible? Thanks
- 1930 Views
- 3 replies
- 0 kudos
- 0 kudos
When using the JobsAPI you need to specify dbutils.notebook.exit("returnValue") to pass the results once the notebook finished it's job (https://docs.databricks.com/notebooks/notebook-workflows.html#notebook-workflows-exit).Then you can get notebook_...
- 0 kudos
- 5843 Views
- 6 replies
- 1 kudos
Resolved! how to check dataframe column value
in my dataframe it have one column name like count, if that particular column value is greater than zero, the job needs to get failed, how can i perform that one?​
- 5843 Views
- 6 replies
- 1 kudos
- 1 kudos
Code without collect, which should not be used in production:if df.filter("count > 0").count() > 0: dbutils.notebook.exit('Notebook Failed')you can also use a more aggressive version:if df.filter("count > 0").count() > 0: raise Exception("count bigge...
- 1 kudos
- 3238 Views
- 4 replies
- 3 kudos
Resolved! Is there a known issue regarding Databricks JDBC driver character values such as Japanese etc?
A Parquet file contains character data for various languages and is shown by the Data Explorer UX. A simple "select *" query using the Databricks JDBC driver (version 2.6.29) with a tool such as SQLSquirrel displays invalid characters.
- 3238 Views
- 4 replies
- 3 kudos
- 3 kudos
The issue encountered has been confirmed to be a defect in the Databricks JDBC driver.
- 3 kudos
- 2414 Views
- 3 replies
- 2 kudos
Job runs indefinitely after integrating with PyDeequ
I'm using PyDeequ data quality checks in one of our jobs. After adding this check, I noticed that the job does not complete and keeps running indefinitely after PyDeequ checks are completed and results are returned.As stated in Pydeequ documentation ...
- 2414 Views
- 3 replies
- 2 kudos
- 2 kudos
Hm, deequ certainly works as I have read about multiple people using it.And when reading the issues (open/closed) on the github pages of pydeequ, databricks is mentioned in some issues so it might be possible after all.But I think you need to check y...
- 2 kudos
- 3472 Views
- 4 replies
- 6 kudos
Resolved! How to parameterize key of spark config in the job clusterlinked service from ADF
how can we parameterize key of the spark-config in the job cluster linked service from Azure datafactory, we can parameterize the values but any idea how can we parameterize the key so that when deploying to further environment it takes the PROD/QA v...
- 3472 Views
- 4 replies
- 6 kudos
- 6 kudos
@KVNARK .​ You can use Databricks Secrets (create a Secret scope from AKV https://learn.microsoft.com/en-us/azure/databricks/security/secrets/secret-scopes) and then reference a secret in spark configuration (https://learn.microsoft.com/en-us/azure/d...
- 6 kudos
- 3997 Views
- 2 replies
- 1 kudos
Resolved! Attach instance profile to service principal.
Hey Guys, I'm having some permission issues using service principal and instance profile and i hope you could help me.I created a service principal and attached to it an instance profile - databricks-my-profile.I have a s3 bucket with policy that all...
- 3997 Views
- 2 replies
- 1 kudos
- 1 kudos
Hey @Kaniz Fatma​ , @Debayan Mukherjee​, Thanks for your answers.Actually, Databricks is not support using DBFS API with service principal & attached instance profile on a mounted s3 bucket.I'm not sure if this exists in docs (might miss it) but thi...
- 1 kudos
- 7401 Views
- 2 replies
- 0 kudos
Relative path in absolute URI when reading a folder with files containing ":" colons in filename
I am trying to read a folder with partition files where each partition is date/hour/timestamp.csv where timestamp is the exact timestamp in ISO format, e.g. 09-2022-12-05T20:35:15.2786966Z It seems like spark having issues with reading files with col...
- 7401 Views
- 2 replies
- 0 kudos
- 0 kudos
The issue was reopened again https://issues.apache.org/jira/browse/HDFS-14762
- 0 kudos
- 9538 Views
- 5 replies
- 7 kudos
Databricks Azure Blob Storage access
I am trying to access files stored in Azure blob storage and have followed the documentation linked below:https://docs.databricks.com/external-data/azure-storage.htmlI was successful in mounting the Azure blob storage on dbfs but it seems that the me...
- 9538 Views
- 5 replies
- 7 kudos
- 7 kudos
Hi, @Ravindra Ch​ , could you please check the firewall settings in Azure networking?
- 7 kudos
- 3932 Views
- 2 replies
- 2 kudos
Transforming/Saving Python Class Instances to Delta Rows
I'm trying to reuse a Python Package to do a very complex series of parsing binary files into workable data in Delta Format. I have made the first part (binary file parsing) work with a UDF:asffileparser = F.udf(File()._parseBytes,AsfFileDelta.getSch...
- 3932 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi, did you try to follow, "Fix it by registering a custom IObjectConstructor for this class."?Also, could you please provide us the full error?
- 2 kudos