- 1326 Views
- 1 replies
- 0 kudos
Databricks Data Engineer Associate certification exam suspended suddenly
Hi @Cert-TeamI hope this message finds you well. I am writing to request a review of my recently suspended exam. I believe that my situation warrants reconsideration, and I would like to provide some context for your understanding.I applied for Datab...
- 1326 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Sujitha or @Kaniz Can you please respond to the above query swiftly. As I need to complete my test quickly. Kindly help me to resume or restart my test.Regards,Fasih Ahmed
- 0 kudos
- 1634 Views
- 0 replies
- 0 kudos
Databricks Certification voucher not received
Hi Team,I have attended the Advantage Lakehouse: Fueling Innovation in the Era of Data and AI webinar.Also completed Databricks Lakehouse Fundamentals and feedback survey, but still I have not received the Databricks voucher.Could you please look i...
- 1634 Views
- 0 replies
- 0 kudos
- 1573 Views
- 0 replies
- 0 kudos
Sharing compute between tasks of a job
Is there a way to set up a workflow with multiple tasks, so that different tasks can share the same compute resource, at the same time?I understand that an instance pool may be an option, here. Wasn't sure if there were other possible options to cons...
- 1573 Views
- 0 replies
- 0 kudos
- 1597 Views
- 0 replies
- 0 kudos
Databricks data engineer associate got paused
Hi team,I've faced a disappointing experience during my first certification attempt and need help in resolving the issue.While attending the certification - Databricks data engineer associate on each 2-3 questions I kept receiving a message that the ...
- 1597 Views
- 0 replies
- 0 kudos
- 1812 Views
- 0 replies
- 0 kudos
dataframe column structtype metadata not getting saved to unitycatalog
I have the below schemaschema = StructType([StructField(name="Test",dataType=StringType(),nullable=False,metadata={"description": "This is to test metadata description."})])data = [('Test1',), ('Test2',), ('Test3',)] df = spark.createDataFrame(data, ...
- 1812 Views
- 0 replies
- 0 kudos
- 18032 Views
- 1 replies
- 1 kudos
Resolved! Some streams terminated before this command could finish! -> java.lang.NoClassDefFoundError: scala/c
HelloI do face:Some streams terminated before this command could finish!java.lang.NoClassDefFoundError: scala/compat/java8/FutureConverters$Running some very simple query on eventhub :df = spark \.readStream \.format("eventhubs") \.options(**ehConf) ...
- 18032 Views
- 1 replies
- 1 kudos
- 1 kudos
Of course just after writing that post I did realized how dummy this question is .... after adding scala_java8_compat_2_12_1_0_2.jar it works as expected
- 1 kudos
- 1591 Views
- 0 replies
- 0 kudos
Thoughts on how to improve string search queries
Please see sample code I am running below. What options can I explore to improve speed of query execution in such a scenario? Current full code takes about 4 hrs to run on 1.5 billion rows. Thanks!SELECT fullVisitorId ,VisitId ,EventDate ,PagePath ,d...
- 1591 Views
- 0 replies
- 0 kudos
- 3498 Views
- 1 replies
- 0 kudos
API for Databricks code functionality
I have a Databricks notebook for which I want to create an API. From that API I will have to call the notebook and perform certain operations. Result will be sent back to API. I dont want to do via Postman, as someone has to install Postman at their ...
- 3498 Views
- 1 replies
- 0 kudos
- 1177 Views
- 1 replies
- 0 kudos
Error ingesting files with databricks jobs
The source path that i want to ingest files with is:"gs://bucket-name/folder1/folder2/*/*.json"I have a file in this path that ends with ".json.gz" and the databricks job ingests this file even though it doesn't suppose to.How can i fix it?Thanks.
- 1177 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks Kaniz
- 0 kudos
- 9366 Views
- 0 replies
- 0 kudos
Deleted the s3 bucket assocated with metastore
I deleted the aws s3 bucket for the databricks metastore by mistake.How to fix this? can I re-create the s3 bucket? Or can I delete the metastore (I don't have much data in it), and re-generate one? Thank you!
- 9366 Views
- 0 replies
- 0 kudos
- 1728 Views
- 0 replies
- 0 kudos
org.apache.spark.SparkException - FileReadException
Sometimes getting this kind of error "org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 12224.0 failed 4 times, most recent failure: Lost task 1.5 in stage 12224.0 (TID ) (12.xxx.x.xxx executor 1): com.datab...
- 1728 Views
- 0 replies
- 0 kudos
- 2735 Views
- 3 replies
- 1 kudos
Add Oracle Jar to Databricks cluster policy
I created a policy for users to use when they create their own Job clusters. When I'm editing the policy, I don't have the UI options for adding library (I can only see Definitions and Permissions tabs). I need to add via JSON the option to allows th...
- 2735 Views
- 3 replies
- 1 kudos
- 1 kudos
@adrianhernandez are you admin to workspace, if not you might be missing permissions, if you have policies enabled, admin can allow you.https://docs.databricks.com/en/administration-guide/clusters/policies.html#librariesif your workspace is Unity cat...
- 1 kudos
- 2040 Views
- 0 replies
- 2 kudos
dbutils.fs.ls MAX_LIST_SIZE_EXCEEDED
Hi!I'm experiencing different behaviours between two DBX Workspaces when trying to list file contents from an abfss: location.In workspace A running len(dbutils.fs.ls('abfss://~~@~~~~.dfs.core.windows.net/~~/')) results in "Out[1]: 1551", while runni...
- 2040 Views
- 0 replies
- 2 kudos
- 3765 Views
- 3 replies
- 1 kudos
getArgument works fine in interactive cluster 10.4 LTS, raises error in interactive cluster 10.4 LTS
Hello,I am trying to use the getArgument() function in a spark.sql query. It works fine if I run the notebook via an interactive cluster, but gives an error when executed via a job run in an instance Pool.query:OPTIMIZE <table>where date = replace(re...
- 3765 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Retired_mod,Would you be able to respond to my last comment? I couldn't manage to get it working yet.Thank you in advance.
- 1 kudos
- 2353 Views
- 1 replies
- 0 kudos
AWS Databricks VS AWS EMR
HiWhich services should I use for data lake implementation?any cost comparison between Databricks and aws emr.which one is best to choose
- 2353 Views
- 1 replies
- 0 kudos
- 0 kudos
@AH that depends on use case, if your implementation involves Data Lake, ML, Data engineering tasks better to go with databricks as it has got good UI and there good governance using unity catalog for your data lake and you have good consumer tool su...
- 0 kudos
