- 1952 Views
- 2 replies
- 0 kudos
Resolved! Handling Schema Mismatch in DLT Pipeline with from_protobuf Function
Hello Databricks Community,I'm working with a DLT pipeline where I consume Protobuf-serialized messages and attempt to decode them using the Spark "from_protobuf" function. My function kafka_msg_dlt_view is outlined as follows:def kafka_msg_dlt_view(...
- 1952 Views
- 2 replies
- 0 kudos
- 0 kudos
I finally found the appropriate method for configuring the PERMISSIVE mode. With this setup, corrupted protobuf messages will be processed without throwing an exception .withColumn("msg",from_protobuf(col("value"), message_name, desc_file_path, {"mo...
- 0 kudos
- 520 Views
- 0 replies
- 0 kudos
Download Spark Driver logs and event logs from Databricks using API
Reference:PY SDK I can't find any API directly to download the logs. Did I miss anything?Otherwise, I can download via DBFS cluster logs location.
- 520 Views
- 0 replies
- 0 kudos
- 896 Views
- 1 replies
- 0 kudos
Databricks Data Engineer Associate certification exam suspended suddenly
Hi @Cert-TeamI hope this message finds you well. I am writing to request a review of my recently suspended exam. I believe that my situation warrants reconsideration, and I would like to provide some context for your understanding.I applied for Datab...
- 896 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Sujitha or @Kaniz Can you please respond to the above query swiftly. As I need to complete my test quickly. Kindly help me to resume or restart my test.Regards,Fasih Ahmed
- 0 kudos
- 1408 Views
- 1 replies
- 0 kudos
Resolved! Databricks Certification voucher not received
Hi Team,I have attended the Advantage Lakehouse: Fueling Innovation in the Era of Data and AI webinar.Also completed Databricks Lakehouse Fundamentals and feedback survey, but still I have not received the Databricks voucher.Could you please look i...
- 1408 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Mahesh2 , Thank you for posting your concern on Community! To expedite your request, please list your concerns on our ticketing portal. Our support staff would be able to act faster on the resolution (our standard resolution time is 24-48 hours).
- 0 kudos
- 1467 Views
- 0 replies
- 0 kudos
dataframe column structtype metadata not getting saved to unitycatalog
I have the below schemaschema = StructType([StructField(name="Test",dataType=StringType(),nullable=False,metadata={"description": "This is to test metadata description."})])data = [('Test1',), ('Test2',), ('Test3',)] df = spark.createDataFrame(data, ...
- 1467 Views
- 0 replies
- 0 kudos
- 11515 Views
- 8 replies
- 0 kudos
Resolved! databricks data engineer associate exam
Hello Team, I encountered Pathetic experience while attempting my 1st Databricks certification. I was giving the exam and Abruptly, Proctor asked me to show my desk, everything i showed every corner of my bed.. It was neat and clean with no suspiciou...
- 11515 Views
- 8 replies
- 0 kudos
- 3018 Views
- 3 replies
- 0 kudos
API for Databricks code functionality
I have a Databricks notebook for which I want to create an API. From that API I will have to call the notebook and perform certain operations. Result will be sent back to API. I dont want to do via Postman, as someone has to install Postman at their ...
- 3018 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @AJ270990 , Thank you for posting your question in our community! We are happy to assist you. To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
- 0 kudos
- 966 Views
- 2 replies
- 0 kudos
Error ingesting files with databricks jobs
The source path that i want to ingest files with is:"gs://bucket-name/folder1/folder2/*/*.json"I have a file in this path that ends with ".json.gz" and the databricks job ingests this file even though it doesn't suppose to.How can i fix it?Thanks.
- 966 Views
- 2 replies
- 0 kudos
- 1461 Views
- 1 replies
- 0 kudos
DataBricks Certification Exam Got Suspended. Require support for the same.
Hello Team, I encountered Pathetic experience while attempting my 1st DataBricks certification. Abruptly, Proctor asked me to show my desk, after showing he/she asked multiple times.. wasted my time and then suspended my exam, saying I have exceeded ...
- 1461 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @surya_1527 , Thank you for posting your concern on Community! To expedite your request, please list your concerns on our ticketing portal. Our support staff would be able to act faster on the resolution (our standard resolution time is 24-48 hour...
- 0 kudos
- 1612 Views
- 1 replies
- 0 kudos
How to search on empty string on text filter with Lakeview Dashboards
Hi,I have created a lakeview dashboard with a couple of filters and a table. Now I would like to search if a certain filter (column) has an empty string but if I search for ' ' then it goes 'no data'. I am wondering how can I search for an empty stri...
- 1612 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Siebert_Looije , To search for an empty string in a text field filter in the Databricks Lakehouse Table view, you can use the "Null" filter option, which also includes empty strings. Here's how you can filter out rows with empty string values in...
- 0 kudos
- 2044 Views
- 3 replies
- 1 kudos
Add Oracle Jar to Databricks cluster policy
I created a policy for users to use when they create their own Job clusters. When I'm editing the policy, I don't have the UI options for adding library (I can only see Definitions and Permissions tabs). I need to add via JSON the option to allows th...
- 2044 Views
- 3 replies
- 1 kudos
- 1 kudos
@adrianhernandez are you admin to workspace, if not you might be missing permissions, if you have policies enabled, admin can allow you.https://docs.databricks.com/en/administration-guide/clusters/policies.html#librariesif your workspace is Unity cat...
- 1 kudos
- 8159 Views
- 1 replies
- 2 kudos
Databricks Asia-Pacific LLM Cup 2023 - So you think you can hack?
You are invited to participate in Databricks Asia-Pacific LLM Cup 2023, an exciting virtual hackathon which kicks off in the month of October. Businesses across Asia-Pacific and Japan will have the opportunity to build, iterate and polish your LLM id...
- 8159 Views
- 1 replies
- 2 kudos
- 2 kudos
7 Up 7 Down: Elevate Your Casino Experience with Vijaybet Online CasinoAre you ready to take your online casino gaming to a whole new level? If you're seeking a thrilling and immersive casino experience, look no further than Vijaybet Online Casino. O...
- 2 kudos
- 1422 Views
- 0 replies
- 2 kudos
dbutils.fs.ls MAX_LIST_SIZE_EXCEEDED
Hi!I'm experiencing different behaviours between two DBX Workspaces when trying to list file contents from an abfss: location.In workspace A running len(dbutils.fs.ls('abfss://~~@~~~~.dfs.core.windows.net/~~/')) results in "Out[1]: 1551", while runni...
- 1422 Views
- 0 replies
- 2 kudos
- 2314 Views
- 5 replies
- 1 kudos
getArgument works fine in interactive cluster 10.4 LTS, raises error in interactive cluster 10.4 LTS
Hello,I am trying to use the getArgument() function in a spark.sql query. It works fine if I run the notebook via an interactive cluster, but gives an error when executed via a job run in an instance Pool.query:OPTIMIZE <table>where date = replace(re...
- 2314 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi @Kaniz_Fatma,Would you be able to respond to my last comment? I couldn't manage to get it working yet.Thank you in advance.
- 1 kudos
- 1768 Views
- 1 replies
- 0 kudos
AWS Databricks VS AWS EMR
HiWhich services should I use for data lake implementation?any cost comparison between Databricks and aws emr.which one is best to choose
- 1768 Views
- 1 replies
- 0 kudos
- 0 kudos
@AH that depends on use case, if your implementation involves Data Lake, ML, Data engineering tasks better to go with databricks as it has got good UI and there good governance using unity catalog for your data lake and you have good consumer tool su...
- 0 kudos
