- 7334 Views
- 3 replies
- 2 kudos
Performance issues when loading an Excel file from DBFS using R
I have uploaded small Excel files on my DBFS. I then use function read_xlsx() from the "readxl" package in R to import the file into the R memory. I use a standard cluster (12.1, non ML). The function works but it takes ages. E.g. a simple Excel tabl...
- 7334 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Marcel Baumgartner​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best an...
- 2 kudos
- 6342 Views
- 4 replies
- 0 kudos
Databricks Associate Exam Webinar Survey link absent
Hi Databricks,I completed the Data Engineer Associate Exam Webinar held on the 11th of April 2020 and haven't received any survey link. How can I give the survey for the exam voucher?Please Guide me on this..
- 6342 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Dhiraj Singh​,Just a friendly follow-up. Do you still need help to get your certificate? or you no longer need help? please les us know
- 0 kudos
- 1606 Views
- 3 replies
- 2 kudos
 Hello Everyone, I am thrilled to announce that we have our first winner for the raffle contest - @Vincent Slegers​. Please join me in congratulating ...
Hello Everyone,I am thrilled to announce that we have our first winner for the raffle contest - @Vincent Slegers​. Please join me in congratulating him on this remarkable achievement!Vincent, your dedication and hard work have paid off, and we are d...
- 1606 Views
- 3 replies
- 2 kudos
- 2832 Views
- 2 replies
- 0 kudos
Issue while using community edition
Hello,Is anyone facing issue with their community edition.?Shows the below error and cant access the workspace or previosly created notebooks..Tried accessing from different devices..(not a cache issue)Error popup:(screenshots attached)Unable to view...
- 2832 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Hari N​ Thank you for reaching out, and we’re sorry to hear about this log-in issue! We have this Community Edition login troubleshooting post on Community. Please take a look, and follow the troubleshooting steps. If the steps do not resolve the...
- 0 kudos
- 2482 Views
- 2 replies
- 0 kudos
Pyspark Structured Streaming Avro integration to Azure Schema Registry with Kafka/Eventhub in Databricks environment.
I am looking for a simple way to have a structured streaming pipeline that would automatically register a schema to Azure schema registry when converting a df col into avro and that would be able to deserialize an avro col based on schema registry ur...
- 2482 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Tomas Sedlon​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
- 0 kudos
- 2186 Views
- 3 replies
- 2 kudos
www.dbdemos.ai
Getting started with Databricks is being made very easy now. Presenting dbdemos.If you're looking to get started with Databricks, there's good news: dbdemos makes it easier than ever. This platform offers a range of demos that you can install directl...
- 2186 Views
- 3 replies
- 2 kudos
- 2 kudos
That's a great share Suteja. Is that supposed to work with the Databricks Community edition account? Had a strange error while trying. Any help is appreciated!Thanks,F
- 2 kudos
- 2980 Views
- 5 replies
- 1 kudos
Does thrift only exist in databrick control plane?
Hi all, I'm a user of Azure databricks. We recently found there is a thrift vulnerability issue (CVE-2020-13949) in Spark Hive. We have tried to fix it at our side. We also found there is a open issue at Spark jira board - https://issues.apache.org/j...
- 2980 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi @Jimin Hsieh​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so w...
- 1 kudos
- 6011 Views
- 11 replies
- 0 kudos
Process batches in a streaming pipeline - identifying deletes
OK. So I think I'm probably missing the obvious and tying myself in knots here.Here is the scenario:batch datasets arrive in json format in an Azure data lakeeach batch is a complete set of "current" records (the complete table)these are processed us...
- 6011 Views
- 11 replies
- 0 kudos
- 0 kudos
Hi @Kearon McNicol​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...
- 0 kudos
- 9605 Views
- 19 replies
- 1 kudos
Can not access SQL files in the Shared workspace
Hey,we have an issue in that we can access the SQL files whenever the notebook is in the repo path but whenever the CICD pipeline imports the repo notebooks and SQL files to the shared workspace, we can list the SQL files but can not read them.we cha...
- 9605 Views
- 19 replies
- 1 kudos
- 1 kudos
@Nermin Yehia​ yes, as you are moving files to different location manually , just update as can manage permissions in target and that should take care of everything
- 1 kudos
- 2055 Views
- 3 replies
- 0 kudos
Resolved! Delta Live Table Service Upgrade
Dear experts, Might I know what will happen to the delta live table pipeline which is in a cancelled state, when there is a runtime service upgrade? Thanks!
- 2055 Views
- 3 replies
- 0 kudos
- 0 kudos
@KS LAU​ :When a runtime service upgrade occurs in Databricks, any running tasks or pipelines may be temporarily interrupted while the upgrade is being applied. In the case of a cancelled Delta Live Table pipeline, it will not be impacted by the upgr...
- 0 kudos
- 4520 Views
- 4 replies
- 2 kudos
Resolved! DLT target schema - get value during run time
Hi,I would like to know if it is possible to get the target schema, programmatically, inside a DLT.In DLT pipeline settings, destination, target schema.I want to run more idempotent pipelines. For example, my target table has the fields: reference_da...
- 4520 Views
- 4 replies
- 2 kudos
- 2 kudos
Thank you @Suteja Kanuri​ ,Looks like you solution is working, thank you.Regards,
- 2 kudos
- 2807 Views
- 1 replies
- 1 kudos
Resolved! sedona/shapely error Unknown WKB type 16
Hi,i stream data from postgis to s3 using debezium. postgis->debezium->s3->spark(databricks)once read it i decode it and i can see that the binary representation is similiar to what i have in postgis, on a wkb formated column.once i try to read it ei...
- 2807 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi,i stream data from postgis to s3 using debezium. postgis->debezium->s3->spark(databricks)once read it i decode it and i can see that the binary representation is similiar to what i have in postgis, on a wkb formated column.once i try to read it ei...
- 1 kudos
- 6501 Views
- 3 replies
- 1 kudos
- 6501 Views
- 3 replies
- 1 kudos
- 5365 Views
- 7 replies
- 5 kudos
DBR 12.2: DeltaOptimizedWriter: Resolved attribute(s) missing from in operator
After upgrading from DBR 11.3 LTS to DBR 12.2 LTS we started to observe the following error during "read from parquet and write to delta" piece of logic.AnalysisException: Resolved attribute(s) group_id#72,display_name#73,parent_id#74,path#75,path_li...
- 5365 Views
- 7 replies
- 5 kudos
- 5 kudos
I can confirm that this issue is resolved for us as well in the latest 12.2 release.
- 5 kudos
- 1807 Views
- 2 replies
- 1 kudos
Reading data in Azure Databricks Delta Lake from AWS Redshift
We have Databricks set up and running on Azure. Now we want to connect it with Redshift (AWS) to perform further downstream analysis for our redshift users.I could find the documentation on how to do it within the same cloud (Either AWS or Azure) but...
- 1807 Views
- 2 replies
- 1 kudos
- 1 kudos
@Manny Cato​ :To allow Redshift to read data from Delta Lake hosted on Azure, you can use AWS Glue Data Catalog as an intermediary. The Glue Data Catalog is a fully managed metadata catalog that integrates with a variety of data sources, including De...
- 1 kudos