- 2055 Views
- 3 replies
- 2 kudos
 Hello Everyone, I am thrilled to announce that we have our first winner for the raffle contest - @Vincent Slegers​. Please join me in congratulating ...
Hello Everyone,I am thrilled to announce that we have our first winner for the raffle contest - @Vincent Slegers​. Please join me in congratulating him on this remarkable achievement!Vincent, your dedication and hard work have paid off, and we are d...
- 2055 Views
- 3 replies
- 2 kudos
- 3353 Views
- 2 replies
- 0 kudos
Issue while using community edition
Hello,Is anyone facing issue with their community edition.?Shows the below error and cant access the workspace or previosly created notebooks..Tried accessing from different devices..(not a cache issue)Error popup:(screenshots attached)Unable to view...
- 3353 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Hari N​ Thank you for reaching out, and we’re sorry to hear about this log-in issue! We have this Community Edition login troubleshooting post on Community. Please take a look, and follow the troubleshooting steps. If the steps do not resolve the...
- 0 kudos
- 3213 Views
- 2 replies
- 0 kudos
Pyspark Structured Streaming Avro integration to Azure Schema Registry with Kafka/Eventhub in Databricks environment.
I am looking for a simple way to have a structured streaming pipeline that would automatically register a schema to Azure schema registry when converting a df col into avro and that would be able to deserialize an avro col based on schema registry ur...
- 3213 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Tomas Sedlon​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
- 0 kudos
- 2820 Views
- 3 replies
- 2 kudos
www.dbdemos.ai
Getting started with Databricks is being made very easy now. Presenting dbdemos.If you're looking to get started with Databricks, there's good news: dbdemos makes it easier than ever. This platform offers a range of demos that you can install directl...
- 2820 Views
- 3 replies
- 2 kudos
- 2 kudos
That's a great share Suteja. Is that supposed to work with the Databricks Community edition account? Had a strange error while trying. Any help is appreciated!Thanks,F
- 2 kudos
- 3999 Views
- 5 replies
- 1 kudos
Does thrift only exist in databrick control plane?
Hi all, I'm a user of Azure databricks. We recently found there is a thrift vulnerability issue (CVE-2020-13949) in Spark Hive. We have tried to fix it at our side. We also found there is a open issue at Spark jira board - https://issues.apache.org/j...
- 3999 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi @Jimin Hsieh​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so w...
- 1 kudos
- 8308 Views
- 11 replies
- 0 kudos
Process batches in a streaming pipeline - identifying deletes
OK. So I think I'm probably missing the obvious and tying myself in knots here.Here is the scenario:batch datasets arrive in json format in an Azure data lakeeach batch is a complete set of "current" records (the complete table)these are processed us...
- 8308 Views
- 11 replies
- 0 kudos
- 0 kudos
Hi @Kearon McNicol​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...
- 0 kudos
- 16308 Views
- 19 replies
- 1 kudos
Can not access SQL files in the Shared workspace
Hey,we have an issue in that we can access the SQL files whenever the notebook is in the repo path but whenever the CICD pipeline imports the repo notebooks and SQL files to the shared workspace, we can list the SQL files but can not read them.we cha...
- 16308 Views
- 19 replies
- 1 kudos
- 1 kudos
@Nermin Yehia​ yes, as you are moving files to different location manually , just update as can manage permissions in target and that should take care of everything
- 1 kudos
- 2728 Views
- 3 replies
- 0 kudos
Resolved! Delta Live Table Service Upgrade
Dear experts, Might I know what will happen to the delta live table pipeline which is in a cancelled state, when there is a runtime service upgrade? Thanks!
- 2728 Views
- 3 replies
- 0 kudos
- 0 kudos
@KS LAU​ :When a runtime service upgrade occurs in Databricks, any running tasks or pipelines may be temporarily interrupted while the upgrade is being applied. In the case of a cancelled Delta Live Table pipeline, it will not be impacted by the upgr...
- 0 kudos
- 5768 Views
- 4 replies
- 2 kudos
Resolved! DLT target schema - get value during run time
Hi,I would like to know if it is possible to get the target schema, programmatically, inside a DLT.In DLT pipeline settings, destination, target schema.I want to run more idempotent pipelines. For example, my target table has the fields: reference_da...
- 5768 Views
- 4 replies
- 2 kudos
- 2 kudos
Thank you @Suteja Kanuri​ ,Looks like you solution is working, thank you.Regards,
- 2 kudos
- 3582 Views
- 1 replies
- 1 kudos
Resolved! sedona/shapely error Unknown WKB type 16
Hi,i stream data from postgis to s3 using debezium. postgis->debezium->s3->spark(databricks)once read it i decode it and i can see that the binary representation is similiar to what i have in postgis, on a wkb formated column.once i try to read it ei...
- 3582 Views
- 1 replies
- 1 kudos
- 6736 Views
- 7 replies
- 5 kudos
DBR 12.2: DeltaOptimizedWriter: Resolved attribute(s) missing from in operator
After upgrading from DBR 11.3 LTS to DBR 12.2 LTS we started to observe the following error during "read from parquet and write to delta" piece of logic.AnalysisException: Resolved attribute(s) group_id#72,display_name#73,parent_id#74,path#75,path_li...
- 6736 Views
- 7 replies
- 5 kudos
- 5 kudos
I can confirm that this issue is resolved for us as well in the latest 12.2 release.
- 5 kudos
- 2462 Views
- 2 replies
- 1 kudos
Reading data in Azure Databricks Delta Lake from AWS Redshift
We have Databricks set up and running on Azure. Now we want to connect it with Redshift (AWS) to perform further downstream analysis for our redshift users.I could find the documentation on how to do it within the same cloud (Either AWS or Azure) but...
- 2462 Views
- 2 replies
- 1 kudos
- 1 kudos
@Manny Cato​ :To allow Redshift to read data from Delta Lake hosted on Azure, you can use AWS Glue Data Catalog as an intermediary. The Glue Data Catalog is a fully managed metadata catalog that integrates with a variety of data sources, including De...
- 1 kudos
- 1981 Views
- 2 replies
- 0 kudos
Securing the Account Owner
Hey,As I understand, you cannot enable SSO and MFA for the Account Owner.Is there any way on the Databricks side to secure the Account Owner beyond username/password? Is there a lockout that is set up automatically for this user?What are the best pra...
- 1981 Views
- 2 replies
- 0 kudos
- 0 kudos
@Domonkos Rozsa​ :You are correct that Databricks does not support SSO and MFA for the Account Owner. However, there are several built-in mechanisms that can help secure the Account Owner account and protect it from unauthorized access:Password polic...
- 0 kudos
- 4818 Views
- 1 replies
- 0 kudos
Resolved! what mode is the deploy-mode when calling spark in databricks/
https://spark.apache.org/docs/latest/submitting-applications.htmlmainly want to know if extra class path could be used or not when i submit a job
- 4818 Views
- 1 replies
- 0 kudos
- 0 kudos
@min shi​ :In Databricks, when you run a job, you are submitting a Spark application to run in the cluster. The deploy-mode that is used by default depends on the type of job you are running:For interactive clusters, the deploy-mode is client. This m...
- 0 kudos
- 1727 Views
- 2 replies
- 8 kudos
Databricks has added new metrics to its control panel, replacing the outdated Ganglia tool. These new metrics allow users to monitor the following clu...
Databricks has added new metrics to its control panel, replacing the outdated Ganglia tool. These new metrics allow users to monitor the following cluster performance metrics easily:- CPU utilization- Memory usage- Free filesystem space- Network traf...
- 1727 Views
- 2 replies
- 8 kudos
| User | Count |
|---|---|
| 1613 | |
| 771 | |
| 349 | |
| 286 | |
| 253 |
