- 997 Views
- 2 replies
- 0 kudos
Got Suspended while taking Databricks Certified Data Analyst Associate Assessment
Hi Team, What I experienced today from the proctor was so not nice, my experience with proctor today was very frustrating and pathetic, I was taking my assessment today 11-26-2023, 2pm, I was already on between question 22nd to 25th when my assessmen...
- 997 Views
- 2 replies
- 0 kudos
- 0 kudos
@osalawu Sorry to hear you had an issue with your exam. In order to protect your Webassessor account information, please file a ticket with our support team. Please include your Webassessor login ID, the exam, and a couple of dates and times that wil...
- 0 kudos
- 1193 Views
- 0 replies
- 0 kudos
Arguments parsing in Databricks python jobs
On Databricks created a job task with task type as Python script from s3. However, when arguments are passed via Parameters option, running into unrecognized arguments' error.Code in s3 file:import argparse def parse_arguments(): parser = argpar...
- 1193 Views
- 0 replies
- 0 kudos
- 1002 Views
- 0 replies
- 0 kudos
Structured Streaming - Kafka Offset Management
In my team, we decided to move from spark streaming to structured streaming, mainly cause it says that it's legacy and we want to benefit new features from structured streaming.However we have an issue with committing offsets.Previously on spark stre...
- 1002 Views
- 0 replies
- 0 kudos
- 8774 Views
- 5 replies
- 0 kudos
Resolved! dataset.cache() not working : NoSuchObjectException(message:There is no database named global_temp)
ERROR RetryingHMSHandler: NoSuchObjectException(message:There is no database named global_temp)at org.apache.hadoop.hive.metastore.ObjectStore.getMDatabase(ObjectStore.java:508)at org.apache.hadoop.hive.metastore.ObjectStore.getDatabase(ObjectStore.j...
- 8774 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @RahulPatidar , The error message you’re encountering, “NoSuchObjectException (message: There is no database named global_temp),” is related to the use of the special database called “global_temp” in Spark.Here’s what you need to know: Global Tem...
- 0 kudos
- 790 Views
- 0 replies
- 0 kudos
I want to perform transpose in my dataframe using pyspark
I have 26 columns and 18k rows which I want to transpose and make 18k columns and 26 rows.Don't want any data summation of aggregation just transpose as it is. Can anyone pls suggest.P.S Not via converting to pandas because of large data
- 790 Views
- 0 replies
- 0 kudos
- 1098 Views
- 1 replies
- 0 kudos
Delta Live Tables Slowly Changing Dimensions Type 2 with Joins
Hi,I may be missing something really obvious here. The organisation I work for has started using Delta Live Tables in Databricks for data modelling, recently. One of the dimensions I am trying to model takes data from 3 existing tables in our data la...
- 1098 Views
- 1 replies
- 0 kudos
- 0 kudos
Can it be because the default join is `inner` and that means the row must exists in both tables
- 0 kudos
- 2673 Views
- 2 replies
- 1 kudos
Resolved! Rendering markdown images hard coded as data image png base64 in notebook
Hi all,For training purposes, I have cloned a repo from John Snow Labs into my Databricks account and am working in the notebook that you can review at https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/open-source-nlp/03.0.SparkNLP_Pretr...
- 2673 Views
- 2 replies
- 1 kudos
- 1 kudos
Try changing the magic command for that cell from %md to %md-sandbox to see if that helps the image to render appropriately.
- 1 kudos
- 5841 Views
- 2 replies
- 0 kudos
Resolved! Unable to add column comment on a View. Any way to update comments on multiple columns in bulk?
I noticed that unlike "Alter Table" there is no "Alter View" command to add comment on a column in the existing view. This is a regular view created on Tables (and not Materialized view). If the underlying table column has comment then the View inh...
- 5841 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @prasad_vaze, Let’s address your questions regarding altering views and mass-applying column comments: Adding Comments to View Columns: While there is no direct “ALTER VIEW” command to add comments to specific columns in an existing view, you ca...
- 0 kudos
- 548 Views
- 0 replies
- 0 kudos
Type 2 - Slowly Changing Dimensions (SCD) without Streaming Tables?
Is there a way for me to get Type 2 SCD changes without using streaming tables?I'm worried streaming tables may have limitations that interfere with adoption.
- 548 Views
- 0 replies
- 0 kudos
- 642 Views
- 0 replies
- 0 kudos
MLflow CVE-2023-6015 (vulnerabilities)
Hi,what mlflow version is used for the "Databricks Managed MLflow"Is there a workarround for MLflow CVE-2023-6015 (vulnerabilities) ?RegardsTimo
- 642 Views
- 0 replies
- 0 kudos
- 5205 Views
- 1 replies
- 0 kudos
Resolved! how to use R in databricks
Hello everyone.I am a new user of databricks, they implemented it in the company where I work. I am a business analyst and I know something about R, not much either, when I saw that databricks could use R I was very excited because I thought that the...
- 5205 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @JCamiloCS , Certainly! I apologize for the oversight. Let’s dive into using R in Databricks. Databricks is a powerful platform for big data analytics and machine learning, and it integrates seamlessly with Apache Spark™. As a business analyst, l...
- 0 kudos
- 16946 Views
- 2 replies
- 1 kudos
Resolved! Some streams terminated before this command could finish! -> java.lang.NoClassDefFoundError: scala/c
HelloI do face:Some streams terminated before this command could finish!java.lang.NoClassDefFoundError: scala/compat/java8/FutureConverters$Running some very simple query on eventhub :df = spark \.readStream \.format("eventhubs") \.options(**ehConf) ...
- 16946 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @PiotrU , I want to express my gratitude for your effort in selecting the most suitable solution. It's great to hear that your query has been successfully resolved. Thank you for your contribution.
- 1 kudos
- 889 Views
- 0 replies
- 0 kudos
transform a dataframe column as concatenated string
Hello,I have a single column dataframe and I want to transform the content into a stringEG df=abcdefxyzToabc, def, xyz Thanks
- 889 Views
- 0 replies
- 0 kudos
- 1494 Views
- 1 replies
- 0 kudos
org.apache.spark.SparkException - FileReadException
Sometimes getting this kind of error "org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 12224.0 failed 4 times, most recent failure: Lost task 1.5 in stage 12224.0 (TID ) (12.xxx.x.xxx executor 1): com.datab...
- 1494 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Madhawa, One possible reason for this error is that the file you are trying to read is corrupted or incomplete. You can try to verify the integrity of the file by checking its hash value or by comparing it with a known good copy of the file. Ano...
- 0 kudos
- 3981 Views
- 2 replies
- 0 kudos
Delta table definition - Identity column
Hello,Would anyone know if it is possible to create a delta table using Python that includes a column that is generated by default as identity (identity column for which the value inserted can be manually overriden)?There seems to be a way to create ...
- 3981 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @oleprince, Yes, it is possible to create a Delta table using Python that includes a column that is generated by default as identity. However, the generatedByDefaultAs option is not available in the DeltaTableBuilder API. As a workaround, you ca...
- 0 kudos
