Hi everyone,I am working on a project that requires highly performant pipelines for managing data ingestion, validation, and processing large data volumes from IOT devices.I am interested in knowing:- The best way to ingest from EventHub/Kafka sinks-...
Hi Team, What I experienced today from the proctor was so not nice, my experience with proctor today was very frustrating and pathetic, I was taking my assessment today 11-26-2023, 2pm, I was already on between question 22nd to 25th when my assessmen...
@osalawu Sorry to hear you had an issue with your exam. In order to protect your Webassessor account information, please file a ticket with our support team. Please include your Webassessor login ID, the exam, and a couple of dates and times that wil...
Hi,I'm not sure if this is a possible scenario, but is there, by any chance a way to query all the columns of a table for searching a value? Explanation: I want to search for a specific value in all the columns of a databricks table. I don't know whi...
On Databricks created a job task with task type as Python script from s3. However, when arguments are passed via Parameters option, running into unrecognized arguments' error.Code in s3 file:import argparse def parse_arguments(): parser = argpar...
In my team, we decided to move from spark streaming to structured streaming, mainly cause it says that it's legacy and we want to benefit new features from structured streaming.However we have an issue with committing offsets.Previously on spark stre...
HelloI have issue with overwriting schema while using writestream - I do not receive any error - however schema remain unchangedBelow exampledf_abc = spark.readstream .format("cloudFiles") .option("cloudFiles.format", "parquet") .option"cloudF...
ERROR RetryingHMSHandler: NoSuchObjectException(message:There is no database named global_temp)at org.apache.hadoop.hive.metastore.ObjectStore.getMDatabase(ObjectStore.java:508)at org.apache.hadoop.hive.metastore.ObjectStore.getDatabase(ObjectStore.j...
I have 26 columns and 18k rows which I want to transpose and make 18k columns and 26 rows.Don't want any data summation of aggregation just transpose as it is. Can anyone pls suggest.P.S Not via converting to pandas because of large data
Hi,I may be missing something really obvious here. The organisation I work for has started using Delta Live Tables in Databricks for data modelling, recently. One of the dimensions I am trying to model takes data from 3 existing tables in our data la...
Can it be because the default join is `inner` and that means the row must exists in both tables
Hi all,For training purposes, I have cloned a repo from John Snow Labs into my Databricks account and am working in the notebook that you can review at https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/open-source-nlp/03.0.SparkNLP_Pretr...
Try changing the magic command for that cell from %md to %md-sandbox to see if that helps the image to render appropriately.
I noticed that unlike "Alter Table" there is no "Alter View" command to add comment on a column in the existing view. This is a regular view created on Tables (and not Materialized view). If the underlying table column has comment then the View inh...
I noticed that unlike "Alter Table" there is no "Alter View" command to add comment on a column in the existing view. This is a regular view created on Tables (and not Materialized view). If the underlying table column has comment then the View inh...
Is there a way for me to get Type 2 SCD changes without using streaming tables?I'm worried streaming tables may have limitations that interfere with adoption.
Hi,what mlflow version is used for the "Databricks Managed MLflow"Is there a workarround for MLflow CVE-2023-6015 (vulnerabilities) ?RegardsTimo
Hello,I have a single column dataframe and I want to transform the content into a stringEG df=abcdefxyzToabc, def, xyz Thanks
Hi Expert,How we can setup multiple notebook in a sequence order in flow for an example 1 pipeline have notebook1 - sequence 1,Notebook2- Sequence 2(in 1pipeline only)
Not sure how to approach your challenge but something you can is to use the Databricks Job Scheduler or if you want an external solution in Azure you can call several notebooks from DataFactory.
