I have 26 columns and 18k rows which I want to transpose and make 18k columns and 26 rows.Don't want any data summation of aggregation just transpose as it is. Can anyone pls suggest.P.S Not via converting to pandas because of large data
Hi,I may be missing something really obvious here. The organisation I work for has started using Delta Live Tables in Databricks for data modelling, recently. One of the dimensions I am trying to model takes data from 3 existing tables in our data la...
Can it be because the default join is `inner` and that means the row must exists in both tables
Hi all,For training purposes, I have cloned a repo from John Snow Labs into my Databricks account and am working in the notebook that you can review at https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/open-source-nlp/03.0.SparkNLP_Pretr...
Try changing the magic command for that cell from %md to %md-sandbox to see if that helps the image to render appropriately.
I noticed that unlike "Alter Table" there is no "Alter View" command to add comment on a column in the existing view. This is a regular view created on Tables (and not Materialized view). If the underlying table column has comment then the View inh...
Hi @prasad_vaze, Let’s address your questions regarding altering views and mass-applying column comments: Adding Comments to View Columns: While there is no direct “ALTER VIEW” command to add comments to specific columns in an existing view, you ca...
Is there a way for me to get Type 2 SCD changes without using streaming tables?I'm worried streaming tables may have limitations that interfere with adoption.
Hi,what mlflow version is used for the "Databricks Managed MLflow"Is there a workarround for MLflow CVE-2023-6015 (vulnerabilities) ?RegardsTimo
Hello everyone.I am a new user of databricks, they implemented it in the company where I work. I am a business analyst and I know something about R, not much either, when I saw that databricks could use R I was very excited because I thought that the...
Hi @JCamiloCS , Certainly! I apologize for the oversight. Let’s dive into using R in Databricks. Databricks is a powerful platform for big data analytics and machine learning, and it integrates seamlessly with Apache Spark™. As a business analyst, l...
HelloI do face:Some streams terminated before this command could finish!java.lang.NoClassDefFoundError: scala/compat/java8/FutureConverters$Running some very simple query on eventhub :df = spark \.readStream \.format("eventhubs") \.options(**ehConf) ...
Hi @PiotrU , I want to express my gratitude for your effort in selecting the most suitable solution. It's great to hear that your query has been successfully resolved. Thank you for your contribution.
Hello,I have a single column dataframe and I want to transform the content into a stringEG df=abcdefxyzToabc, def, xyz Thanks
Sometimes getting this kind of error "org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 12224.0 failed 4 times, most recent failure: Lost task 1.5 in stage 12224.0 (TID ) (12.xxx.x.xxx executor 1): com.datab...
Hi @Madhawa, One possible reason for this error is that the file you are trying to read is corrupted or incomplete. You can try to verify the integrity of the file by checking its hash value or by comparing it with a known good copy of the file. Ano...
Hello,Would anyone know if it is possible to create a delta table using Python that includes a column that is generated by default as identity (identity column for which the value inserted can be manually overriden)?There seems to be a way to create ...
Hi @oleprince, Yes, it is possible to create a Delta table using Python that includes a column that is generated by default as identity. However, the generatedByDefaultAs option is not available in the DeltaTableBuilder API. As a workaround, you ca...
Hi Expert,How we can setup multiple notebook in a sequence order in flow for an example 1 pipeline have notebook1 - sequence 1,Notebook2- Sequence 2(in 1pipeline only)
Not sure how to approach your challenge but something you can is to use the Databricks Job Scheduler or if you want an external solution in Azure you can call several notebooks from DataFactory.
Hi,I'm trying to call the DLT api to kickoff my delta live table flow with a web API call block from Azure Data Factory. I have two environments: one DEV and one PROD.The DEV environment works fine, the response is giving me the update_id, but the PR...
Hi. Recently, I've found that Databricks is really slow when editing notebooks, such as adding cells, copying and pasting text or cells, etc. It's just the past few weeks actually. I'm using Chrome version 118.0.5993.118. Everything else with Chrome ...
It seems related to the notebook length (number of cells). The notebook that was really slow had about 40-50 cells, which I've done before without issue. Anyway after starting a new notebook using Chrome, it seems useable again. So without a specific...
I have installed graphframes library from maven repository in the cluster (13.3 LTS (includes Apache Spark 3.4.1, Scala 2.12), Standard DS4_v2). The library that I have installed is graphframes:graphframes:0.8.3-spark3.5-s_2.13.I can import the graph...
