- 10144 Views
- 2 replies
- 4 kudos
Resolved! ProtocolChangedException on concurrent blind appends to delta table
Hello, I am developing an application that runs multiple processes that write their results to a common delta table as blind appends. According to the docs I've read online: https://docs.databricks.com/delta/concurrency-control.html#protocolchangedex...
- 10144 Views
- 2 replies
- 4 kudos
- 4 kudos
I think you are right, the mergeSchema will change the schema of the table, but if you both write to that same table with another schema, which one will it be?Can you check if both of you actually write the same schema, or remove the mergeschema?
- 4 kudos
- 4520 Views
- 2 replies
- 3 kudos
Resolved! Error creating delta table over an existing delta schema
I created a delta table through a cluster over a dbfs location .Schema :create external table tmp_db.delta_data(delta_id int ,delta_name varchar(20) , delta_variation decimal(10,4) ,delta_incoming_timestamp timestamp,delta_date date generated always ...
- 4520 Views
- 2 replies
- 3 kudos
- 3 kudos
varchartype is only available as from spark 3.1 I think.https://spark.apache.org/docs/latest/sql-ref-datatypes.htmlThe link is for spark 3.2, and 3.1 also has varchartype. So can you check your spark version?Also if the table definition still exists...
- 3 kudos
- 4049 Views
- 4 replies
- 0 kudos
Queries regarding workspace Migration to Premium
We are planning to migrate from standard to premium workspaceWe need to know if below artifacts will be maintainedneed to check on streaming Job DowntimeAccess token DBFS Access Production Cluster /JobsCluster ID Job ID and other properties like URL ...
- 4049 Views
- 4 replies
- 0 kudos
- 0 kudos
hi @Kaniz Fatma​ then I can assume there wont be any impact on metastore and all the metadata(table definition,schema ) will be available post upgradation
- 0 kudos
- 2352 Views
- 1 replies
- 0 kudos
Delta Live Table - How to get details of which records were excluded in Quality Checks?
I've been experimenting with DLT and it works well. I'd like to understand where can I see details of which records didn't meet the quality critera?
- 2352 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello, @Paresh J​! Welcome and thank you for asking! My name is Piper, and I'm a moderator for Databricks.Let's give the community some time to help before we circle back to you. Thanks in advance for your patience.
- 0 kudos
- 779 Views
- 0 replies
- 0 kudos
res.cloudinary.com
Link UG300 ada menyediakan depo tipe e- wallet yang dapat dipakai unyuk dapat melaksanakan pembelian ataupun top up saldo ke e- wallet tujuan yang telah ada di dalam web. Adanya berita gembira buat kalian yang tidak mempunyai rekening bank, Jika kali...
- 779 Views
- 0 replies
- 0 kudos
- 3620 Views
- 1 replies
- 0 kudos
- 3620 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello, @Ravi Param​ - My name is Piper, and I'm one of the moderators here. Thank you for your question! Let's give the community a chance to respond and then we'll circle back if necessary.
- 0 kudos
- 17784 Views
- 4 replies
- 0 kudos
Resolved! Command prompt won't let me type the Databricks token
Hi, I am trying to set up Databricks CLI using the command prompt on my computer. I downloaded the Python 3.9 app and successfully ran the command pip install databricks-cliWhen I try to set up the Databricks token, I am able to type my Databricks Ho...
- 17784 Views
- 4 replies
- 0 kudos
- 38352 Views
- 5 replies
- 4 kudos
Resolved! Export Databricks results to Blob in a csv file
Hello everyone,I want to export my data from Databricks to the blob. My Databricks commands select some pdf from my blob, run Form Recognizer and export the output results in my blob. Here is the code: %pip install azure.storage.blob %pip install...
- 38352 Views
- 5 replies
- 4 kudos
- 6628 Views
- 2 replies
- 0 kudos
Resolved! how to Intercept Spark Listener with Pyspark ?
hi guys,​It`s possible to intercept Spark Listener with Pyspark to collect indicator like shuffle, skew ratio, etc ?
- 6628 Views
- 2 replies
- 0 kudos
- 0 kudos
interesting question.I know that you can use the SparkListener to collect info, f.e. here.Mind that the class is written in Scala, so my first thought was that it is not possible in python/pyspark.But SO says it is possible, but with a lot of overhea...
- 0 kudos
- 3684 Views
- 2 replies
- 4 kudos
Resolved! Converting dataframe to delta.
Is it possible to convert the dataframe to a delta table without saving the dataframe on the storage?
- 3684 Views
- 2 replies
- 4 kudos
- 19547 Views
- 6 replies
- 5 kudos
Resolved! ModuleNotFoundError: No module named 'mlflow' when running a notebook
I am running a notebook on the Coursera platform.my configuration file, Classroom-Setup, looks like this:%python spark.conf.set("com.databricks.training.module-name", "deep-learning") spark.conf.set("com.databricks.training.expected-dbr", "6.4") ...
- 19547 Views
- 6 replies
- 5 kudos
- 5 kudos
Hi @Maria Bruevich​ ,From the error description, it looks like the mlflow library is not present. You can use ML cluster as these type of cluster already have mlflow library. Please check the below document:https://docs.databricks.com/release-notes/r...
- 5 kudos
- 9487 Views
- 3 replies
- 0 kudos
Resolved! Help trying to calculate a percentage
The image below shows what my source data is (HAVE) and what I'm trying to get to (WANT).I want to be able to calculate the percentage of bad messages (where formattedMessage = false) by source and date.I'm not sure how to achieve this in DatabricksS...
- 9487 Views
- 3 replies
- 0 kudos
- 0 kudos
you could use a windows function over source and date with a sum of messagecount. This gives you the total per source/date repeated on every line.Then apply a filter on formattedmessage == false and divide messagecount by the sum above.
- 0 kudos
- 7413 Views
- 0 replies
- 1 kudos
Access data within the blob storage without downloading
Our customer is using Azure’s blob storage service to save big files so that we can work with them using an Azure online service, like Databricks.We want to read and work with these files with a computing resource obtained by Azure directly without d...
- 7413 Views
- 0 replies
- 1 kudos
- 891 Views
- 0 replies
- 0 kudos
data=[['x', 20220118, 'FALSE', 3],['x', 20220118, 'TRUE', 97],['x', 20220119, 'FALSE', 1],['x'...
data=[['x', 20220118, 'FALSE', 3],['x', 20220118, 'TRUE', 97],['x', 20220119, 'FALSE', 1],['x', 20220119, 'TRUE', 49],['Y', 20220118, 'FALSE', 100],['Y', 20220118, 'TRUE', 900],['Y', 20220119, 'FALSE', 200],['Y', 20220119, 'TRUE', 800]]df=spark.creat...
- 891 Views
- 0 replies
- 0 kudos
- 3140 Views
- 3 replies
- 2 kudos
Resolved! Query RestAPI end point in Databricks Standard Workspace
Do we have option to query delta table using Standard Workspace as a endpoint instead of JDBC
- 3140 Views
- 3 replies
- 2 kudos
- 2 kudos
@somanath Sankaran​ - Would you be happy to mark @Hubert Dudek​'s answer as best if it resolved the problem? That helps other members who are searching for answers find the solution more quickly.
- 2 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
