- 4917 Views
- 2 replies
- 3 kudos
Resolved! Error creating delta table over an existing delta schema
I created a delta table through a cluster over a dbfs location .Schema :create external table tmp_db.delta_data(delta_id int ,delta_name varchar(20) , delta_variation decimal(10,4) ,delta_incoming_timestamp timestamp,delta_date date generated always ...
- 4917 Views
- 2 replies
- 3 kudos
- 3 kudos
varchartype is only available as from spark 3.1 I think.https://spark.apache.org/docs/latest/sql-ref-datatypes.htmlThe link is for spark 3.2, and 3.1 also has varchartype. So can you check your spark version?Also if the table definition still exists...
- 3 kudos
- 4322 Views
- 4 replies
- 0 kudos
Queries regarding workspace Migration to Premium
We are planning to migrate from standard to premium workspaceWe need to know if below artifacts will be maintainedneed to check on streaming Job DowntimeAccess token DBFS Access Production Cluster /JobsCluster ID Job ID and other properties like URL ...
- 4322 Views
- 4 replies
- 0 kudos
- 0 kudos
hi @Kaniz Fatma​ then I can assume there wont be any impact on metastore and all the metadata(table definition,schema ) will be available post upgradation
- 0 kudos
- 2464 Views
- 1 replies
- 0 kudos
Delta Live Table - How to get details of which records were excluded in Quality Checks?
I've been experimenting with DLT and it works well. I'd like to understand where can I see details of which records didn't meet the quality critera?
- 2464 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello, @Paresh J​! Welcome and thank you for asking! My name is Piper, and I'm a moderator for Databricks.Let's give the community some time to help before we circle back to you. Thanks in advance for your patience.
- 0 kudos
- 890 Views
- 0 replies
- 0 kudos
res.cloudinary.com
Link UG300 ada menyediakan depo tipe e- wallet yang dapat dipakai unyuk dapat melaksanakan pembelian ataupun top up saldo ke e- wallet tujuan yang telah ada di dalam web. Adanya berita gembira buat kalian yang tidak mempunyai rekening bank, Jika kali...
- 890 Views
- 0 replies
- 0 kudos
- 3740 Views
- 1 replies
- 0 kudos
- 3740 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello, @Ravi Param​ - My name is Piper, and I'm one of the moderators here. Thank you for your question! Let's give the community a chance to respond and then we'll circle back if necessary.
- 0 kudos
- 18517 Views
- 4 replies
- 0 kudos
Resolved! Command prompt won't let me type the Databricks token
Hi, I am trying to set up Databricks CLI using the command prompt on my computer. I downloaded the Python 3.9 app and successfully ran the command pip install databricks-cliWhen I try to set up the Databricks token, I am able to type my Databricks Ho...
- 18517 Views
- 4 replies
- 0 kudos
- 38920 Views
- 5 replies
- 4 kudos
Resolved! Export Databricks results to Blob in a csv file
Hello everyone,I want to export my data from Databricks to the blob. My Databricks commands select some pdf from my blob, run Form Recognizer and export the output results in my blob. Here is the code: %pip install azure.storage.blob %pip install...
- 38920 Views
- 5 replies
- 4 kudos
- 6946 Views
- 2 replies
- 0 kudos
Resolved! how to Intercept Spark Listener with Pyspark ?
hi guys,​It`s possible to intercept Spark Listener with Pyspark to collect indicator like shuffle, skew ratio, etc ?
- 6946 Views
- 2 replies
- 0 kudos
- 0 kudos
interesting question.I know that you can use the SparkListener to collect info, f.e. here.Mind that the class is written in Scala, so my first thought was that it is not possible in python/pyspark.But SO says it is possible, but with a lot of overhea...
- 0 kudos
- 4004 Views
- 2 replies
- 4 kudos
Resolved! Converting dataframe to delta.
Is it possible to convert the dataframe to a delta table without saving the dataframe on the storage?
- 4004 Views
- 2 replies
- 4 kudos
- 20493 Views
- 6 replies
- 5 kudos
Resolved! ModuleNotFoundError: No module named 'mlflow' when running a notebook
I am running a notebook on the Coursera platform.my configuration file, Classroom-Setup, looks like this:%python spark.conf.set("com.databricks.training.module-name", "deep-learning") spark.conf.set("com.databricks.training.expected-dbr", "6.4") ...
- 20493 Views
- 6 replies
- 5 kudos
- 5 kudos
Hi @Maria Bruevich​ ,From the error description, it looks like the mlflow library is not present. You can use ML cluster as these type of cluster already have mlflow library. Please check the below document:https://docs.databricks.com/release-notes/r...
- 5 kudos
- 10067 Views
- 3 replies
- 0 kudos
Resolved! Help trying to calculate a percentage
The image below shows what my source data is (HAVE) and what I'm trying to get to (WANT).I want to be able to calculate the percentage of bad messages (where formattedMessage = false) by source and date.I'm not sure how to achieve this in DatabricksS...
- 10067 Views
- 3 replies
- 0 kudos
- 0 kudos
you could use a windows function over source and date with a sum of messagecount. This gives you the total per source/date repeated on every line.Then apply a filter on formattedmessage == false and divide messagecount by the sum above.
- 0 kudos
- 7858 Views
- 0 replies
- 1 kudos
Access data within the blob storage without downloading
Our customer is using Azure’s blob storage service to save big files so that we can work with them using an Azure online service, like Databricks.We want to read and work with these files with a computing resource obtained by Azure directly without d...
- 7858 Views
- 0 replies
- 1 kudos
- 992 Views
- 0 replies
- 0 kudos
data=[['x', 20220118, 'FALSE', 3],['x', 20220118, 'TRUE', 97],['x', 20220119, 'FALSE', 1],['x'...
data=[['x', 20220118, 'FALSE', 3],['x', 20220118, 'TRUE', 97],['x', 20220119, 'FALSE', 1],['x', 20220119, 'TRUE', 49],['Y', 20220118, 'FALSE', 100],['Y', 20220118, 'TRUE', 900],['Y', 20220119, 'FALSE', 200],['Y', 20220119, 'TRUE', 800]]df=spark.creat...
- 992 Views
- 0 replies
- 0 kudos
- 3456 Views
- 3 replies
- 2 kudos
Resolved! Query RestAPI end point in Databricks Standard Workspace
Do we have option to query delta table using Standard Workspace as a endpoint instead of JDBC
- 3456 Views
- 3 replies
- 2 kudos
- 2 kudos
@somanath Sankaran​ - Would you be happy to mark @Hubert Dudek​'s answer as best if it resolved the problem? That helps other members who are searching for answers find the solution more quickly.
- 2 kudos
- 6434 Views
- 4 replies
- 4 kudos
Resolved! Schema Parsing issue when datatype of source field is mapped incorrect
I have complex json file which has massive struct column. We regularly have issues when we try to parse this json file by forming our case class to extract the fields from schema. With this approach the issue we are facing is that if one data type of...
- 6434 Views
- 4 replies
- 4 kudos
- 4 kudos
Hey there, @Matt M​ - If @Hubert Dudek​'s response solved the issue, would you be happy to mark his answer as best? It helps other members find the solution more quickly.
- 4 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 576 | |
| 349 | |
| 287 |
