- 3811 Views
- 5 replies
- 5 kudos
Not able to install geomesa on my Databricks cluster
Hi team, I have attempting to install Geomesa (2.12:3.4.1) library on my cluster but it keeps failing with the below error:Library installation attempted on the driver node of cluster 0824-052900-76icyj32 and failed. Please refer to the following err...
- 3811 Views
- 5 replies
- 5 kudos
- 5 kudos
Hi @Ayushi Pandey​ I could see the package is available in the maven repo. https://mvnrepository.com/artifact/org.locationtech.geomesa/geomesa_2.12/3.4.1Have you tried downloading the package to dbfs location and installed on the cluster?
- 5 kudos
- 2176 Views
- 1 replies
- 2 kudos
Why am I getting a FileNotFoundError after providing the file path?
I used copy file path to get the file path of the notebook I am trying to run from another notebook.file_path = "/Users/ankur.lohiya@workday.com/PAS/Training/Ingest/TrainingQueries-Cloned.py/"ddi = DatabricksDataIngestion(file_path=file_path, ...
- 2176 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello @Ankur Lohiya​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...
- 2 kudos
- 4542 Views
- 2 replies
- 1 kudos
Driver context not found for python spark for spark_submit_task using Jobs API submit run endpoint
I am trying to run a multi file python job in databricks without using notebooks. I have tried setting this up by:creating a docker image using the DBRT 10.4 LTS as a base and adding the zipped python application to that.make a call to the run submit...
- 4542 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Umar Khan​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 1 kudos
- 4531 Views
- 2 replies
- 1 kudos
Resolved! What are the options to offer a low latency API for small tables derived from big tables?
I have a big dataset which gets divided into smaller datasets. For some of these smaller datasets I'd like to offer a low latency API (*** ms) to query them. Big dataset 1B entriesSmaller dataset 1 Mio entriesWhat's the best way to do it?I thought ab...
- 4531 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Jan R​ Does @Tian Tan​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
- 1 kudos
- 3895 Views
- 2 replies
- 0 kudos
Resolved! Writing transformed DataFrame to a persistent table is unbearable slow
I want to transform a DF with a simple UDF. Afterwards I want to store the resulting DF in a new table (see code below)key = "test_key" schema = StructType([ StructField("***", StringType(), True), StructField("yyy", StringType(), True), StructF...
- 3895 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @Jan R​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 0 kudos
- 1802 Views
- 2 replies
- 1 kudos
I need help finding the right mode for my course
How do I find the Data Brick Community edition?
- 1802 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Kester Truman​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...
- 1 kudos
- 4193 Views
- 2 replies
- 2 kudos
Create dropdown-list in Markdown
In the first cell of my notebooks, I record a changelog for all changes done in the notebook in Markdown. However, as this list becomes longer and longer, I want to implement a dropdown list. Is there anyway to do this in Markdown in databricks?For t...
- 4193 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Jesse vd S​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 2 kudos
- 1588 Views
- 1 replies
- 0 kudos
A basic DataFrame transformation query
I want to know how dataframe transformations work.Suppose I have a DataFrame instance df1. I apply some operation on it, say a filter. As every operation gives a new dataframe, so lets say now we have df2. So we have two DataFrame instances now, df1 ...
- 1588 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @mghildiy​ Does @Kaniz Fatma​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
- 0 kudos
- 2838 Views
- 2 replies
- 2 kudos
Resolved! Where is Databricks Tunnel (and is Databricks connect cool again?)
Two related questions:1: There has been several mentions in this forum about "Databricks Tunnel", which should allow us to connect from our local IDE to a remote databricks cluster and develop stuff locally. The roumors said early 2022, is there some...
- 2838 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi there @Erik Parmann​ Does @Youssef Mrini​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks
- 2 kudos
- 2253 Views
- 3 replies
- 1 kudos
Any plans to provide Databricks SQL / Alerts API
Hi, Databricks! You are my favorite Big Data tool, but I've recently faced an issue I didn't expect to have. For our agriculture customers, we're trying to use Databricks SQL Platform to keep our data accurate all day. We use Alerts to validate our d...
- 2253 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Dmytro Imshenetskyi​ Does @Hubert Dudek​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
- 1 kudos
- 1565 Views
- 1 replies
- 0 kudos
Cannot sign in at databricks partner-academy portal
Hi thereI have used my company email to register an account for customer-academy.databricks.com a while back.Now what I need to do is create an account with partner-academy.databricks.com using my company email too.However when I register at partner-...
- 1565 Views
- 1 replies
- 0 kudos
- 4507 Views
- 2 replies
- 1 kudos
Resolved! Error deleting a table
I'm trying to delete a table that was created from a csv and due to the file deletion, I can't execute the deletion, with the following error: I'm new to Databricks and I don't know how to fix this. Some help?
- 4507 Views
- 2 replies
- 1 kudos
- 1 kudos
To delete the table, it's looking for underlying delta log file and because the file doesn't exist, it's throwing you that error.Just drop the table.drop table <table_name>
- 1 kudos
- 3068 Views
- 4 replies
- 7 kudos
Resolved! Azure data bricks delta tables .Issue
Hello Team,I have written Spark SQL Query in data bricks :DROP TABLE IF EXISTS Salesforce.Location;CREATE EXTERNAL TABLE Salesforce.Location (Id STRING,OwnerId STRING,IsDeleted bigint,Name STRING,CurrencyIsoCode STRING,CreatedDate bigint,CreatedById ...
- 3068 Views
- 4 replies
- 7 kudos
- 7 kudos
You need to provide one of the following value for 'data_source':TEXTAVROCSVJSONPARQUETORCDELTAeg: USING PARQUETIf you skip USING clause, then the default data source is DELTAhttps://docs.databricks.com/sql/language-manual/sql-ref-syntax-ddl-create-t...
- 7 kudos
- 8709 Views
- 6 replies
- 4 kudos
Resolved! Lesson 6.1 of Data Engineering. Error when reading stream - java.lang.UnsupportedOperationException: com.databricks.backend.daemon.data.client.DBFSV1.resolvePathOnPhysicalStorage(path: Path)
Below function executes fine: def autoload_to_table(data_source, source_format, table_name, checkpoint_directory): query = (spark.readStream .format("cloudFiles") .option("cloudFiles.format", source_format) .option("cloudFile...
- 8709 Views
- 6 replies
- 4 kudos
- 3445 Views
- 2 replies
- 2 kudos
Resolved! Delta Live Tables not inferring table schema properly.
I have a delta live tables pipeline that is loading and transforming data. Currently I am having a problem that the schema inferred by DLT does not match the actual schema of the table. The table is generated via a groupby.pivot operation as follows:...
- 3445 Views
- 2 replies
- 2 kudos
- 2 kudos
I was able to get around this by specifying the table schema in the table decorator.
- 2 kudos
| User | Count |
|---|---|
| 1630 | |
| 790 | |
| 512 | |
| 349 | |
| 287 |
