- 7668 Views
- 2 replies
- 2 kudos
- 7668 Views
- 2 replies
- 2 kudos
- 2 kudos
Thanks @Kaniz Fatma​ Actually, I was looking for the same and then got this blog
- 2 kudos
- 6531 Views
- 3 replies
- 3 kudos
What gets overridden when writing overriding a delta lake table?
I just tried to write to a delta lake table using override mode, and I found that history is reserved. It's unclear to me how the data is overridden, and how long the history could be preserved. As they say, a code is better than a thousand words: my...
- 6531 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Rami ALZEBAK​ overwrite means first it will remove the data and again it will write the whole data.If you want to see history use can use DESCRIBE HISTORY command
- 3 kudos
- 2559 Views
- 1 replies
- 2 kudos
What are the options for extracting data from the delta lake for a vendor?
Our vendor is looking to use Microsoft API Manager to retrieve data from a variety of sources. Is it possible to extract records from the delta lake by using an API?What I've tried:I reviewed the available databricks API's it looks like most of them ...
- 2559 Views
- 1 replies
- 2 kudos
- 2 kudos
Another possibility for this potentially is to stand up a cluster and have a notebook running flask to create an API interface. I'm still looking into options, but it seems like there should be a baked in solution besides the JDBC connector. I'm not ...
- 2 kudos
- 3727 Views
- 0 replies
- 1 kudos
Spark always performing broad casts irrespective of spark.sql.autoBroadcastJoinThreshold during streaming merge operation with DeltaTable.
I am trying to do a streaming merge between delta tables using this guide - https://docs.delta.io/latest/delta-update.html#upsert-from-streaming-queries-using-foreachbatchOur Code Sample (Java): Dataset<Row> sourceDf = sparkSession ...
- 3727 Views
- 0 replies
- 1 kudos
- 7901 Views
- 4 replies
- 8 kudos
Is it possible to create a sqlite database and export it?
I am trying to create a sqlite database in databricks and add a few tables to it. Ultimately, I want to export this using Azure. Is this possible?
- 7901 Views
- 4 replies
- 8 kudos
- 8 kudos
@Hubert Dudek​ We currently have a process in place that reads in a SQLite file. We recently transitioned to using Databricks. We were hoping to be able to create a SQLite file so we didn't have to alter the current process we have in place.
- 8 kudos
- 5003 Views
- 3 replies
- 1 kudos
I have attended Data + AI World Tour Asia Pacific this week but did not received post confirmation email.
I have attended Data + AI World Tour Asia Pacific this week but did not received post confirmation email. After webinar I received short survey and then thank you note for participation. But unexpectedly did not received any email with feedback link ...
- 5003 Views
- 3 replies
- 1 kudos
- 5331 Views
- 1 replies
- 3 kudos
How to extract JSON object from a pyspark data frame. I was able to extract data from another column which in array format using "Explode" function, but Explode is not working for Object type. Its returning with type mismatch error.
I have tried below code to extract data which in Array:df2 = df_deidentifieddocuments_tst.select(F.explode('annotationId').alias('annotationId')).select('annotationId.$oid')It was working fine.. but,its not working for JSON object type. Below is colu...
- 5331 Views
- 1 replies
- 3 kudos
- 3 kudos
Did you try extracting that column data using from_json function ?
- 3 kudos
- 4091 Views
- 1 replies
- 3 kudos
pyspark code error
rdd4 = rdd3.reducByKey(lambda x,y: x+y)AttributeError: 'PipelinedRDD' object has no attribute 'reducByKey'Pls help me out with this
- 4091 Views
- 1 replies
- 3 kudos
- 3 kudos
Is it a typo or are you really using reducByKey instead of reduceByKey ?
- 3 kudos
- 5590 Views
- 4 replies
- 1 kudos
How do I "Earn 100 points to the Databricks Community Rewards Store" ? (As advertised on Databricks Academy)
Hello, how do I join the Databricks Community study group for 100points, as advertised on the Databricks Academy website?
- 5590 Views
- 4 replies
- 1 kudos
- 1 kudos
@Alex Serlovsky​ You need to earn the lakehouse fundamental credetial certification, then you can join this community group. Within 24 to 48 hours you will get 100 reward points. But As per databricks, you need to earn the credential on or before Nov...
- 1 kudos
- 2645 Views
- 0 replies
- 1 kudos
Natively Query Delta Lake with R
I have a large delta table that I need to analyze in native R. The only option I have currently is to query the delta table then use collect() to bring that spark dataframe into an R dataframe. Is there an alternative method that would allow me to qu...
- 2645 Views
- 0 replies
- 1 kudos
- 5651 Views
- 4 replies
- 4 kudos
Cannot CREATE TABLE with 'No Isolation Shared' cluster
Recently I ran into a number issues running with our notebooks in Interactive Mode. For example, we can't create (delta) table. The command would run and then idle for no apparent exception. The path is created on AWS S3 but delta log is never create...
- 5651 Views
- 4 replies
- 4 kudos
- 4 kudos
The Admin can disable the possibility to use the no Isolate Shared cluster. I recommend you to switch to Single user where UC is activated. Don't worry you won't need to change your code. If you encounter this kind of issues, make sure to open a tick...
- 4 kudos
- 27956 Views
- 7 replies
- 6 kudos
Resolved! How to programmatically download png files from matplotlib plots in notebook?
I am creating plots in databricks using python and matplotlib. These look great in notebook and I can save them to the dbfs usingplt.savefig("/dbfs/FileStore/tables/[plot_name].png")I can then download the png files to my computer individually by pas...
- 27956 Views
- 7 replies
- 6 kudos
- 6 kudos
Thanks everyone! I am already at a place where I can download a png to FileStore and use a url to download that file locally. What I was wondering was if there is some databricks function I can use to launch the url that references the png file and d...
- 6 kudos
- 4187 Views
- 3 replies
- 2 kudos
Resolved! Is there a way to tell if a created job is not compliant against configured cluster policies before it runs?
As a DevOps engineer, I want to enforce cluster policies at deployment time when the job is deployed/created, well before it is time to actually use it (i.e. before its scheduled/triggered run time without actually running it).
- 4187 Views
- 3 replies
- 2 kudos
- 2 kudos
Is it not the linked service that defines the kind of cluster created or used for any job?So i believe you could control the configuration via the linked service settings.
- 2 kudos
- 3461 Views
- 3 replies
- 20 kudos
Resolved! Create Databricks Workspace with different email address on Azure
Hi team,I wonder if we can create a Databricks Workspace that not releated with Azure email address.Thanks
- 3461 Views
- 3 replies
- 20 kudos
- 3320 Views
- 3 replies
- 14 kudos
Get a new badge or new certified for version 3 of DE exam
I took a certified of DE exam (version 2). Do I receive a new badge or certified when I pass newest version of DE exam?I'm going to take that and review my knowlege.
- 3320 Views
- 3 replies
- 14 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
