- 25328 Views
- 3 replies
- 7 kudos
Resolved! can we update a Parquet file??
i have copied a table in to a Parquet file now can i update a row or a column in a parquet file without rewriting all the data as the data is huge.using Databricks or ADFThank You
- 25328 Views
- 3 replies
- 7 kudos
- 7 kudos
You can only append Data with Parquet that's why you need to convert your parquet table to Delta. It will be much easier.
- 7 kudos
- 14189 Views
- 5 replies
- 5 kudos
Resolved! Override and Merge mode write using AutoLoader in Databricks
We are reading files using Autoloader in Databricks. Source system is giving full snapshot of complete data in files. So we want to read the data and write in delta table in override mode so all old data is replaced by the new data. Similarly for oth...
- 14189 Views
- 5 replies
- 5 kudos
- 5 kudos
@Ranjeet Jaiswal​ ,afaik merge is supported:https://docs.databricks.com/_static/notebooks/merge-in-streaming.htmlThis link does some aggregation but that can be ommitted of course.​The interesting part here is outputMode("update"), and the foreachBat...
- 5 kudos
- 4315 Views
- 4 replies
- 6 kudos
Where to find documentation about : spark.databricks.driver.strace.enabled
Hello ,For a support request, Microsoft support ask me to add spark.databricks.driver.strace.enabled trueto my cluster configuration.MS was not able to send me a link to the documentation and I did not find it on the databricks website.Can someone he...
- 4315 Views
- 4 replies
- 6 kudos
- 6 kudos
Yes no problem. I have a python program, called "post ingestion", that run on a databricks job cluster during the night and consist of :inserting data to a deltalake tableexecuting an optimize command on that tableexecuting a vacuum command on that t...
- 6 kudos
- 3276 Views
- 2 replies
- 3 kudos
Resolved! Don't receiving password reset email
Hi, our admin created new user in https://accounts.cloud.databricks.com/ with my email dusan.vystrcil@datasentics.com ​But I didn't received any confirmation email. When I try to sign in and click on "reset password", I still didn't received any emai...
- 3276 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @karthik p​ Thank you for reaching out, and we’re sorry to hear about this log-in issue! We have this Community Edition login troubleshooting post on Community. Please take a look, and follow the troubleshooting steps. If the steps do not resolve ...
- 3 kudos
- 1807 Views
- 0 replies
- 6 kudos
Happy to share that #WAVICLE​ was able to do a hands-on workshop on #[Databricks notebook]​ #[Databricks SQL]​ #[Databricks cluster]​ Fundamentals wi...
Happy to share that #WAVICLE​ was able to do a hands-on workshop on #[Databricks notebook]​ #[Databricks SQL]​ #[Databricks cluster]​ Fundamentals with KCT College, Coimbatore, India.
- 1807 Views
- 0 replies
- 6 kudos
- 2242 Views
- 1 replies
- 3 kudos
Hi I'm Deiry �� I'm 25 (almost 26) years old, I'm a Databricks expert ��  Or at least that's my goal I work at Celerik....
Hi I'm Deiry I'm 25 (almost 26) years old, I'm a Databricks expert Or at least that's my goalI work at Celerik.My goal is to be a certified Machine Learning professional, so here we go
- 2242 Views
- 1 replies
- 3 kudos
- 3 kudos
Very confident, go ahead. :D​
- 3 kudos
- 3790 Views
- 3 replies
- 1 kudos
Resolved! When should I use STREAM() when defining a DLT table?
Hi, I am a little confused when I should use STREAM() when we define a table based on a DLT table. There is a pattern explained in the documentation. CREATE OR REFRESH STREAMING LIVE TABLE streaming_bronze AS SELECT * FROM cloud_files( "s3://p...
- 3790 Views
- 3 replies
- 1 kudos
- 1 kudos
Thanks @Landan George​ Since "streaming_silver" is a streaming live table, I expected the last line of the code to be:AS SELECT count(*) FROM STREAM(LIVE.streaming_silver) GROUP BY user_idBut, as you can see the "live_gold" is defined by: AS SELECT c...
- 1 kudos
- 1548 Views
- 0 replies
- 0 kudos
- 1548 Views
- 0 replies
- 0 kudos
- 2558 Views
- 2 replies
- 2 kudos
ADLS Gen 2 Delta Tables memory allocation
if i mount a gen2(ADLS 1) to another gen2(ADLS2) account and create a delta table on ADLS2 will it copy the data or just create something link External table.i don't want to duplicate the the data.
- 2558 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @keerthi kumar​ ,so basically you can CREATE EXTERNAL TABLES on top of the data stored somewhere - in your case ADLS. Data won't be copied, it will stay where it is, by creating external tables you are actually storing the metadata in your metasto...
- 2 kudos
- 2880 Views
- 2 replies
- 1 kudos
Would like to start a discussion regarding techniques for joining two relatively large tables of roughly equal size on a daily basis. I realize this may be a bit of a conundrum with databricks, but review the details.
Input Data:One batch load of a daily dataset, roughly 10 million items a day of transactions.Another daily batch load of roughly the same size.Each row in one dataset should have a corresponding row in the other dataset.Problem to solve:The problem i...
- 2880 Views
- 2 replies
- 1 kudos
- 1 kudos
I've dealt with something similar in the past.There was an order system that had order items that was supposed to be matched up against corresponding products in another system that acted as a master and handled invoicing.As for unqiue considerations...
- 1 kudos
- 21045 Views
- 2 replies
- 12 kudos
Resolved! How can Delta table protocol version be downgraded from higher version to lower version the table properties minReader from 2 to 1 and MaxWriter from 5 to 3.
Is there a possibility to downgrade the Delta Table protocol versions minReader from 2 to 1 and maxWriter from 5 to 3? I have set the TBL properties to 2 and 5 and columnmapping mode to rename the columns in the DeltaTable but the other users are rea...
- 21045 Views
- 2 replies
- 12 kudos
- 12 kudos
Unfortunately You can't downgrade the version. it's an irreversible operation.
- 12 kudos
- 1556 Views
- 0 replies
- 2 kudos
get diagnostics using pyspark instead of microsoft diagnostics
I know you can use microsoft diagnostics to receive notifications such as commandSubmit or commandFinish in json. But is it possible to get these diagnostics in databricks itself using pyspark?I would like to get these diagnostics inside of a noteboo...
- 1556 Views
- 0 replies
- 2 kudos
- 12916 Views
- 3 replies
- 3 kudos
Resolved! Databricks SQL dashboard refresh
We have scheduled a dashboard to automatically refresh at some specific time.However, some visualizations in the dashboard don't get refreshed at the scheduled time.Checking the query logs, we found the source queries were properly executed, but the ...
- 12916 Views
- 3 replies
- 3 kudos
- 3 kudos
Thank you for the answers.I'm using Databricks SQL environment, not the Data Science & Engineering one. And, I scheduled the dashboard following this guideline: https://docs.databricks.com/sql/user/dashboards/index.html#automatically-refresh-a-dashbo...
- 3 kudos
- 3065 Views
- 3 replies
- 2 kudos
Which is quicker: grouping a table that is a join of several others or querying data?
This may be a tricky question, so please bear with meIn a real life scenario, i have a dataframe (i'm using pyspark) called age, with is a groupBy of other 4 dataframes. I join these 4 so at the end I have a few million rows, but after the groupBy th...
- 3065 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Marcos Dias​ ,Frankly, I think we need more detail to answer your question:Are these 4 dataframes​ updated their data?How often you use the groupBy-dataframe?
- 2 kudos
- 6385 Views
- 5 replies
- 4 kudos
remote database connection error
Hi,I am using databricks through azure. I am trying to connect to remote oracle database using jdbc url. I am getting an error of no suitable driver found."java.sql.SQLException: No suitable driver"can somebody help mw with this?
- 6385 Views
- 5 replies
- 4 kudos
- 4 kudos
Hi @vikas k​ Hope all is well! Does @Hubert Dudek​ response were able to resolve your issue, and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 4 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 576 | |
| 349 | |
| 287 |
