- 7597 Views
- 7 replies
- 2 kudos
What happens if we change the logic for the delta live tables and we do an incremental update. Does the table get reset (refresh) automatically or would it only apply the logic to new incoming data? would we have to trigger a reset in this case?
- 7597 Views
- 7 replies
- 2 kudos
Latest Reply
Here is my finding on when to refresh (reset) the table: If it is a complete table all the changes would be apply automatically. If the table is incremental table, you need to do a manually reset (full refresh).
6 More Replies
by
Brose
• New Contributor III
- 13677 Views
- 9 replies
- 2 kudos
I am trying to create a delta table for streaming data, but I am getting the following error; Error in SQL statement: ParseException: mismatched input 'CREATE' expecting {<EOF>, ';'}(line 2, pos 0).My statement is as follows;%sqlDROP TABLE IF EXISTS ...
- 13677 Views
- 9 replies
- 2 kudos
Latest Reply
@Ambrose Walker - If Jose's answer resolved your issue, would you be happy to mark that post as best? That will help others find the solution more quickly.
8 More Replies
by
guruv
• New Contributor III
- 5412 Views
- 4 replies
- 2 kudos
HI,i have several delta tables on Azure adls gen 2 storage account running databricks runtime 7.3. there are only write/read operation on delta tables and no update/delete.As part of release pipeline, below commands are executed in a new notebook in...
- 5412 Views
- 4 replies
- 2 kudos
Latest Reply
the auto optimize is sufficient, unless you run into performance issues.Then I would trigger an optimize. This will generate files of 1GB (so larger than the standard size of auto optimize). And of course the Z-Order if necessary.The suggestion to ...
3 More Replies
- 2325 Views
- 2 replies
- 6 kudos
Have you checked out the new Bill Inmon Book, Building the Data Lakehouse? https://dbricks.co/3uxCXjO What were your thoughts if you read it?
- 2325 Views
- 2 replies
- 6 kudos
Latest Reply
The quality of the book depends on the audience IMO. For people who have no background in data warehousing it will be interesting to read. For the others the book is too general and descriptive. The 'HOW do you do x' is missing.
1 More Replies
- 2178 Views
- 1 replies
- 3 kudos
Specifically for write and read streaming data to HDFS or s3 etc. For IoT specific scenario how it performs on time series transactional data. Can we consider delta table as time series table?
- 2178 Views
- 1 replies
- 3 kudos
Latest Reply
Hi @Arindam Halder , Delta lake is more performant compared to a regular parquet table. pls check below for some stats on the performancehttps://docs.azuredatabricks.net/_static/notebooks/delta/optimize-python.htmlyes, you can use it for time series...
- 4303 Views
- 2 replies
- 0 kudos
I have an existing data pipeline which looks like this: A small MySQL data source (around 250 GB) and data passes through Debezium/ Kafka / a custom data redactor -> to Glue ETL jobs and finally lands on Redshift, but the scale of the data is too sm...
- 4303 Views
- 2 replies
- 0 kudos
Latest Reply
Dan_Z
Databricks Employee
There is a lot in this question, so generally speaking I suggest you reach out to the sales team at Databricks. You can talk to a solutions architect who get into more detail. Here are my general thoughts having seen a lot of customer arch:Generally,...
1 More Replies
- 1576 Views
- 0 replies
- 0 kudos
What's the best way to federate a query to delta lake or the databricks from presto sql without having to create external tables? PrestoSQL doesn't have access to S3. Can PrestoSQL be configured with jdbc driver or plugin?
- 1576 Views
- 0 replies
- 0 kudos
