- 1844 Views
- 1 replies
- 0 kudos
Resolved! versioning of delta table while writing from a structured streaming job
Does writing to a Delta table create a versioning for every micro-batch of stream
- 1844 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes it is correct - Every commits to the delta create a version so definitely each micro batch create a version More Info -: https://databricks.com/blog/2019/02/04/introducing-delta-time-travel-for-large-scale-data-lakes.html
- 0 kudos
- 3114 Views
- 1 replies
- 1 kudos
spark data frame parquet vs delta : rows Doesn't match
I have data written in Delta on ADLS. As I understand the delta also internal file in parquet format but when Iread the file in different format I got different record countspark.read.parquet() or spark.read.format('delta').load()df = spark.read.for...
- 3114 Views
- 1 replies
- 1 kudos
- 1 kudos
I think you have written in delta twice using overwrite mode .But Delta is versioned data format - when you use overwrite, it doesn't delete previous data, it just writes new files, and don't delete files immediately - they are just marked as delete...
- 1 kudos
- 14116 Views
- 3 replies
- 1 kudos
- 14116 Views
- 3 replies
- 1 kudos
- 1 kudos
Databricks does not charge DBUs while instances are idle in the pool. Instance provider billing does apply.Please refer here for more information - https://docs.databricks.com/clusters/instance-pools/index.html
- 1 kudos
- 6134 Views
- 1 replies
- 0 kudos
Resolved! Best ways to copy the parquet files in the staging directory to the Delta table
I have some parquet data in a temporary directory. Can I copy them into the delta table directly, what are the best options.
- 6134 Views
- 1 replies
- 0 kudos
- 0 kudos
The easiest solution is to use the COPY INTO command. The COPY INTO command ensures idempotency, so even if the operation fails there are no data inconsistencies. COPY INTO command utilizes the resources on the Spark cluster hence completes faster. h...
- 0 kudos
- 4583 Views
- 1 replies
- 0 kudos
Resolved! Unable to download the files from the notebook UI
I used to download the SQL query output from the Notebook UI. but right now I am unable to download files now
- 4583 Views
- 1 replies
- 0 kudos
- 0 kudos
This is a workspace-level configuration. Probably your workspace admin disabled it. If you have admin privilege on your workspace you can enable it from the Admin Console -> Workspace Settings
- 0 kudos
- 2644 Views
- 1 replies
- 0 kudos
MSCK REPAIR TABLE doesn't work in delta
I have a delta table in adls and for the same table, I have defined an external table in hive After creating the hive table and generating manifests, I am loading the partitions using MSCK REPAIR TABLE. All the partition columns are in same But s...
- 2644 Views
- 1 replies
- 0 kudos
- 0 kudos
Can you please check partition column order, does it in same sequence as before or it has changed
- 0 kudos
- 6381 Views
- 1 replies
- 0 kudos
Resolved! I can't find my cluster
I had a cluster that I used in the past. I do not see the cluster any longer. I checked with the admin and my team and everyone confirmed that there no user deletion.
- 6381 Views
- 1 replies
- 0 kudos
- 0 kudos
If the cluster is unsued for 30 days, Databricks removes the cluster. This is a general clean-up policy. It's possible to whitelist a cluster from this clean-up by Pinning the cluster. https://docs.databricks.com/clusters/clusters-manage.html#pin-a-c...
- 0 kudos
- 3503 Views
- 1 replies
- 0 kudos
Resolved! Delta adds a new partition making the old partition unreadable
In Notebook, My code read and write the data to delta , My delta is partitioned by calendar_date. After the initial load i am able to read the delta file and look the data just fine.But after the second load for data for 6 month , the previous part...
- 3503 Views
- 1 replies
- 0 kudos
- 0 kudos
I think you are writing the data in override mode. what happens in delta is it doesn't delete the data for certain days even it is written by overwrite mode for versioning , and you will be able to query only most recent data,But in format parque...
- 0 kudos
- 1840 Views
- 1 replies
- 0 kudos
Resolved! Additional permission for Delta compared to Parquet
I am trying to create a Delta table and it seems the Delta table requires additional permissions on the parent folder of the table. The command failed telling permission errors. I tried to create a parquet table and it works fine.
- 1840 Views
- 1 replies
- 0 kudos
- 0 kudos
Delta table is the non-hive compatible format. So there must also be permissions for a client to access the path to the database’s location so that it can create a new temporary “directory” there. This comes from Spark SQL’s handling of external tabl...
- 0 kudos
- 3091 Views
- 1 replies
- 1 kudos
Resolved! How many active connections are made to Hive metastore
We are using an internal metastore implementation. ie the metastore is hosted at the Dataricks side. However, we believe the metastore instance made available for my workspace is not adequate enough to handle the load. How can I monitor the number of...
- 3091 Views
- 1 replies
- 1 kudos
- 1 kudos
Use the below code snippet from a notebook%scala import java.sql.Connection import java.sql.DriverManager import java.sql.ResultSet import java.sql.SQLException /** * For details on what this query means, checkout https://dev.mysql.com/doc/refma...
- 1 kudos
- 3745 Views
- 1 replies
- 1 kudos
Resolved! Does Databricks integrate with Immuta?
My company uses Immuta for data governance. Will Databricks be able to fit into our existing security patterns?
- 3745 Views
- 1 replies
- 1 kudos
- 1 kudos
Yes, check out the immuta web page on the Databricks Integration. https://www.immuta.com/integrations/databricks
- 1 kudos
- 3628 Views
- 1 replies
- 0 kudos
Resolved! Compute cost for the SQL Analytics query
Is there a way to get some kind of compute the cost associated with every SQL analytics query?
- 3628 Views
- 1 replies
- 0 kudos
- 0 kudos
Right now, we do not have an option to measure the compute cost at a query level.
- 0 kudos
- 1028 Views
- 0 replies
- 0 kudos
docs.databricks.com
Best practices: Cluster configuration | Databricks on AWSLearn best practices when creating and configuring Databricks clusters.https://docs.databricks.com/clusters/cluster-config-best-practices.html
- 1028 Views
- 0 replies
- 0 kudos
- 870 Views
- 0 replies
- 0 kudos
docs.gcp.databricks.com
Best practices | Databricks on Google CloudLearn best practices when using or administering Databricks.https://docs.gcp.databricks.com/best-practices-index.html
- 870 Views
- 0 replies
- 0 kudos
- 922 Views
- 0 replies
- 0 kudos
docs.microsoft.com
Best practices - Azure DatabricksLearn best practices when using or administering Azure Databricks.https://docs.microsoft.com/en-us/azure/databricks/best-practices-index
- 922 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
