- 1674 Views
- 1 replies
- 1 kudos
- 1674 Views
- 1 replies
- 1 kudos
- 1 kudos
val oldestVersionAvailable = val newestVersionAvailable = val pathToDeltaTable = "" val pathToFileName = "" (oldestVersionAvailable to newestVersionAvailable).map { version => var df1 = spark.read.json(f"$pathToDeltaTable/_delta_log/$version%0...
- 1 kudos
- 3759 Views
- 1 replies
- 1 kudos
Trying to write my dataframe out as a tab separated .txt file but getting an error
When I try to save my file I getorg.apache.spark.sql.AnalysisException: Text data source supports only a single column, and you have 2 columns.; Is there any way to save a dataframe with more than one column to a .txt file?
- 3759 Views
- 1 replies
- 1 kudos
- 1 kudos
Would pyspark.sql.DataFrameWriter.csv work? You could specify the separator (sep) as tabdf.write.csv(os.path.join(tempfile.mkdtemp(), 'data'))
- 1 kudos
- 2185 Views
- 1 replies
- 1 kudos
- 2185 Views
- 1 replies
- 1 kudos
- 1 kudos
%scala display(spark.read.json("//path-to-delta-table/_delta_log/0000000000000000000x.json") .where("add is not null") .select("add.path"))
- 1 kudos
- 3417 Views
- 1 replies
- 1 kudos
- 3417 Views
- 1 replies
- 1 kudos
- 1 kudos
SELECT * FROM delta_table_name@v2 EXCEPT ALL FROM delta_table_name@v0
- 1 kudos
- 2856 Views
- 2 replies
- 0 kudos
Is there a way so set DBU or cost limits so I don't get an unexpected bill?
I'm wondering if there's a way to set a monthly budget and have my workloads stop running if I hit it.
- 2856 Views
- 2 replies
- 0 kudos
- 0 kudos
Cluster Policies would help with this not only from a cost management perspective but also standardization of resources across the organization as well simplification for a better user experience. You can find Best Practices on leveraging cluster pol...
- 0 kudos
- 2485 Views
- 1 replies
- 0 kudos
What is the default location where dataframes are written if I don't specify a location?
If I save a dataframe without specifying a location, where will it end up?
- 2485 Views
- 1 replies
- 0 kudos
- 0 kudos
You cant save a dataframe without specifying a location. If you are using saveAsTable API then the table will be created in the hive warehouse location. The default location is user.hive.warehouse
- 0 kudos
- 2144 Views
- 1 replies
- 2 kudos
Why would I make a deep clone of a Delta table vs reading the table and writing a copy to a new location?
It seems like with both techniques I would end up with a copy of my table. Trying to understand when I should be using a deep clone.
- 2144 Views
- 1 replies
- 2 kudos
- 2 kudos
A deep clone is recommended way as it holds the history of the table. Also, the DEEP clone is faster than the read-write approach.
- 2 kudos
- 2377 Views
- 1 replies
- 0 kudos
How can I run OPTIMIZE on a table if I am streaming to it 24/7?
I have a table that I need to be continuously streaming into. I know it's best practice to run Optimize on my tables periodically. But if I never stop writing to the table, how and when can I run OPTIMIZE against it?
- 2377 Views
- 1 replies
- 0 kudos
- 0 kudos
If the streaming job is making bling appends to the delta table, then it's perfectly fine to run OPTIMIZE query in parallel.However, if the streaming job is performing MERGE or UPDATE then it can conflict with the OPTIMIZE operations. In such cases w...
- 0 kudos
- 2626 Views
- 1 replies
- 0 kudos
- 2626 Views
- 1 replies
- 0 kudos
- 0 kudos
delta.logRetentionDuration - 30 daysdelta.deletedFileRetentionDuration - 7 days
- 0 kudos
- 2331 Views
- 1 replies
- 0 kudos
- 2331 Views
- 1 replies
- 0 kudos
- 0 kudos
Using cluster tags we can get the cluster name spark.conf.get("spark.databricks.clusterUsageTags.clusterName")
- 0 kudos
- 1310 Views
- 1 replies
- 0 kudos
Resolved! Best practices for DStream application in Databricks
I do not see any best practice guide for the DStream application in Databricks docs. Any reference
- 1310 Views
- 1 replies
- 0 kudos
- 0 kudos
Dstream is unsupported by Databricks. Databrcks strongly recommend migrating the Dstream applications to use Structured Streaminghttps://kb.databricks.com/streaming/dstream-not-supported.html
- 0 kudos
- 1235 Views
- 1 replies
- 1 kudos
Optimize Command not performing the bin packing
I have a daily OPTIMIZE job running, however, the number of files in the storage is not going down. Looks like the optimize is not helping to reduce the files.
- 1235 Views
- 1 replies
- 1 kudos
- 1 kudos
The files are not physically removed from the Storage by the optimize command. A VACUUM command has to be executed to achieve the same
- 1 kudos
- 1286 Views
- 1 replies
- 0 kudos
- 1286 Views
- 1 replies
- 0 kudos
- 0 kudos
https://academy.databricks.com/category/self-paced ?
- 0 kudos
- 16330 Views
- 1 replies
- 0 kudos
How to run multiple spark streaming application on databricks cluster?
I started working on databricks. I need to migrate few streaming jobs from Ambari to Databricks. I deployed one job using jar and it. is working fine. But when I deploy the second job I faced an error " multiple spark streaming context not allowed". ...
- 16330 Views
- 1 replies
- 0 kudos
- 0 kudos
You can run multiple streaming applications in databricks clusters. By default, this would run in the same fair scheduling pool. To enable multiple streaming queries to execute jobs concurrently and to share the cluster efficiently, you can set the q...
- 0 kudos
- 1330 Views
- 1 replies
- 0 kudos
- 1330 Views
- 1 replies
- 0 kudos
- 0 kudos
We still require a single user to be an owner. But you can set a group to have CAN_MANAGE which unblocks most of the necessary updates. It is released in all Premium workspaces that have Jobs ACLs. The official OWNER is whose identity is used to crea...
- 0 kudos
| User | Count |
|---|---|
| 1623 | |
| 790 | |
| 501 | |
| 349 | |
| 287 |
