- 1727 Views
- 1 replies
- 0 kudos
- 1727 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes, in your write stream you can save it as a table in the delta format without a problem. In DBR 8, the default table format is delta. See this code, please note that the "..." is supplied to show that additional options may be required: df.writeSt...
- 0 kudos
- 2384 Views
- 1 replies
- 0 kudos
When using Delta Live Tables, how do I set a table to be incremental vs complete using Python?
When using SQL, I can use the Create Live Table command and the Create Incremental Live Table command to set the run type I want the table to use. But I don't seem to have that same syntax for python. How can I set this table type while using Python?
- 2384 Views
- 1 replies
- 0 kudos
- 0 kudos
The documentation at https://docs.databricks.com/data-engineering/delta-live-tables/delta-live-tables-user-guide.html#mixing-complete-tables-and-incremental-tables has an example the first two functions load data incrementally and the last one loads...
- 0 kudos
- 2672 Views
- 1 replies
- 0 kudos
Is it possible to disable the maintenance job associated with a Delta Live Table?
After creating my Delta Live Table and running it once, I notice that the maintenance job that was created along with it continues to run at the scheduled time. I have not made any updated to the DLT, so the maintenance job theoretically shouldn't ha...
- 2672 Views
- 1 replies
- 0 kudos
- 0 kudos
You could change the table properties of the associated tables to disable automatic scheduled optimizations. More details at https://docs.databricks.com/data-engineering/delta-live-tables/delta-live-tables-language-ref.html#table-properties
- 0 kudos
- 3627 Views
- 1 replies
- 1 kudos
Secrets in databricks
I created a secret on databricks using the secrets API.Code :Scope_name : {"scope": "dbtest", "initial_manage_principal":"user"} Resp= requests.post('https://instancename.net/mynoteid/api/2.0/secrets/scopes/create',json=Scope_name)Similar way, I adde...
- 3627 Views
- 1 replies
- 1 kudos
- 1 kudos
You'll have to specify the scope and the key in the format below to get the value. dbutils.secret.get(scope="dbtest", key="user") Probably a good idea to review the Secret Management documentation for details on how to get this setup the right way - ...
- 1 kudos
- 2326 Views
- 1 replies
- 0 kudos
Resolved! MLflow Model Serving latency expectations
What kind of latency should I expect when using the built in model serving capability in MLflow. Evaluating whether it would be a good fit for our use case
- 2326 Views
- 1 replies
- 0 kudos
- 0 kudos
What are your throughput requirements in addition to latency. Currently this is in private preview and databricks recommends this only for low throughput and non-critical applications. However, as it move towards GA, this would change. Please get in...
- 0 kudos
- 1581 Views
- 1 replies
- 1 kudos
- 1581 Views
- 1 replies
- 1 kudos
- 1 kudos
val oldestVersionAvailable = val newestVersionAvailable = val pathToDeltaTable = "" val pathToFileName = "" (oldestVersionAvailable to newestVersionAvailable).map { version => var df1 = spark.read.json(f"$pathToDeltaTable/_delta_log/$version%0...
- 1 kudos
- 3567 Views
- 1 replies
- 1 kudos
Trying to write my dataframe out as a tab separated .txt file but getting an error
When I try to save my file I getorg.apache.spark.sql.AnalysisException: Text data source supports only a single column, and you have 2 columns.; Is there any way to save a dataframe with more than one column to a .txt file?
- 3567 Views
- 1 replies
- 1 kudos
- 1 kudos
Would pyspark.sql.DataFrameWriter.csv work? You could specify the separator (sep) as tabdf.write.csv(os.path.join(tempfile.mkdtemp(), 'data'))
- 1 kudos
- 2075 Views
- 1 replies
- 1 kudos
- 2075 Views
- 1 replies
- 1 kudos
- 1 kudos
%scala display(spark.read.json("//path-to-delta-table/_delta_log/0000000000000000000x.json") .where("add is not null") .select("add.path"))
- 1 kudos
- 3301 Views
- 1 replies
- 1 kudos
- 3301 Views
- 1 replies
- 1 kudos
- 1 kudos
SELECT * FROM delta_table_name@v2 EXCEPT ALL FROM delta_table_name@v0
- 1 kudos
- 2714 Views
- 2 replies
- 0 kudos
Is there a way so set DBU or cost limits so I don't get an unexpected bill?
I'm wondering if there's a way to set a monthly budget and have my workloads stop running if I hit it.
- 2714 Views
- 2 replies
- 0 kudos
- 0 kudos
Cluster Policies would help with this not only from a cost management perspective but also standardization of resources across the organization as well simplification for a better user experience. You can find Best Practices on leveraging cluster pol...
- 0 kudos
- 2386 Views
- 1 replies
- 0 kudos
What is the default location where dataframes are written if I don't specify a location?
If I save a dataframe without specifying a location, where will it end up?
- 2386 Views
- 1 replies
- 0 kudos
- 0 kudos
You cant save a dataframe without specifying a location. If you are using saveAsTable API then the table will be created in the hive warehouse location. The default location is user.hive.warehouse
- 0 kudos
- 2033 Views
- 1 replies
- 2 kudos
Why would I make a deep clone of a Delta table vs reading the table and writing a copy to a new location?
It seems like with both techniques I would end up with a copy of my table. Trying to understand when I should be using a deep clone.
- 2033 Views
- 1 replies
- 2 kudos
- 2 kudos
A deep clone is recommended way as it holds the history of the table. Also, the DEEP clone is faster than the read-write approach.
- 2 kudos
- 2282 Views
- 1 replies
- 0 kudos
How can I run OPTIMIZE on a table if I am streaming to it 24/7?
I have a table that I need to be continuously streaming into. I know it's best practice to run Optimize on my tables periodically. But if I never stop writing to the table, how and when can I run OPTIMIZE against it?
- 2282 Views
- 1 replies
- 0 kudos
- 0 kudos
If the streaming job is making bling appends to the delta table, then it's perfectly fine to run OPTIMIZE query in parallel.However, if the streaming job is performing MERGE or UPDATE then it can conflict with the OPTIMIZE operations. In such cases w...
- 0 kudos
- 2544 Views
- 1 replies
- 0 kudos
- 2544 Views
- 1 replies
- 0 kudos
- 0 kudos
delta.logRetentionDuration - 30 daysdelta.deletedFileRetentionDuration - 7 days
- 0 kudos
- 2236 Views
- 1 replies
- 0 kudos
- 2236 Views
- 1 replies
- 0 kudos
- 0 kudos
Using cluster tags we can get the cluster name spark.conf.get("spark.databricks.clusterUsageTags.clusterName")
- 0 kudos
| User | Count |
|---|---|
| 1616 | |
| 790 | |
| 468 | |
| 349 | |
| 287 |
