https://community.databricks.com/t5/technical-blog/attributing-costs-in-databricks-model-serving/ba-p/64818 <P><SPAN><STRONG>Authors:</STRONG> Sean Wilkinson, Ashwin Srikant, Cathy Zdravevski</SPAN></P> <P><SPAN>Databricks Model Serving provides a scalable, low-latency hosting service for AI models. It supports models ranging from small custom models to best-in-class large language models (LLMs). In this blog we’ll describe the pricing model associated with Databricks Model Serving and demonstrate how to allocate costs per endpoint or per use case.</SPAN></P> <H3><SPAN>Understanding billable usage</SPAN></H3> <P><SPAN>Databricks Model Serving now includes three distinct pricing methods. Regardless of the method you choose, the price is inclusive of all cloud infrastructure costs. The three different methods are covered briefly here:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><STRONG>Model and Feature Serving</STRONG><SPAN>: Choose a compute type (CPU/GPU) and a size that corresponds to a range of concurrent requests that the endpoint can handle. The serverless endpoint will scale seamlessly within this range and you pay for the actual compute allocated. If “scale to zero” is enabled, a $.07 charge per launch is incurred (max 2/hour).</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Foundation Model APIs (Provisioned Throughput)</STRONG><SPAN>: For large language model use cases that require consistent, low latency responses with high concurrency. It provides dedicated compute that scales between a configured set range. Databricks only charges for the actual compute used.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Foundation Model APIs (Pay-per-Token)</STRONG><SPAN>: Choose one of the available </SPAN><A href="https://docs.databricks.com/en/machine-learning/foundation-models/index.html#pay-per-token-foundation-model-apis" target="_blank" rel="noopener"><SPAN>state-of-the-art Foundation Models</SPAN></A><SPAN> and query it directly. Customers pay only for the input and output tokens consumed and produced by the model.</SPAN></LI> </UL> <P><SPAN>The best way to track model servings costs in Databricks is through the </SPAN><A href="https://docs.databricks.com/en/administration-guide/system-tables/billing.html" target="_blank" rel="noopener"><SPAN>billable usage system table</SPAN></A><SPAN>. Once </SPAN><A href="https://docs.databricks.com/en/administration-guide/system-tables/index.html#enable-system-table-schemas" target="_blank" rel="noopener"><SPAN>enabled</SPAN></A><SPAN>, the table automatically populates with the latest usage in your Databricks account. No matter which of the three model serving methods you choose, your costs will appear in the </SPAN><I><SPAN>system.billing.usage</SPAN></I><SPAN> table with column </SPAN><I><SPAN>sku_name</SPAN></I> <SPAN>as either:</SPAN></P> <P><STRONG>&lt;tier&gt;</STRONG><SPAN>_SERVERLESS_REAL_TIME_INFERENCE_LAUNCH_</SPAN><STRONG>&lt;region&gt;</STRONG></P> <P><SPAN>which includes all DBUs accrued when an endpoint starts after scaling to zero. All other model serving costs are grouped under:&nbsp;</SPAN></P> <P><STRONG>&lt;tier&gt;</STRONG><SPAN>_SERVERLESS_REAL_TIME_INFERENCE_</SPAN><STRONG>&lt;region&gt;</STRONG></P> <P><SPAN>where ti</SPAN><SPAN>er corresponds to your </SPAN><A href="https://www.databricks.com/product/pricing/platform-addons" target="_blank" rel="noopener"><SPAN>Databricks platform tier</SPAN></A><SPAN> and region corresponds to the cloud region of your Databricks deployment.&nbsp;</SPAN></P> <H3><SPAN>Querying and visualizing Model Serving usage</SPAN></H3> <P><SPAN>You can easily query the </SPAN><I><SPAN>system.billing.usage</SPAN></I><SPAN> table to aggregate all DBUs (Databricks Units) associated with Databricks model serving. Here is an example query that aggregates model serving DBUs per day for the last 30 days:</SPAN></P> <TABLE style="border-style: hidden; width: 100%;" border="1" width="100%"> <TBODY> <TR> <TD width="100%"><LI-CODE lang="python">SELECT SUM(usage_quantity) AS model_serving_dbus, usage_date FROM system.billing.usage WHERE sku_name LIKE '%SERVERLESS_REAL_TIME_INFERENCE%' GROUP BY(usage_date) ORDER BY usage_date DESC LIMIT 30</LI-CODE></TD> </TR> </TBODY> </TABLE> <P><FONT color="#000000"><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="Model Serving DBUs per Day For Last 30 Days" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/6836i3CD8CAA8262B8A93/image-size/large?v=v2&amp;px=999" role="button" title="sewi_ml_0-1711555204535.png" alt="Model Serving DBUs per Day For Last 30 Days" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Model Serving DBUs per Day For Last 30 Days</span></span></FONT></P> <H3><SPAN>Cost attribution with custom tags</SPAN></H3> <P><SPAN>Aggregated costs may be sufficient for simple use cases, but as the number of endpoints grows it is desirable to break out costs based on use case, business unit, or other custom identifiers. Optional key/value tags can be applied to custom models endpoints. All custom tags applied to Databricks Model Serving endpoints propagate to the </SPAN><I><SPAN>system.billing.usage</SPAN></I><SPAN> table under the </SPAN><I><SPAN>custom_tags</SPAN></I><SPAN> column and can be used to aggregate and visualize costs. Databricks recommends adding descriptive tags to each endpoint for precise cost tracking.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="Applying Custom Tags" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/6837i67F06DB2AA938D71/image-size/large?v=v2&amp;px=999" role="button" title="sewi_ml_1-1711555204609.png" alt="Applying Custom Tags" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Applying Custom Tags</span></span></P> <P><SPAN>Below is an example query that separates model serving costs by values of a specific tag for the Databricks account over the last 30 days.</SPAN></P> <TABLE style="border-style: hidden; width: 100%;" border="1" width="100%"> <TBODY> <TR> <TD width="100%"><LI-CODE lang="python">SELECT value, SUM(usage_quantity) AS DBUs FROM ( SELECT usage_date, usage_quantity, -- Use the built in EXPLODE() function to create a new row per tag. EXPLODE(custom_tags) FROM system.billing.usage WHERE sku_name LIKE '%SERVERLESS_REAL_TIME_INFERENCE%' AND usage_date &gt; DATE_SUB(CURRENT_DATE(), 30) ORDER BY usage_date DESC ) WHERE key = {{ filter_key }} GROUP BY value ORDER BY DBUs DESC</LI-CODE></TD> </TR> </TBODY> </TABLE> <P><SPAN>Running the query in the Databricks SQL Editor breaks out model serving costs by value of the tag over the past month:</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="DBUs by Endpoint Owner Over Past 30 Days" style="width: 754px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/6838i0AE3EDAAAAB9DC38/image-size/large?v=v2&amp;px=999" role="button" title="sewi_ml_2-1711555204565.png" alt="DBUs by Endpoint Owner Over Past 30 Days" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">DBUs by Endpoint Owner Over Past 30 Days</span></span></P> <H3><SPAN><BR />Conclusion</SPAN></H3> <P><SPAN>This is just the start of what you can view and visualize using the </SPAN><I><SPAN>system.billing.usage</SPAN></I><SPAN> tables in Databricks! Stay tuned as Databricks plans to roll out additional tables and metrics within the system catalog.</SPAN></P> Wed, 14 Aug 2024 14:34:51 GMT sewi_ml 2024-08-14T14:34:51Z https://community.databricks.com/t5/technical-blog/attributing-costs-in-databricks-model-serving/ba-p/64818 <P>Learn how to allocate Databricks Model Serving costs per endpoint, use case, or other custom identifier!</P> Wed, 14 Aug 2024 14:34:51 GMT https://community.databricks.com/t5/technical-blog/attributing-costs-in-databricks-model-serving/ba-p/64818 sewi_ml 2024-08-14T14:34:51Z