Hi @kbmv ,

Based on my experience deploying Deepseek-R1-Distilled-Llama on Databricks, here are my answers to your questions:

1. Compute Requirements for MLflow Registration (70B vs 8B Model) • Llama-8B was successfully registered using a cluster with 192GB memory, 40 cores, and GPU. • Llama-70B failed to register on the same setup, indicating that it requires even more resources. • A CPU-only cluster with high memory was also tested, but it failed due to insufficient memory. • Conclusion: For me, the recommendation of using g6e.4xlarge (128GB CPU, 48GB GPU memory) seems to be the minimum needed for Llama-70B registration.
2. GPU Requirement for Deployment • The blog states that GPUs are not strictly required for deployment if enough memory is available. • However, in practice, serving the 70B model without GPUs is not feasible due to high memory consumption and inference latency. • For Llama-8B, serving without GPUs is possible, but performance may be impacted. • Conclusion: MLflow registration is best done with GPUs, and for efficient inference serving, GPUs are strongly recommended, especially for 70B.
3. Pricing for Provisioned Throughput Serving • As of now, Deepseek-R1-Distilled models are not explicitly listed in the pricing documentation. • However, given that Deepseek-R1-Distilled-70B is based on Llama 3.3 70B, it is likely that pricing will be similar to Llama 3.3 70B. The 8B version may align with Llama 3.1B pricing, but confirmation from Databricks would be required.
4. Compute Selection for RAG Chains and Agent Models • For Llama-70B, the best practice is to use a GPU-enabled cluster, as inference latency will be too high on CPU. • For Llama-8B, CPU may work for some use cases, but performance will degrade significantly. • The compute type selection (CPU, GPU small, etc.) applies to Deepseek-R1 models as well, and choosing GPU is recommended for real-time applications.
   
   🙂