https://community.databricks.com/t5/technical-blog/mlops-gym-beginners-guide-to-mlflow/ba-p/62593 <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="SepidehEb_0-1710925847561.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/6720iB3EF1364FCE31725/image-size/medium?v=v2&amp;px=400" role="button" title="SepidehEb_0-1710925847561.png" alt="SepidehEb_0-1710925847561.png" /></span></P> <P>&nbsp;</P> <H1><SPAN>MLflow</SPAN></H1> <P><A href="https://mlflow.org/docs/latest/introduction/index.html" target="_blank" rel="noopener"><SPAN>MLflow</SPAN></A><SPAN> stands out as the leading open source MLOps tool, and we strongly recommend its integration into your machine learning lifecycle. With its diverse components, MLflow significantly boosts productivity across various stages of your machine learning journey. The utilisation of MLflow is integral to many of the patterns we showcase in the MLOps Gym.</SPAN></P> <P><SPAN>Built on top of OS MLflow, Databricks offers a managed MLflow service that focuses on enterprise reliability, security, and scalability. Among its many advantages, the managed version of MLflow natively integrates with </SPAN><A href="https://docs.databricks.com/en/notebooks/index.html" target="_blank" rel="noopener"><SPAN>Databricks Notebooks</SPAN></A><SPAN>, making it simpler to kickstart your MLOps journey. For a detailed comparison between OS and Managed MLflow, refer to the </SPAN><A href="https://www.databricks.com/product/managed-mlflow" target="_blank" rel="noopener"><SPAN>Databricks Managed MLflow product page</SPAN></A><SPAN>.</SPAN></P> <P><SPAN>Although MLflow encompasses seven core components (</SPAN><A href="https://mlflow.org/docs/latest/tracking.html#tracking" target="_blank" rel="noopener"><SPAN>Tracking</SPAN></A><SPAN>, </SPAN><A href="https://mlflow.org/docs/latest/model-registry.html#registry" target="_blank" rel="noopener"><SPAN>Model Registry</SPAN></A><SPAN>, </SPAN><A href="https://mlflow.org/docs/latest/llms/deployments/index.html#deployments" target="_blank" rel="noopener"><SPAN>MLflow Deployment for LLMs</SPAN></A><SPAN>, </SPAN><A href="https://mlflow.org/docs/latest/models.html#model-evaluation" target="_blank" rel="noopener"><SPAN>Evaluate</SPAN></A><SPAN>, </SPAN><A href="https://mlflow.org/docs/latest/llms/prompt-engineering/index.html#prompt-engineering" target="_blank" rel="noopener"><SPAN>Prompt Engineering UI</SPAN></A><SPAN>, </SPAN><A href="https://mlflow.org/docs/latest/recipes.html#recipes" target="_blank" rel="noopener"><SPAN>Recipes</SPAN></A><SPAN>, and </SPAN><A href="https://mlflow.org/docs/latest/projects.html#projects" target="_blank" rel="noopener"><SPAN>Projects</SPAN></A><SPAN> ) as of the current date (March 2024), our focus in this article will be on the initial five components. We chose these components because they are most effective in solving MLOps challenges for our customers with minimal friction. We will provide recommendations on when and how to leverage them effectively.</SPAN></P> <P><SPAN>We have broken this guide to MLflow into three parts:</SPAN><SPAN><BR /><BR /></SPAN></P> <OL> <LI style="font-weight: 400;" aria-level="1"><I><SPAN>Beginners’ guide to MLflow </SPAN></I><SPAN>will cover MLflow essentials for all ML practitioners. In this article, we discuss </SPAN><STRONG>Tracking</STRONG><SPAN> and </SPAN><STRONG>Model Registry</STRONG><SPAN> components.</SPAN><SPAN><BR /></SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><A href="https://community.databricks.com/t5/technical-blog/mlops-gym-evaluating-large-language-models-with-mlflow/ba-p/72815" target="_blank"><I><SPAN>Evaluating Large Language Models with MLflow</SPAN></I></A> <SPAN>is dedicated to the </SPAN><STRONG>Evaluate </STRONG><SPAN>component.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Deploying Large Language Models with MLflow will cover the remainder of MLflow’s LLM-related components such as <STRONG style="color: #1b3139; font-family: inherit;">MLflow Deployment for LLMs</STRONG><SPAN> and</SPAN> <SPAN>the</SPAN><STRONG style="color: #1b3139; font-family: inherit;"> Prompt Engineering UI</STRONG><SPAN>.</SPAN><BR /></SPAN></LI> </OL> <P><SPAN>Here are our suggested steps for getting started with MLflow. Remember that in the remainder of this article, “MLflow” refers to the Databricks-managed version of MLflow.&nbsp;</SPAN></P> <H2 id="toc-hId--440979465"><SPAN>You won't want to miss this...</SPAN></H2> <P><SPAN>As a companion to this article, we spoke to&nbsp;one of our most valued partners,&nbsp;<A href="https://www.linkedin.com/in/gavitaregunath/" target="_self" rel="nofollow noopener noreferrer">Gavi Regunath</A>, about this subject. You can watch the recording of that discussion on the&nbsp;<A href="https://www.youtube.com/@AdvancingAnalytics" target="_self" rel="nofollow noopener noreferrer">Advancing Analytics YouTube channel</A>.</SPAN></P> <P><IFRAME src="https://www.youtube.com/embed/xsHPiIuEC1I?si=LEc262QKLMqE4aGg" width="100%" height="400" frameborder="0" allowfullscreen="" title="YouTube video player" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin"></IFRAME></P> <H2><SPAN><BR />Use MLflow Tracking</SPAN></H2> <P><SPAN>Your entry point to the MLflow world is probably MLflow Tracking. Using this component, you can log every aspect of your machine learning experiment, such as source properties, parameters, metrics, tags, and artifacts. MLflow Tracking assists data scientists during training and evaluating models, obviating the need to manually record the parameters and results of every experiment and facilitating reproducibility.</SPAN></P> <P><SPAN>&nbsp;MLflow Tracking comprises the following components:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><A href="https://www.mlflow.org/docs/latest/tracking/server.html" target="_blank" rel="noopener"><SPAN>Tracking Server</SPAN></A><SPAN>, which is an HTTP server hosted on Databricks Control Plane for tracking runs/ experiments.</SPAN></LI> </UL> <UL> <LI style="font-weight: 400;" aria-level="1"><A href="https://www.mlflow.org/docs/latest/tracking/backend-stores.html" target="_blank" rel="noopener"><SPAN>Backend Store</SPAN></A><SPAN>, which is a SQL database hosted on Databricks Control Plane to store metadata for runs/experiments such as run id, start and end time, parameters, etc.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><A href="https://www.mlflow.org/docs/latest/tracking/artifacts-stores.html" target="_blank" rel="noopener"><SPAN>Artifact store</SPAN></A><SPAN>, which is where MLflow stores the actual artifacts of runs such as model weights, data files, etc. Databricks uses </SPAN><A href="https://docs.databricks.com/en/dbfs/index.html" target="_blank" rel="noopener"><SPAN>DBFS</SPAN></A><SPAN> as its default artifact store for MLflow.&nbsp;</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Tracking </SPAN><A href="https://www.mlflow.org/docs/latest/tracking.html#tracking-ui" target="_blank" rel="noopener"><SPAN>UI</SPAN></A><SPAN> and </SPAN><A href="https://www.mlflow.org/docs/latest/tracking/tracking-api.html" target="_blank" rel="noopener"><SPAN>APIs</SPAN></A><SPAN>, which allow you to interact with MLflow Tracking API through UI or API calls, respectively. The UI has tools for exploring metadata and artifacts and comparing performance metrics between runs.</SPAN></LI> </UL> <P><SPAN>The key concepts in the MLflow Tracking realm are </SPAN><I><SPAN>runs</SPAN></I><SPAN> and </SPAN><I><SPAN>experiments</SPAN></I><SPAN>. Runs are a detailed record of individual executions of code. Each run captures the associated metadata and resulting output or “artifacts”. An experiment is a collection of runs related to a specific ML task that allows for easier comparison.&nbsp;&nbsp;</SPAN></P> <P><SPAN>For example, for a single command execution or "run" of <FONT face="courier new,courier">python train.py</FONT>, MLflow Tracking would record the artifacts such as the model weights, images, etc.; as well as metadata such as the metrics, parameters, and start/end times. A collection of these runs with different variations (e.g. the ML algorithm) could be grouped as an experiment which allows you to more easily compare and select the best-fitting model.</SPAN></P> <P><SPAN>For more information, you can refer to the </SPAN><A href="https://mlflow.org/docs/latest/tracking.html#concepts" target="_blank" rel="noopener"><SPAN>MLflow documentation</SPAN></A><SPAN>:</SPAN></P> <P><SPAN>On Databricks, matters are made even simpler. By default, a notebook is linked to an experiment with the same name, and the runs of that experiment are all bundled together.</SPAN></P> <H3><SPAN>How do you interact with MLflow Tracking on Databricks?</SPAN></H3> <P><SPAN>MLflow Tracking tightly integrates with Databricks ecosystem. There are multiple touchpoints with MLflow Tracking on the Databricks platform, including ML Runtimes, Notebooks, and the Tracking UI (Experiments).</SPAN></P> <H4><SPAN>ML Runtime</SPAN></H4> <P><SPAN>MLflow is provided out-of-the-box with </SPAN><A href="https://www.databricks.com/product/machine-learning-runtime" target="_blank" rel="noopener"><SPAN>ML Runtimes</SPAN></A><SPAN>. On Standard Runtimes, you can</SPAN><A href="https://pypi.org/project/mlflow/" target="_blank" rel="noopener"><SPAN> install MLflow using PyPi</SPAN></A><SPAN>.&nbsp;</SPAN></P> <H4><SPAN>Notebooks</SPAN></H4> <P><SPAN>MLflow is natively integrated with Databricks Notebooks. That means two things:</SPAN><SPAN><BR /></SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>You can import MLflow and call its various methods using your API of choice (</SPAN><A href="https://mlflow.org/docs/latest/python_api/index.html#python-api" target="_blank" rel="noopener"><SPAN>Python</SPAN></A><SPAN>, </SPAN><A href="https://mlflow.org/docs/latest/rest-api.html#rest-api" target="_blank" rel="noopener"><SPAN>REST</SPAN></A><SPAN>, </SPAN><A href="https://mlflow.org/docs/latest/R-api.html#r-api" target="_blank" rel="noopener"><SPAN>R API</SPAN></A><SPAN>, and </SPAN><A href="https://mlflow.org/docs/latest/java_api/index.html#java-api" target="_blank" rel="noopener"><SPAN>Java API</SPAN></A><SPAN>). The managed Tracking Server will capture and log whatever you ask it to such as parameters, metrics, artifacts, etc. MLflow provides these capabilities out of the box without you needing to set up any infrastructure or tweak any knobs to get this functionality. Additionally, each run preserves the version of the notebook associated with it, making it completely reproducible.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>The </SPAN><A href="https://www.databricks.com/blog/2019/04/30/introducing-mlflow-run-sidebar-in-databricks-notebooks.html" target="_blank" rel="noopener"><SPAN>MLflow Run Sidebar</SPAN></A><SPAN>, located on the right-hand side of your Notebooks, lets you monitor runs directly within your notebooks. It enables you to capture a snapshot of each run, ensuring that you can revisit earlier versions of your code at any time.&nbsp;&nbsp;&nbsp;</SPAN></LI> </UL> <H5><SPAN>Autologging</SPAN></H5> <P><SPAN>As of ML Runtime 10.3, </SPAN><A href="https://docs.databricks.com/en/mlflow/databricks-autologging.html#databricks-autologging" target="_blank" rel="noopener"><SPAN>Autologging</SPAN></A><SPAN> is available on all Databricks notebooks. Autologging means you don’t need to import and call MLflow explicitly to capture your experiments. All you need to do is write your training code using </SPAN><A href="https://docs.databricks.com/en/mlflow/databricks-autologging.html#supported-environments-and-frameworks" target="_blank" rel="noopener"><SPAN>a variety of supported ML libraries</SPAN></A><SPAN> (e.g., sklearn), and pretty much every aspect of the experiment, including metrics, parameters, model signature, artifacts, and dataset, will be logged automatically.&nbsp;</SPAN></P> <P><SPAN>When you attach your notebook to an ML Runtime, Databricks Autologging calls <FONT face="courier new,courier">mlflow.autolog()</FONT> which then uses a predefined set of parameters, metrics, artifacts, etc. to log your experiments. If you want to log any other aspect of your experiment not covered by Autologging (e.g., SHAPLY values), you can </SPAN><A href="https://docs.databricks.com/en/mlflow/databricks-autologging.html#track-additional-content" target="_blank" rel="noopener"><SPAN>manually log them using the MLflow client</SPAN></A><SPAN>.</SPAN></P> <P><SPAN>One important point to note is that when creating a run using the </SPAN><A href="https://mlflow.org/docs/latest/python_api/mlflow.html#module-mlflow" target="_blank" rel="noopener"><SPAN>MLflow fluent API</SPAN></A><SPAN>, autologging isn’t applied automatically. You will have to call <FONT face="courier new,courier">mlflow.autolog()</FONT> manually within the “with” block for it to take effect.<BR /><BR /></SPAN></P> <PRE class="highlight"><CODE>with mlflow.start_run() as run:<BR />&nbsp;&nbsp;mlflow.autolog()<BR />&nbsp;&nbsp;....</CODE><CODE></CODE></PRE> <H4><SPAN>Tracking UI</SPAN></H4> <P><SPAN>Everything you capture in your notebooks using MLflow Tracking is visible under the Experiments tab in Databricks. By default, there is a 1:1 mapping between notebooks and experiments. Therefore in the Tracking UI, you can find all the notebook runs under one experiment. If you click on an experiment, you can further explore the experiment's runs and drill down on relevant information about that experiment.</SPAN><SPAN><BR /></SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="SepidehEb_1-1710925847528.png" style="width: 727px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/6719iA19AE81E20B7FF78/image-dimensions/727x218?v=v2" width="727" height="218" role="button" title="SepidehEb_1-1710925847528.png" alt="SepidehEb_1-1710925847528.png" /></span></P> <P>&nbsp;</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="SepidehEb_2-1710925847508.png" style="width: 726px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/6718iE7E80054DF920DD7/image-dimensions/726x254?v=v2" width="726" height="254" role="button" title="SepidehEb_2-1710925847508.png" alt="SepidehEb_2-1710925847508.png" /></span></P> <H3><SPAN>How can I start using MLflow Tracking?</SPAN></H3> <P><SPAN>Now that you know the essentials of MLflow Tracking, open a notebook and begin writing your ML code. Autologging will automatically capture the training run metadata for you. Navigate to the Experiments tab and look for the experiment with the same name as your notebook. Analyse the run metrics, parameters, etc., embedded within that experiment.&nbsp;</SPAN></P> <H2><SPAN>Use MLflow Model Registry with Unity Catalog</SPAN></H2> <P><SPAN>Another critical component of MLflow is the Model Registry. It provides a centralised repository for managing and organising machine learning models. You can think of Model Registry as a one-stop shop for all the models you want to “keep” for deployment, testing, or any other purposes. Models that are “logged” with MLflow Tracking can be “registered” to the Model Registry.&nbsp;</SPAN></P> <H3><SPAN>What is a model flavour?</SPAN></H3> <P><SPAN>A model flavour refers to how a machine learning model is packaged and stored. When registering a model, you can register it with a “</SPAN><A href="https://mlflow.org/docs/latest/models.html#built-in-model-flavors" target="_blank" rel="noopener"><SPAN>native</SPAN></A><SPAN>” flavour such as sklearn, TensorFlow, etc., or with a Python flavour known as “</SPAN><A href="https://mlflow.org/docs/latest/models.html#python-function-python-function" target="_blank" rel="noopener"><SPAN>pyfunc</SPAN></A><SPAN>.” A native flavour saves the model as a native object (e.g., a pickle file for sklearn flavour). In contrast, the pyfunc flavour stores the models as a generic </SPAN><A href="https://mlflow.org/docs/latest/python_api/mlflow.pyfunc.html#filesystem-format" target="_blank" rel="noopener"><SPAN>filesystem format</SPAN></A><SPAN> defined and supported by MLflow.</SPAN></P> <P><SPAN>If you register a model with a pyfunc flavour, you can only load and use it as a Python object. However, suppose you register a model with a native flavour, Databricks will automatically create an additional Python flavour for you, enabling you to load and use the model both as a native object or a Python one.</SPAN></P> <H3><SPAN>Workspace Model Registry vs Unity Catalog Model Registry</SPAN></H3> <P><SPAN>Before the introduction of </SPAN><A href="https://docs.databricks.com/en/mlflow/models-in-uc.html" target="_blank" rel="noopener"><SPAN>Models in Unity Catalog (UC</SPAN></A><SPAN>), the default scope for the Model Registry home was limited to a workspace. This meant you could only access a model registered to your Model Registry within the boundaries of that specific workspace. If you wanted to move models from one workspace to another (e.g., to promote from a dev workspace to a prod one), you had to go through the extra trouble of using</SPAN><A href="https://docs.databricks.com/en/mlflow/migrate-mlflow-objects.html#copy-mlflow-objects-between-workspaces" target="_blank" rel="noopener"><SPAN> an open source tool</SPAN></A><SPAN> to move objects between workspaces. Another option was to </SPAN><A href="https://docs.databricks.com/en/machine-learning/manage-model-lifecycle/multiple-workspaces.html" target="_blank" rel="noopener"><SPAN>set up a remote Model Registry</SPAN></A><SPAN> and share it between workspaces, which was also laboursome.</SPAN></P> <P><SPAN>Models in UC solve this problem and much more. Now, models live in UC alongside your features and raw data, making lineage tracking seamless. UC takes the concept of Model Registry outside the boundaries of a workspace and holds it at an account level. You can register your models in a schema of a catalog, easily accessed between different workspaces or moved to other catalogs/schemas.&nbsp;</SPAN></P> <P><SPAN>One major change when moving from workspace Model Registry to UC is using “aliases” over “stages” to promote models from development to production. When working with the workspace Model Registry, you would use MLflow model stages (None, Staging, Production, Archived) to progress models from the development stage (typically None) to testing (Staging) and then to production (Production). Once done with a model, you would move it to the “Archived” stage. However, in the UC world, the concept of stages is replaced by model aliases. You have aliases such as “Challenger” and “Champion” for models to be tested and in production, respectively.</SPAN></P> <H3><SPAN>How can I start using the MLflow Model Registry?</SPAN></H3> <P><SPAN>You can use <FONT face="courier new,courier">mlflow.register_model()</FONT> to register any model you have logged during a run. Another option is to pass a “registered_model_name” argument when you are logging your model with <FONT face="courier new,courier">mlflow.&lt;MODEL_FLAVOR&gt;.log_model()</FONT>, and it will both log and register the model at the same time.</SPAN></P> <P><SPAN>To use UC as your model registry, all you need to do is:<BR /><BR /></SPAN></P> <PRE class="highlight"><SPAN>import</SPAN> <SPAN>mlflow<BR /></SPAN><SPAN>mlflow.set_registry_uri(</SPAN><SPAN>"databricks-uc"</SPAN><SPAN>)</SPAN></PRE> <P><SPAN>Then use a 3-level namespace for your model names following the <FONT face="courier new,courier">&lt;CATALOG&gt;.&lt;SCHEMA&gt;.&lt;MODEL_NAME&gt;</FONT> pattern.</SPAN></P> <H2><SPAN>End-to-end ML Experience with MLflow (Beginners version)</SPAN></H2> <P><SPAN>Let’s look at a step-by-step experience using MLflow from experimentation through registering a model to Model Registry.</SPAN></P> <OL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Create a notebook and write your ML code in it (e.g. sklearn)</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Check the Experiments tab and explore your results by navigating to the experiment with the same name as your notebook. If you have multiple runs within the same experiment, you can select and compare their different metrics in the UI.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Decide which model is the “winner.” Then, use mlflow.register_model() to register it to the Model Registry.&nbsp;</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Load the model in a notebook using <FONT face="courier new,courier">mlflow.&lt;MODEL_FLAVOR&gt;.load_model()</FONT> and run inference.</SPAN></LI> </OL> <H2><SPAN>Coming up next!</SPAN></H2> <P><SPAN>Next blog in this series: <A href="https://community.databricks.com/t5/technical-blog/mlops-gym-unity-catalog-setup-for-mlops/ba-p/64943" target="_self">MLOps Gym - Unity Catalog Setup for MLOps</A></SPAN></P> Thu, 13 Jun 2024 15:35:29 GMT SepidehEb 2024-06-13T15:35:29Z https://community.databricks.com/t5/technical-blog/mlops-gym-beginners-guide-to-mlflow/ba-p/62593 <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="image (2).png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/6723i5627B0B8F6FA25C2/image-size/medium?v=v2&amp;px=400" role="button" title="image (2).png" alt="image (2).png" /></span></P> <P> </P> Thu, 13 Jun 2024 15:35:29 GMT https://community.databricks.com/t5/technical-blog/mlops-gym-beginners-guide-to-mlflow/ba-p/62593 SepidehEb 2024-06-13T15:35:29Z https://community.databricks.com/t5/technical-blog/mlops-gym-beginners-guide-to-mlflow/bc-p/64770#M144 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/575">@SepidehEb</a>&nbsp; : Thank you for starting this series. I am new to MLOps and I found this post extremely educative. In fact I am looking for more clarity about promoting models in UC from Dev to staging to Prod through a CI/CD pipeline. Hope the upcoming posts will be insightful in that regard.</P><P>&nbsp;</P><P>Thank you again !</P><P>Nihar</P> Wed, 27 Mar 2024 09:16:44 GMT https://community.databricks.com/t5/technical-blog/mlops-gym-beginners-guide-to-mlflow/bc-p/64770#M144 kumarnihar 2024-03-27T09:16:44Z https://community.databricks.com/t5/technical-blog/mlops-gym-beginners-guide-to-mlflow/bc-p/64918#M146 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/102896">@kumarnihar</a>&nbsp;Thank you for the feedback. We are planning to publish more articles on CICD for MLOps in the Walk phase of the MLOps Gym in the coming months. You can also find a lot of good content in <A href="https://www.databricks.com/resources/ebook/the-big-book-of-mlops?scid=7018Y000001Fi0oQAC&amp;utm_medium=paid+search&amp;utm_source=google&amp;utm_campaign=17152786057&amp;utm_adgroup=139126362889&amp;utm_content=ebook&amp;utm_offer=the-big-book-of-mlops&amp;utm_ad=678212171657&amp;utm_term=the%20big%20book%20of%20mlops&amp;gad_source=1&amp;gclid=Cj0KCQjwqpSwBhClARIsADlZ_TnTbvlb5JK0B524vHGwnQSATabO1u456q8X5VhZjzpyxd28nL6gfIcaAvqJEALw_wcB" target="_self">The Big Book of MLOps</A>.</P> Thu, 28 Mar 2024 14:59:18 GMT https://community.databricks.com/t5/technical-blog/mlops-gym-beginners-guide-to-mlflow/bc-p/64918#M146 SepidehEb 2024-03-28T14:59:18Z