article Using JFrog Artifactory PyPi libraries in Databricks Apps in Technical Blog

article Using JFrog Artifactory PyPi libraries in Databricks Apps in Technical Blog https://community.databricks.com/t5/technical-blog/using-jfrog-artifactory-pypi-libraries-in-databricks-apps/ba-p/128099 <P><SPAN>In today's dynamic application development landscape, it is paramount to be able to leverage your own custom libraries within an App environment. These libraries often contain proprietary logic, critical algorithms, or specific data access patterns that are essential to your application's functionality. Furthermore, using a private repository like </SPAN><A href="https://jfrog.com/artifactory/" target="_blank" rel="noopener"><SPAN>JFrog Artifactory</SPAN></A><SPAN> for these libraries significantly enhances security by isolating your code from public repositories and providing granular access control. This ensures that your valuable intellectual property remains protected.</SPAN></P> <P><SPAN>This blog post guides you on how to install a private PyPI package from JFrog Artifactory within a </SPAN><A href="https://docs.databricks.com/aws/en/dev-tools/databricks-apps/" target="_blank" rel="noopener"><SPAN>Databricks App</SPAN></A><SPAN> (DBApps) environment. </SPAN></P> <H2><SPAN>Adding a private PyPi package to JFrog Artifactory</SPAN></H2> <P><SPAN>For this blog post, we have created a private PyPI package named “custom_calculator” in JFrog Artifactory and showed how that private package can be installed within DBApps. The package custom_calculator implements some simple arithmetic operations:</SPAN></P> <LI-CODE lang="python">""" Basic arithmetic operations module. """ def add(a: float, b: float) -> float: """Add two numbers.""" return a + b def subtract(a: float, b: float) -> float: """Subtract b from a.""" return a - b def multiply(a: float, b: float) -> float: """Multiply two numbers.""" return a * b def divide(a: float, b: float) -> float: """Divide a by b.""" if b == 0: raise ValueError("Cannot divide by zero") return a / b</LI-CODE> <P><SPAN>We won’t focus on how this project will be </SPAN><A href="https://packaging.python.org/en/latest/tutorials/packaging-projects/" target="_blank" rel="noopener"><SPAN>built and packaged</SPAN></A><SPAN>. Any standard python packaging solution should be able to build the project and generate a wheel file encapsulating the custom_calculator package. This package is then uploaded into and indexed inside a Jfrog Artifactory repository. Below is how it appears in the repository:</SPAN></P> <P><SPAN> </SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="uday_satapathy_4-1754936233330.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/18954iCB08BAA589A17583/image-size/large?v=v2&px=999" role="button" title="uday_satapathy_4-1754936233330.png" alt="uday_satapathy_4-1754936233330.png" /></span></P> <H2><SPAN>Configuring the Databricks App</SPAN></H2> <P><SPAN>You can use this </SPAN><A href="https://docs.databricks.com/aws/en/dev-tools/databricks-apps/" target="_blank" rel="noopener"><SPAN>article</SPAN></A><SPAN> to create a Databricks App. The steps we outline below can be used to make the package </SPAN><SPAN>custom_calculator accessible from this App’s environment.</SPAN></P> <H3><STRONG>Grab the PyPI index-URL from JFrog Artifactory:</STRONG></H3> <P><STRONG><BR /></STRONG><SPAN>To connect your Databricks App to your private repository, you first need the specific PyPI index-URL provided by your JFrog Artifactory instance. This URL acts as the entry point for accessing your private packages. Below is the format of the index-url from JFrog:</SPAN></P> <LI-CODE lang="python">https://<username>:<password>@mycompany.jfrog.io/artifactory/api/pypi/pypi-local/simple</LI-CODE> <P><SPAN>You can notice that credentials are included in the provided URL, which are essential for authenticating with JFrog Artifactory and accessing libraries. To maintain security and avoid exposing these credentials in application code, the index-url should be stored as a </SPAN><A href="https://docs.databricks.com/aws/en/security/secrets/" target="_blank" rel="noopener"><SPAN>Databricks Secret</SPAN></A><SPAN>. This approach enables secure credential management and ensures proper referencing within the Databricks environment.</SPAN></P> <H3><STRONG>Put the value of the above index-URL into a Databricks Secret:</STRONG></H3> <P><SPAN>Using </SPAN><A href="https://docs.databricks.com/aws/en/dev-tools/cli/" target="_blank" rel="noopener"><SPAN>Databricks CLI’s</SPAN></A> <A href="https://docs.databricks.com/aws/en/security/secrets/example-secret-workflow" target="_blank" rel="noopener"><SPAN>Secrets</SPAN></A><SPAN> module, you can add a secret named, say, jfrog-pypi-url into a Scope (a container of secrets).</SPAN></P> <LI-CODE lang="python">databricks secrets create-scope my-secret-scope databricks secrets put-secret my-secret-scope jfrog-pypi-url</LI-CODE> <P> </P> <H3><STRONG>Add a pointer to this secret in the Databricks Apps - Resources:</STRONG><STRONG><BR /></STRONG></H3> <P><STRONG><BR /></STRONG><SPAN>Once the secret is created, you need to make it accessible to your application. This is done by adding a pointer to the secret in the Databricks Apps - Resources section. This step links the secret to your app's deployment configuration.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="uday_satapathy_5-1754936233336.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/18956iC6E18D3CD6ECF4E8/image-size/large?v=v2&px=999" role="button" title="uday_satapathy_5-1754936233336.png" alt="uday_satapathy_5-1754936233336.png" /></span></P> <H3> </H3> <H3><STRONG>Set PIP_EXTRA_INDEX_URL inside app.yaml:</STRONG></H3> <P><STRONG><BR /></STRONG><SPAN>The app.yaml file in a Databricks app defines how your app is executed. In your app.yaml file, add a PIP_EXTRA_INDEX_URL environment variable. This variable tells pip to look for packages in your private JFrog Artifactory. In this case, pip first checks the public PyPI and then falls back to your private Artifactory. The value of this variable should be sourced from the 'key' you defined in the previous step, ensuring that the Artifactory URL is dynamically retrieved from the secret.</SPAN></P> <P><SPAN>app.yaml:</SPAN></P> <LI-CODE lang="python">command: [ "flask", "--app", "app.py", "run" ] env: - name: "PIP_EXTRA_INDEX_URL" valueFrom: "external-pypi-url"</LI-CODE> <P> </P> <H3><STRONG>Add the library in </STRONG><STRONG><I>requirements.txt</I></STRONG><STRONG>:</STRONG><STRONG><BR /></STRONG></H3> <P><SPAN>To ensure that your private library is installed during the app's deployment, add its name (e.g., custom_calculator) to your </SPAN><I><SPAN>requirements.txt</SPAN></I><SPAN> file. This file lists all the Python dependencies required by your application.</SPAN></P> <P><SPAN>requirements.txt:</SPAN></P> <LI-CODE lang="python">flask custom_calculator</LI-CODE> <H3> </H3> <H3><STRONG>Import the package in your code</STRONG></H3> <P><SPAN>As a part of demonstrating the successful import of this package, we will display the execution of these arithmetic operations inside a Flask Application, created as a Databricks App. The app.py file for this application looks like this:</SPAN></P> <LI-CODE lang="python">from flask import Flask, render_template_string, request from custom_calculator import operations import logging log = logging.getLogger('werkzeug') log.setLevel(logging.ERROR) app = Flask(__name__) HTML_TEMPLATE = """ <!DOCTYPE html> <html> <head> <title>Custom Calculator Sample Execution</title> </head> <body> <h1>Custom Calculator Sample Execution</h1> <form method="post"> <label for="a">a:</label> <input type="number" step="any" name="a" id="a" value="{{ a }}"> <label for="b">b:</label> <input type="number" step="any" name="b" id="b" value="{{ b }}"> <button type="submit">Calculate</button> </form> {% if results %} <ul> <li>Add: {{ a }} + {{ b }} = {{ results['add'] }}</li> <li>Subtract: {{ a }} - {{ b }} = {{ results['subtract'] }}</li> <li>Multiply: {{ a }} * {{ b }} = {{ results['multiply'] }}</li> <li>Divide: {{ a }} / {{ b }} = {{ results['divide'] }}</li> </ul> {% endif %} </body> </html> """ @app.route("/", methods=["GET", "POST"]) def index(): a = request.form.get("a", 10) b = request.form.get("b", 5) results = None try: a_val = float(a) b_val = float(b) results = { "add": operations.add(a_val, b_val), "subtract": operations.subtract(a_val, b_val), "multiply": operations.multiply(a_val, b_val), } try: results["divide"] = operations.divide(a_val, b_val) except Exception as e: results["divide"] = f"Error: {e}" except Exception: results = None return render_template_string(HTML_TEMPLATE, a=a, b=b, results=results) if __name__ == "__main__": app.run(debug=True)</LI-CODE> <P> </P> <H3><STRONG>Deploy + start the App and verify the library installation in the Environment tab:</STRONG></H3> <P><STRONG><BR /></STRONG><SPAN>After configuring your `app.yaml` and `requirements.txt`, deploy and start your Databricks App. You can then verify the successful installation of your private library by checking the Environment tab in your Databricks App's interface. This provides confirmation that your dependencies were correctly resolved and installed.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="uday_satapathy_6-1754936233340.png" style="width: 795px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/18955iD279E01BDB4C55E1/image-size/large?v=v2&px=999" role="button" title="uday_satapathy_6-1754936233340.png" alt="uday_satapathy_6-1754936233340.png" /></span></P> <P> </P> <P><SPAN>Finally, run the App to verify that the logic defined in the imported package is working correctly.</SPAN></P> <H2><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="uday_satapathy_7-1754936233335.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/18957i26A5308662C7BE40/image-size/large?v=v2&px=999" role="button" title="uday_satapathy_7-1754936233335.png" alt="uday_satapathy_7-1754936233335.png" /></span><SPAN>Summary</SPAN></H2> <P><SPAN>In a nutshell, integrating your private PyPI libraries from JFrog Artifactory into your Databricks Apps environment involves securely storing your Artifactory URL as a secret, referencing that secret in your app's configuration, and specifying your private library in your application's requirements. This process ensures that your custom code is deployed and available within your Databricks application securely and efficiently.</SPAN></P> <P> </P> <P><SPAN>Build </SPAN><A href="https://docs.databricks.com/aws/en/dev-tools/databricks-apps/" target="_blank" rel="noopener"><SPAN>Apps</SPAN></A><SPAN> on Databricks today!</SPAN></P> Mon, 11 Aug 2025 20:36:23 GMT uday_satapathy 2025-08-11T20:36:23Z Using JFrog Artifactory PyPi libraries in Databricks Apps https://community.databricks.com/t5/technical-blog/using-jfrog-artifactory-pypi-libraries-in-databricks-apps/ba-p/128099 <P><SPAN>Installing private PyPI packages from JFrog Artifactory within a Databricks Apps </SPAN><SPAN>environment</SPAN></P> Mon, 11 Aug 2025 20:36:23 GMT https://community.databricks.com/t5/technical-blog/using-jfrog-artifactory-pypi-libraries-in-databricks-apps/ba-p/128099 uday_satapathy 2025-08-11T20:36:23Z