Getting Started with a Template Application

Last updated on 2025-04-04 | Edit this page

Overview

Questions

  • How do I quickly set up a starting point for a NOVA project?
  • What files and directories are included in the NOVA template application?
  • How does poetry manage project dependencies and virtual environments?

Objectives

  • Clone the NOVA template application using copier.
  • Understand the basic project structure created by the template.
  • Identify key files in the project (e.g., pyproject.toml).
  • Install project dependencies using poetry.
  • Deploy the template application to NDIP

Getting Started with a Template Application

As mentioned in the introduction, all code examples in this tutorial are based on a template application. In this episode, we will create this starting point by cloning a template using the copier library. This template provides a basic project structure and pre-configured files that will help us get started quickly with our NOVA project, saving us from setting up everything from scratch.

Callout

The setup section detailed the prerequisites required for the tutorial. One of those prerequisites is copier which will be used to clone a template application. If you've not already insalled copier and other dependencies, please follow the instructions in the Setup section.

Cloning the Template

To clone the template application, run the following command:

BASH 

copier copy https://code.ornl.gov/ndip/project-templates/nova-application-template-tutorial.git nova_tutorial

This command will download the template to a directory called nova_tutorial. Copier will prompt you with a series of questions. Please answer the questions as follows:

  • What is your project name?

    Enter Nova Tutorial

  • What is your Python package name (use Python naming conventions)?

    Press enter to accept the default.

  • Do you want to install Mantid for your project?

    Enter no

  • ** Are you developing a GUI application using MVVM pattern?**

    Enter yes

  • ** Which library will you use?**

    Select Trame

  • **Do you want a template with multiple tabs?

    Enter yes

  • Publish to PyPI?

    Enter no

  • Publish documentation to readthedocs.io?

    Enter no

After answering these questions, copier will clone the template repository and create your project within the nova_tutorial directory.

Callout

If your application requires Mantid, you can enter Yes and Mantid will be added to your dockerfile. However, for local development you will still need to properly set up your conda environment.

Install Project Dependencies

  1. Clone the Template: Follow the instructions in the Getting Started with a Template Application section to clone the NOVA template using copier. This will create a new directory (e.g., nova_tutorial) containing your project files.

  2. Navigate to the Project Directory: Open your terminal and navigate to the newly created project directory:

    BASH 

    cd nova_tutorial

  3. Install Dependencies with Poetry: Use Poetry to install the project dependencies defined in the pyproject.toml file:

    BASH 

    poetry install

    This command will create a virtual environment for your project and install all required libraries, including the NOVA libraries and Trame.

Project Structure

The template creates a basic project structure to help get you started quickly. It includes the following directories and files:

  • nova_tutorial/: The root directory of your project
  • nova_tutorial/src/: Contains your application code
  • nova_tutorial/src/nova_tutorial/: The name of your Python package
  • nova_tutorial/tests/: Contains your application's unit tests.
  • nova_tutorial/README.md: A readme file.

Note: The code provided in the code/episode_2 directory represents a simplified version of the template output, focused on the essential files for this tutorial. The full template, as generated by copier, includes additional configuration files (like Dockerfiles and CI setup) that are not strictly necessary for following the tutorial's core concepts.

In the following sections, we will start adding code to this structure to build our NDIP job submission tool.

Updating the Template

After obtaining the template, you may need to update it. This is usually due to one of the following:

  1. You need to change an answer to a question asked during template setup.
  2. Our team has changed the template and you want to pull in the new content.

In both cases, you can update the template with:

BASH 

copier update

copier will ask you the questions from the initial setup again, and if you change your answers the template will be updated to reflect your new answers. If you dont need to change any answers, then you can run the following to keep all of your existing answers:

BASH 

copier update -A

Callout

copier uses git to resolve conflicts between the template and your changes, so if youve heavily modified the template after the initial setup then you may run into merge conflicts during the update.

Run the Initial Tests

The template comes with a basic test suite using pytest. Navigate to the nova_tutorial directory in your terminal and run the tests using the command pytest. Examine the output. Where are the tests located? What does a successful test look like? Modify the test to intentionally fail. Observe the error message. Remember to revert the changes so that the tests pass again..

  • Where are the tests located? The tests are typically located in the tests/ directory, often mirroring the structure of the src/ directory (e.g., tests/nova_tutorial/test_module.py).
  • What does a successful test look like? A successful test will usually result in output from pytest that indicates all tests have passed (e.g., 100% passed). There will be no error messages. The exact output varies slightly depending on the number of tests and the pytest configuration.
  • Modify the test to intentionally fail: To make a test fail, you can change an assertion to be incorrect. For example, if a test asserts that 1 + 1 == 2, change it to 1 + 1 == 3.
  • Observe the error message: The error message will indicate which assertion failed and provide information about the expected and actual values. For example, you might see something like: AssertionError: assert 2 == 3.

Explore Pre-Commit Hooks

The template includes pre-commit hooks for code formatting and linting.

  • Inspect the Configuration: Open the .pre-commit-config.yaml file. What tools are configured to run? What does each tool do (e.g., black, flake8)?
  • Try It Out: Make a deliberate formatting error in one of the Python files (e.g., add extra spaces, make a line too long). Now, run pre-commit run. Observe how the pre-commit hooks automatically fix the formatting issues. Commit your changes. Pre-commit hooks can also be automatically run upon git commit.
  • What tools are configured to run? Open .pre-commit-config.yaml to see the list. Common tools include:
    • black: Auto-formats Python code to adhere to a consistent style.
    • flake8: Lints Python code, checking for style errors and potential bugs.
    • isort: Sorts Python imports alphabetically and separates them into sections.
    • end-of-file-fixer: Ensures that files end with a newline.
    • trailing-whitespace-fixer: Removes trailing whitespace from lines.
  • Observe how the pre-commit hooks automatically fix the formatting issues: When you run pre-commit run, the configured tools will automatically modify the files to correct formatting errors. The output will show which tools were run and which files were modified. You'll need to git add the modified files before committing.

CI/CD Setup with GitLab CI

The template includes a basic GitLab CI configuration file (.gitlab-ci.yml). While we won't fully execute a CI/CD pipeline in this tutorial step, let's understand its purpose.

  • Examine the Configuration: Open the .gitlab-ci.yml file. This file defines the pipeline. What are the key stages defined in the pipeline (e.g., build, test, deploy)? Identify the jobs that install dependencies, run tests, and perform linting. What triggers the pipeline to run (e.g., pushes, merge requests)?
  • GitLab Runner: GitLab CI/CD uses runners to execute the jobs defined in your .gitlab-ci.yml file. These runners can be configured in various ways. (No action required; this is just an informational point.)
  • Discussion: If you were to push this project to a GitLab repository, what would happen when you create a merge request? How could you use CI/CD to automatically verify the code quality of your project? (No action required; this is a thought exercise.)
  • What are the key stages defined in the pipeline? The stages typically include:
    • build: Installs dependencies and prepares the application for testing.
    • test: Runs the unit tests.
    • lint: Performs code linting and formatting checks.
    • deploy (optional): Deploys the application to a server or environment.
  • Identify the jobs that install dependencies, run tests, and perform linting: Look for job definitions that use commands like pip install, pytest, and flake8 (or similar linting tools).
  • What triggers the pipeline to run? The pipeline is typically triggered by pushes to the repository and the creation of merge requests. This is defined in the .gitlab-ci.yml file using keywords like on: [push, merge_requests].
  • If you were to push this project to a GitLab repository, what would happen when you create a merge request? A pipeline would be automatically triggered. The pipeline would run the jobs defined in .gitlab-ci.yml, such as installing dependencies, running tests, and performing linting. The results of the pipeline would be displayed in the merge request, allowing you to see if the code passes all checks before merging it. This helps ensure code quality and prevents broken code from being merged into the main branch.

Deploying Your Tool to NDIP

Now that we have our template application set up, we need to integrate it with the NDIP platform. The template includes built-in utilities to streamline this process, handling the GitLab repository setup and Galaxy tool XML management.

Initialize Your Project Repository

You can initialize your Git repository and push it to the correct location in the NDIP GitLab:

BASH 

poetry run init-repo

Callout

If prompted for a username and password by GitLab, then please use your three-character ID as the username and the Personal Access Token you set up earlier as the password.

This script will:

  1. Initialize a Git repository (if not already done)
  2. Set up the remote to point to the configured repository URL
  3. Add all project files to the repository
  4. Create an initial commit (if needed)
  5. Push the code to the GitLab repository

Continuous Integration and Container Building

Once your code is pushed to GitLab, the included CI/CD pipeline will automatically build a Docker container for your application. The CI configuration is already set up in the .gitlab-ci.yml file and includes:

  1. Running tests to verify your code works correctly
  2. Building a Docker image containing your application
  3. Pushing the image to the Harbor container registry (at savannah.ornl.gov/ndip/tool-sources/tutorial/YOUR_USERNAME-nova-tutorial)

The Docker image tag is derived from your projects version in pyproject.toml. Each time you update the version and push, a new container will be built automatically.

Tool XML File

The template has already generated a Galaxy tool XML file for your project. You can find this file at:

xml/tool.xml

This file defines how your tool appears and functions within the NDIP platform. It includes:

  • A unique tool ID (now manually configured for the tutorial)
  • The correct container reference pointing to your GitLab repository
  • Command to run your application inside the container
  • Help and description text for users

After the manual changes we made in the previous step, your tool XML will be correctly configured for the tutorial environment.

Pushing the Tool XML to Galaxy Tools Repository

To deploy your tool to the NDIP platform, you need to add the XML file to the galaxy-tools repository. The template includes a utility for this:

BASH 

poetry run deploy-tool

This script will:

  1. Clone the Galaxy tools repository
  2. Copy your tool XML file to the correct location (for the tutorial this is configured as tools/neutrons/tutorials/YOUR_USERNAME-nova-tutorial.xml)
  3. Commit the changes
  4. Push to the prototype branch of the galaxy-tools repository

Once your XML file is pushed to the prototype branch, an automated CI job will deploy your tool to the calvera-test instance. You can then access your tool through the NDIP web interface at https://calvera-test.ornl.gov.

Callout

The tool XML utility has been enhanced to check for the existence of your Docker image before proceeding with the push. This helps prevent deployment errors by ensuring your container has been built first.

Understanding Your Tools Integration

Lets understand the key components that make your tool work in NDIP:

  1. Repository Structure:
    • Your code is hosted at https://code.ornl.gov/ndip/tool-sources/tutorial/YOUR_USERNAME-nova-tutorial
    • The Docker container is built automatically by CI and stored at savannah.ornl.gov/ndip/tool-sources/tutorial/YOUR_USERNAME-nova-tutorial
  2. Tool XML File:
  3. Deployment Process:
    • When you push code to your repository CI builds a new container
    • When you run deploy-tool The utility checks if your container exists and pushes your tool XML to the galaxy-tools prototype branch
    • After XML is merged Your tool appears in the NDIP interface

Callout

In a production environment, when your tool is ready for users, you would create a merge request from the prototype branch to the dev branch. The NDIP team reviews these changes, merges them, and your tool will be deployed to the production instance during the next deployment.

References

Key Points

  • Nova provides a template application to help get started developing your application.
  • Use the copier tool to clone the template application.
  • Poetry is a project management tool used to install dependencies and manage virtual environments.
  • The template application includes everything you need to get started such as basic CI, dockerfile, and tests.
  • Docker containers package your application and all its dependencies for deployment.
  • Galaxy tool XML files define how your tool appears and functions in NDIP.
  • Tools are deployed by adding their XML files to the galaxy-tools repositorys prototype branch.