Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Epic - Automated Manifest Testing #650

Closed
13 of 14 tasks
Tracked by #693
jawache opened this issue Apr 21, 2024 · 2 comments
Closed
13 of 14 tasks
Tracked by #693

Epic - Automated Manifest Testing #650

jawache opened this issue Apr 21, 2024 · 2 comments
Labels
EPIC Used to denote an issue that represents a whole epic. Core team only
Milestone

Comments

@jawache
Copy link
Contributor

jawache commented Apr 21, 2024

What: A subset of issues from the broader epic #629 focussing around building our automated test infrastructure based on manifest files.
Why: As a core developer I want a broad set of automated tests based on manifest files so that when a test case fails I have a manifest file I can use to diagnose the root cause.
Context:

  • Manifest files are the core of Impact Framework, they are our user interface, they are the core of everything we do.
  • Manifest files are not configuration files, they are instructions for how to execute an impact calculation and the output manifest files contain the results of that execution.
  • When we report a bug we add the manifest file that triggered that bug, when we create a feature we add a manifest file that should work, if that feature was built.
  • Our goal for the automated testing is to create a test infrastructure which aligns with the fact that manifest files are how we communicate, we want to define tests in terms of manifest files and when tests fail, a developer used the manifest file that failed as the driver for their investigation.
  • To put it in a simple sentence we need a folder full of manifest files that execute different aspects of impact framework, our automated tests run IF against that folder of manifest files and if any of them fail, it reports which one failed.
  • The issues below are a subset of issues from the broader trust epic which would create that infrastructure.

SoW:

Part 1: Updating Existing Tooling

  • 👍🏼 Update IF to dump logs and errors to yaml file #592
    • We need this so we can perform negative testing as well as positive. Currently the output manifest file contains data only if there is NO error. So we can only test when things run correctly. When there is an error the CLI errors out and doesn't generate a manifest file. Instead, if we always generate a manifest file but in the error cast have the manifest file contain the error, then we can have test cases that are manifest files we know will error out.
  • 👍🏼 Update IF to report runtime information to outputs #591
    • if-env needs certain bits of data to be in a manifest file so it can setup an environment that can run it, specifically for automated testing it needs to know the versions and locations of the plugins required so it can know where to install them from.
  • Rename ie to if-run #774

Part 2: New Tooling

Other tasks:

@zanete
Copy link

zanete commented Jun 5, 2024

@jmcook1186 quick check if "Rename ie to if-run (to align with the names of the additional tools being built)" needs to be a separate issue to bring this one to a close?

@jmcook1186
Copy link
Contributor

@zanete I had forgotten about this renaming task. I think it is a good idea to standardize all our commands to have the if- prefix, and it is a very small task, so let's make it a requirement for this epic.

@zanete zanete mentioned this issue Jun 5, 2024
4 tasks
@zanete zanete removed the epic: QA label Jun 5, 2024
@zanete zanete moved this from In Refinement to In Progress in IF Jun 11, 2024
@zanete zanete closed this as completed Jul 8, 2024
@github-project-automation github-project-automation bot moved this from In Progress to Done in IF Jul 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
EPIC Used to denote an issue that represents a whole epic. Core team only
Projects
Status: Done
Development

No branches or pull requests

3 participants