Epic - Automated Manifest Testing

[jawache]

What: A subset of issues from the broader epic #629 focussing around building our automated test infrastructure based on manifest files.
Why: As a core developer I want a broad set of automated tests based on manifest files so that when a test case fails I have a manifest file I can use to diagnose the root cause.
Context:

• Manifest files are the core of Impact Framework, they are our user interface, they are the core of everything we do.
• Manifest files are not configuration files, they are instructions for how to execute an impact calculation and the output manifest files contain the results of that execution.
• When we report a bug we add the manifest file that triggered that bug, when we create a feature we add a manifest file that should work, if that feature was built.
• Our goal for the automated testing is to create a test infrastructure which aligns with the fact that manifest files are how we communicate, we want to define tests in terms of manifest files and when tests fail, a developer used the manifest file that failed as the driver for their investigation.
• To put it in a simple sentence we need a folder full of manifest files that execute different aspects of impact framework, our automated tests run IF against that folder of manifest files and if any of them fail, it reports which one failed.
• The issues below are a subset of issues from the broader trust epic which would create that infrastructure.

SoW:

Part 1: Updating Existing Tooling

• 👍🏼 Update IF to dump logs and errors to yaml file #592
  • We need this so we can perform negative testing as well as positive. Currently the output manifest file contains data only if there is NO error. So we can only test when things run correctly. When there is an error the CLI errors out and doesn't generate a manifest file. Instead, if we always generate a manifest file but in the error cast have the manifest file contain the error, then we can have test cases that are manifest files we know will error out.
• 👍🏼 Update IF to report runtime information to outputs #591
  • if-env needs certain bits of data to be in a manifest file so it can setup an environment that can run it, specifically for automated testing it needs to know the versions and locations of the plugins required so it can know where to install them from.
• Rename ie to if-run #774

Part 2: New Tooling

• 👍🏽 Create a folder of manifest files that broadly tests IF across a range of test cases positive and negative. #615
  • The above folder of manifest files form the core of our testing. They contain manifest files that must always work as expected (they are used in our documentation as examples), the contain manifest files which were created to test new functionality as well as manifest files that triggered bugs (which do not exist anymore since we fixed them). This folder contain our unit, integration and regression tests.
• The next issues have to be build in order
• Build if-env #636
• Build if-diff core feature #637
• Build if-check #638
• 👍🏼 Set up CI/CD to trigger automated tests for PRs #621

Other tasks:

• Verify runtime information with global run #695
• Output file is not generated if manifest is invalid #701
• Exhaust error is not handled  #698
• Apply more advanced matching criteria in if-diff #706
• Add docs for if-diff #707
• Add emojis to if-diff output #745

[zanete]

@jmcook1186 quick check if "Rename ie to if-run (to align with the names of the additional tools being built)" needs to be a separate issue to bring this one to a close?

[jmcook1186]

@zanete I had forgotten about this renaming task. I think it is a good idea to standardize all our commands to have the if- prefix, and it is a very small task, so let's make it a requirement for this epic.