Apply more advanced matching criteria in if-diff

[zanete]

Why: In relation to #637
What:

Update the matching criteria to ignore certain differences between the files, for example ignoring fields in execution section of manifest except status and error. Ensure if-diff conforms to the matching rules defined in this ticket.

Scope of work:

• Implement acceptance criteria
• if-diff outputs conform to the schema defined in the rationale section in Build if-diff core feature #637
• Unit tests includes positive and negative test cases in if repository
• unit tests have 100% coverage and passing
• Incorporate PR feedback

Matching criteria

• report these differences:

  • trees contain different number of nodes (should work the same regardless whether source or target has more nodes)
  • nodes in tree have different names
  • nodes in tree contain non-identical fields and/or values
  • any arrays in tree have unequal lengths and/or non-matching values
  • fields, subfields and values in context are non-identical

• ignore differences in:

  • order of nodes in tree
  • order of fields in context
  • content of execution block EXCEPT status and exception

Acceptance Criteria:

• if-diff implements the above matching criteria

• GIVEN the matching criteria are implemented
  WHEN the following command is run: if-diff --source manifest1.yml --target manifest2.yml
  THEN IF should generate the following mapping of pairs of manifests to outputs:

Scenario 1: trees have different N nodes:
manifest-1.yml:

name: test
description:
tags:
initialize:
  plugins:
    sum:
      method: Sum
      path: "@grnsft/if-plugins"
      global-config:
        input-parameters: ["cpu/energy", "network/energy"]
        output-parameter: "energy"
tree:
  children:
    child:
      pipeline:
        - sum
      config:
        sum:
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001
      outputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.002
          energy: 0.003

manifest2.yml:

name: test
description:
tags:
initialize:
  plugins:
    sum:
      method: Sum
      path: "@grnsft/if-plugins"
      global-config:
        input-parameters: ["cpu/energy", "network/energy"]
        output-parameter: "energy"
tree:
  children:
    child:
      pipeline:
        - sum
      config:
        sum:
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001
      outputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.002
          energy: 0.003
    child-2:
      pipeline:
        - sum
      config:
        sum:
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001
      outputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.002
          energy: 0.003

output:

Files do not match!
tree.children.child1
source: missing
target:  exists

Scenario 2: tree components have different names:

manifest-1.yml:

name: test
description:
tags:
initialize:
  plugins:
    sum:
      method: Sum
      path: "@grnsft/if-plugins"
      global-config:
        input-parameters: ["cpu/energy", "network/energy"]
        output-parameter: "energy"
tree:
  children:
    child:
      pipeline:
        - sum
      config:
        sum:
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001
      outputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.002
          energy: 0.003

manifest2.yml:

name: test
description:
tags:
initialize:
  plugins:
    sum:
      method: Sum
      path: "@grnsft/if-plugins"
      global-config:
        input-parameters: ["cpu/energy", "network/energy"]
        output-parameter: "energy"
tree:
  children:
    child-with-new-name:
      pipeline:
        - sum
      config:
        sum:
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001
      outputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.002
          energy: 0.003

output:

Files do not match!
tree.children.child
source: exists
target:  missing

Scenario 3: fields/values in initialize data do not match (diff vals passed for global config):

manifest-1.yml:

name: test
description:
tags:
initialize:
  plugins:
    sum:
      method: Sum
      path: "@grnsft/if-plugins"
      global-config:
        input-parameters: ["cpu/energy", "network/energy"]
        output-parameter: "energy"
tree:
  children:
    child:
      pipeline:
        - sum
      config:
        sum:
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001
      outputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.002
          energy: 0.003

manifest2.yml:

name: test
description:
tags:
initialize:
  plugins:
    sum:
      method: Sum
      path: "@grnsft/if-plugins"
      global-config:
        input-parameters: ["memory/energy", "network/energy"]
        output-parameter: "energy"
tree:
  children:
    child-with-new-name:
      pipeline:
        - sum
      config:
        sum:
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001
      outputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.002
          energy: 0.003

output:

Files do not match!
initialize.plugins.sum.global-config['input-parameters']
source: ["cpu/energy", "network/energy"]
target:  ["memory/energy", "network/energy"]

Scenario 4: values in execution block error field are different:

manifest-1.yml:

name: test
description:
tags:
initialize:
  plugins:
    sum:
      method:
      path: "@grnsft/if-plugins-dummy"
      global-config:
        input-parameters: ["cpu/energy", "network/energy"]
        output-parameter: "energy"
execution:
  status: fail
  error: 'PluginCredentialError: Initalization param 'method' is missing.'
tree:
  children:
    child:
      pipeline:
        - sum
      config:
        sum:
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001

manifest2.yml:

name: test
description:
tags:
initialize:
  plugins:
    sum:
      method:
      path: "@grnsft/if-plugins-dummy"
      global-config:
        input-parameters: ["cpu/energy", "network/energy"]
        output-parameter: "energy"
execution:
  status: fail
  error: 'PluginCredentialError: Initalization param 'path' is missing.'
tree:
  children:
    child:
      pipeline:
        - sum
      config:
        sum:
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001

output:

Files do not match!
execution.error
source: 'PluginCredentialError: Initalization param 'method' is missing.'
target:  'PluginCredentialError: Initalization param 'path' is missing.'

**Scenario 5: execution info other than status and error are different **

manifest-1.yml:

name: test
description:
tags:
initialize:
  plugins:
    sum:
      method:
      path: "@grnsft/if-plugins-dummy"
      global-config:
        input-parameters: ["cpu/energy", "network/energy"]
        output-parameter: "energy"
execution:
  status: success
  command: ie --manifest examples/basic.yml
  environment:
    if-version: v0.3.2
    os: ubuntu
    os-version: 22.04.6
    node-version: v21.4.0
    date-time: 2023-12-12T00:00:00.000Z (UTC)
    dependencies: 
      - '@babel/[email protected]'
      - ...
tree:
  children:
    child:
      pipeline:
        - sum
      config:
        sum:
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001

manifest2.yml:

name: test
description:
tags:
initialize:
  plugins:
    sum:
      method:
      path: "@grnsft/if-plugins-dummy"
      global-config:
        input-parameters: ["cpu/energy", "network/energy"]
        output-parameter: "energy"
execution:
  status: success
  command: ie --manifest examples/basic.yml
  environment:
    if-version: v0.3.2
    os: Win32
    os-version: 14.5.4
    node-version: v21.2.0
    date-time: 2022-10-10T00:00:00.000Z (UTC)
    dependencies: 
      - '@babel/[email protected]'
      - ...
tree:
  children:
    child:
      pipeline:
        - sum
      config:
        sum:
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001

output:

Files match!

Scenario 6: nodes are in different order (position of server and database reversed) :

manifest-1.yml:

name: test
description:
tags:
initialize:
  plugins:
    sum:
      method: Sum
      path: "@grnsft/if-plugins"
      global-config:
        input-parameters: ["cpu/energy", "network/energy"]
        output-parameter: "energy"
tree:
  children:
    server:
      pipeline:
        - sum
      config:
        sum:
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001
      outputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.002
          energy: 0.003
    database:
      pipeline:
        - sum
      config:
        sum:
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001
      outputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.002
          energy: 0.003

manifest2.yml:

name: test
description:
tags:
initialize:
  plugins:
    sum:
      method: Sum
      path: "@grnsft/if-plugins"
      global-config:
        input-parameters: ["cpu/energy", "network/energy"]
        output-parameter: "energy"
tree:
  children:
    database:
      pipeline:
        - sum
      config:
        sum:
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001
      outputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.002
          energy: 0.003
    server:
      pipeline:
        - sum
      config:
        sum:
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001
      outputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.002
          energy: 0.003

output:

Files match!

[zanete]

@jmcook1186 please sense-check the SoW and AC 🙏

[narekhovhannisyan]

• Scenario 1
• Scenario 2
• Scenario 3
• Scenario 4
• Scenario 5
• Scenario 6

[zanete]

PR that needs to be reviewed / merged is Green-Software-Foundation/if-docs#74 - @narekhovhannisyan pls take a look again 🙏