Release multiple projeccts from same git repo

[ieugen]

Hi,

I'm currently using the old plugin, gradle-git, but considering the switch to this one.
I am wondering if reckon can handle multi project releases from the same git repo?

Our use case is as follows:

We have some gradle modules that use GRPC and we generate code from the proto files. Let's call this project report-generator
We have the proto files in the git repository where we have the server module of our report-generator. We implement the grpc client of this report-generator service in other projects that live in other git repositories.

We use gradle git for releasing the server module of report-generator and this creates a different version of our GRPC API artifact.

We would like to be able to release our grpc module from report-generor service only when there are changes to it and not every time.

Moving this code into another git repo will make things harder to track and manage (submodules is an option but adds it's own issues).

Is there a way to have our report-generator grpc api as an independent project inside the report-generaor git repository and handle releases via separate tags?

Something like:

• v-x.y.z tag format to manage release numbering for the main project report-generator
• reports-api-a.b.c tag format to manage release numbering for report-generator grpc api

What other options do I have if that is not possible?

Thanks,

On release we publish our artifacts to Sonatype nexus and we are

[ajoberstar]

Currently, reckon does not provide a way to customize this. It assumes one repo constitutes a releasable whole (which may produce multiple artifacts/publications).

Even with separate Gradle projects within that one repo, it would still be reading from and writing to the same naming scheme of tags, so I don't believe there's a workaround. Submodules might work, though that runs afoul of JGit (and thus reckon) limitations with their support. That's caused issues in grgit, not sure if it will for the use case here.

Overall, I would view this use case as out of scope, as one of the interests of reckon was to be more narrow and opinionated to keep the code and usage more manageable. That said, if you think there are relatively small changes that could be made to support this, I'm open to discussing that more.

[ieugen]

It assumes one repo constitutes a releasable whole (which may produce multiple artifacts/publications).

What I believe I need is a was to produce multiple publications and some of those publications should have different version sequences.

I believe that should be achievable by supplying reckon/gradle-git with a tag parsing strategy for the modules I would like to have a different version sequence. That way, when determining the version to use for projectA, reckon will only check a subset of tags, that start with projectA.

Like I said in my previous post:
One gradle module could look for tags starting with projectA-<version> and another could look for projectB-<version> . Both should obey the versioning rules previously mentioned.

I think this should work. Am I missing something?

[ajoberstar]

Sorry, forgot to come back to this.

It assumes one repo constitutes a releasable whole (which may produce multiple artifacts/publications).

Just to clarify, meant a "releasable whole that uses a single version number".

One gradle module could look for tags starting with projectA- and another could look for projectB- . Both should obey the versioning rules previously mentioned.

Yeah, so part of it would be allowing custom functions to parse/create tag names. Another challenge is that the plugin (right now) applies to the root project only and the version is used across all projects in the build. This is the core use case, so I wouldn't want the version to recalculate for each project when they're going to have the same result.

[ieugen]

Yeah, so part of it would be allowing custom functions to parse/create tag names. Another challenge is that the plugin (right now) applies to the root project only and the version is used across all projects in the build. This is the core use case, so I wouldn't want the version to recalculate for each project when they're going to have the same result.

I understand know that this could prove tricky. I see some solutions for this problem:

1. Make the modules not part of the same gradle multi-project: they share git but not the same build config. This is kind of a hack, but might work. With this approach we have two independent (from a build configuration POV) projects sharing the same git repo.

2. The second, more flexible approach is to define the parsing/creating function based on the module name a.k.a project.name [1]. I don't know how gradle internal work, but I imagine we could have the code that computes the version run for every project and pass the project name to the it. Based on that name we could have another strategy or resolving tags and determining the proper version to use.

That would mean the plugin will need to be able to iterate over all projects and be able to change the version for each project [2].

It looks like this is possible:

project.ext.version = "foo"

What do you think?

[1] https://docs.gradle.org/current/dsl/org.gradle.api.Project.html
[2] https://docs.gradle.org/current/dsl/org.gradle.api.Project.html

[ajoberstar]

Seems like first solution would just need the tag parser/creator functions and no other changes.

Second solution I'm not seeing how it would let the default case (use the same version for all projects) would work without recalculating it for each one.

[lwasyl]

Another challenge is that the plugin (right now) applies to the root project only and the version is used across all projects in the build. This is the core use case, so I wouldn't want the version to recalculate for each project when they're going to have the same result.

What's the reason to apply the plugin only to the root project? It seems like it's an artificial limitation created due to the fact that in current state it doesn't make sense to have the plugin applied to child projects for the reasons you mentioned, but it wouldn't inherently break the build or the plugin. Thus this limitation could be lifted, if only there was some strategy for the plugin to determine version for the specific project in which it's applied.

Seems to me like e.g. prefixing the tags with fixed project prefix would allow the plugin to use only relevant subset of git tags to determine versions, without more changes. Unless, of course, there's something I'm missing.

The you can only apply reckon to the root project limitation can still be present if e.g. there are no project-specific tag prefixes specified. And everything would be backwards compatible.

[ajoberstar]

@lwasyl The main reason is to only infer the version once and reuse that work across all projects. Beyond that, it is artificial (in line with the intent of making a more constrained, opinionated plugin this time around).

Is the proposal to make something where you have some kind of toggle that puts it into "project-specific" mode? Then all projects use the project name as a prefix when parsing/creating tags? Or am I misunderstanding?

[ieugen]

Hi,

I think something like this could be like following. DSL might not make sense :

reckon {
    normal = scopeFromProp()
    preRelease = stageFromProp('dev', 'rc', 'final')

    subProjectConfig(":my-sub-project") {
        tagPrefix = 'sub-prj-tag'
    }
}

Reckon can compute the configurations centrally during configuration phase , one time and then configure all sub projects with specific versions as per configurations.

I do believe that this change implies a default configuration but that is not different than what is happening right now.

[1] https://docs.gradle.org/current/userguide/build_lifecycle.html#sec:build_phases

[ajoberstar]

I do appreciate the interest in the project and the willingness to discuss and drive changes. I just want to be upfront that I'm still skeptical of the value of this feature.

To just allow multiple projects to configure reckon differently, I think you'd need two changes:

• Remove validation that the plugin was applied to the root project
• Leave the logic the applies to allprojects (which is current project and its subprojects). This would allow you to apply to root and get current behavior. Or apply to each subproject and get different configs for each.
• Add validation to prevent it from being applied to a project that was already configured by a parent project. (Just for sanity, so you don't have config blown away.) This would mean that all releasable projects either need to use the same version or all be subprojects.

Handling the tag names, you need tag parsing and tag "rendering" customization. There's a hook for parsing in GitInventorySupplier, but it's not exposed in the plugin (except for the fact that you can specify the inventory supplier yourself). The remaining issue is that core doesn't handle creating tags, so you could still change the tag being created within the Gradle plugin, but it's not quite ideal.

I'm not quite sure what to advise you two (or anyone else interested in this). If you want to pursue this being added into reckon, the hurdles will be 1) someone volunteering to write it (this feature's not a priority for me) and 2) finding an implementation that keeps things pretty clean internally (i.e. one I'd be comfortable merging in and maintaining). Another option is to use reckon-core directly either in your build file or in a separate plugin.

[ieugen]

Hi @ajoberstar,

Thank you for the insight into how gradle internals work. I was not familiar with all of these and now it seems that the functionality is not as simple as I initially thought.
The other approach is to use more git repositories and sub-modules to handle my use case.
We have just migrated our git hosting and we have unlimited repos so we will go via this route as an alternative.

At some point I hope to be able to come back to this or maybe someone else will pick it. Hopefully the discussion will help out.

Regards,

[ieugen]

I'm coming back to this with more info and insight. I believe more and more that it is a usefull thing to have since I have noticed projects that need this functionality in practice.

I'm swamped with work and can't tackle this but hopefully that time will come or someone else wil beat me to it. In the mean time this can serve as use case documentation.

1. Apache Sling - releases OSGi bundles and uses Maven as a build. Since each bundle has it's own version, they provide 1 git repo / module (bundle). Now when developing you usually need to change code in multiple projects. The solution they use is to use git-repo [1] to automate downloading all of the git repositories locally [2] .

This works but needs another tool to work with. Having multi version support in same git repo will eliminate this need.

2. Lerna . This is a tool from Javascript world that allows you to publish multiple npm packages from same git repo. Now, I believe that npm format and publishing is more limitted than maven ( no support for classifiers, extensions -> you can publish only 1 artifact / package). So they found this approach (bit is another [4] ) to make it so they can publish multiple packages from same git repo.

Of course both approaches use hard-coded versions inside scripts.

[1] https://gerrit.googlesource.com/git-repo/
[2] https://github.com/apache/sling-aggregator
[3] https://lernajs.io/
[4] https://github.com/teambit/bit

[JavierSegoviaCordoba]

@ieugen did you find a workaround?

[ieugen]

No. I am not using reckon for some time now.