Problem
Requiring the committer to specify a unique, meaningful version identifier for every merge to a high traffic repository with a lengthy merge process results in painful merge conflicts and delays in merging otherwise approved code.
Example scenario
I have written a code change that has been approved and is ready to merge into our monorepo. I do a final merge of the trunk into my branch. I decide this code change is a patch according to semantic versioning (semver) and bump the version accordingly. When I go to merge the code, I see that there are two people already trying to merge their code. I do not know if their merges will succeed, so I do not know what my version number will ultimately be. I attempt to merge without changing my proposed version. One of their merges succeeds and my version number is now wrong, so I have to bump it and try again. Repeat until the merge is successful.
Potential Solutions
Reduce traffic to the repository
More people generally commit to repositories when the repositories contain more code. Splitting a repository into multiple repositories often means splitting the traffic between those repositories. The split can be part of an effort to extract library code from a repository or to migrate from a monorepo to microservice architecture. This approach does not solve the underlying problem, but is likely to decrease the severity of its impact on the development cycle.
Related: Version components of the repository separately
A similar approach would be to keep everything in the same repository, but create separately versioned components within that repository. The main difference between this solution and the above is in day-to-day repository management for developers and in the details of the deployment process.
Reduce time between request to merge and merge
In the example, the uncertainty around the version number at the time of your merge is created by the merge queue, in which your merge has to wait until the merge jobs ahead of it in line complete before it can merge. If a merge happened immediately after request, there would effectively be no queue, so the version at merge request time would match the version at merge time. Right now, our merge job runs tests to ensure that the code about to merge does not break anything. Removing these tests is not an option, but moving them to run on push and/or post-merge are viable alternatives.
Running tests on push is a convenient way of knowing whether your changes are correct, but runs the risk of someone merging a change that breaks your change in between your push and your merge. This is particularly likely to occur in high traffic target branches. It is also potentially more expensive than running tests pre-merge because there may be more pushes than attempts to merge. Running tests post-merge does test the final version of the code, but there is not an obvious best course of action if the tests fail. An automatic revert may not be possible due to changes merged during the post-merge tests.
Note that this approach still is affected by merge conflicts due to editing the line storing the version, which will require merging trunk into your branch immediately prior to merge. While this is a best practice regardless, in my experience, people will still complain about the burden this causes and will find this solution insufficient because of that.
Remove meaningful constraint on version identifiers
If version numbers do not have to be meaningful, it would be easy for a developer to generate a unique one using, e.g., a random number generator. This could still cause a merge conflict on every merge if the version is specified in a single place in the repository, because everyone would be editing that line. Therefore, for this approach to be useful, the version would have to be specified in a different way, e.g. as the name of a new file in a versions directory. The problem with this approach is that version identifiers are intended to describe something about the code they refer to. A versioning scheme without that property is inferior to one with that property.
Related: Remove relationship between version identifiers
Instead of having the version identifiers be completely random, let’s instead have them match the branch name they were written on, or be some other short description of the code. This adds meaning to version identifiers making them more useful internally. If we use the branch name, then the committer might not even need to specify a version number; it could be added automatically, which means that it could be a single line of code version bump instead of the file name trick. However, with this versioning scheme, it is still difficult to compare two arbitrary version identifiers to understand the differences between them.
Related: Use merge job identifier as version identifier
One common variant on this that does retain some degree of the relationship between version identifiers is to use the merge job identifier (e.g., Jenkins build number) as the version identifier. This runs into problems if the Jenkins job ever changes because then the numbering would restart, unless we are careful to set the initial build number and have a non-overlapping transition. Another similar approach would be to use a merge job timestamp as a version identifier.
Remove unique version identifier for every merge constraint
If most merges did not need to update the version identifier, then we could avoid the version bumping problem entirely. Many repositories use a system where they only bump the version for a public deploy/release, not for intermediate versions. However, we cannot do this because we would like to be able to deploy each version to our staging environment, integration, for validation and manual testing. In practice, we do not automatically deploy each version to integration, but we do support manually deploying arbitrary versions to integration because we trust developers to identify which builds should be deployed/tested at what cadence.
Related: Use a postfix version identifier for non-released versions
Under this system, we would only bump the main version identifier (major/minor/patch for semver) for public deploys/releases and would use a postfix notation to identify internal-only (dev) versions. These internal-only version identifiers could be based on automatically generated information such as git hash or timestamp. Under this system, we would have to add a separate mechanism for incrementing the main version as part of the deploy/release process, including a mechanism for knowing what the new version identifier should be, since this is no longer provided per-commit. Mechanisms for doing so would likely track the version outside of the codebase, which would have implications for our code and processes.
Separate the committer’s input from the generation of the version number
If we are committed to using semver, then it would be best to develop a system where the committer specifies the change type (major, minor, patch) and then the merge job automatically uses that information to generate the version number.
Specify change type in branch name
The biggest problem with this approach is that the branch name is specified at the beginning of the code review process and the change type may change as a result of code review comments, which means that the branch name would be inaccurate. It is possible to get around this by pushing the change with a new branch name, but there is no clear way to associate the review on the old branch with the change on the new branch.
Specify change type in merge request
Some merge request systems are more configurable than others. We use a webhook triggered by a comment in the Github pull request to trigger a merge job and it turns out that that system can accept arguments. Therefore, we can specify the change type as e.g. “ship it patch”.
Conclusion
We decided to implement “Specify change type in merge request” for a number of reasons. Reducing traffic to the repository by splitting apart versioned components is a difficult task that would reduce the symptoms but not solve the underlying problem. Enforcing repository correctness on merge is more valuable than reducing the duration of the merge job, especially for medical software. Similarly, the ability to deploy arbitrary versions to the integration environment for testing means that having each merge be versioned is valuable. All of 1) removing/minimizing the relationship between version identifiers, 2) adding a postfix internal version identifier, and 3) specifying the change type in the merge request are viable options. However, with 1 we lose a small amount of information, 2 could potentially require some major workflow and automation changes, and 3 was fairly straightforward to implement, so 3 seemed preferable.
