Move to circleci and other dreams

[springmeyer]

Now that http://circleci.com 2.x supports OS X it is likely a viable alternative to http://travis-ci.org for our CI/binary building needs. My hope is that circleci will be faster and more flexible (e.g. circle workflows).

Travis has served us well, however:

• The OS X builds are often slow or broken due to travis OS X infra outages
• We've seen an increasing amount of network failures on travis

I therefore think we should try to move mason to circle. The steps needed to get there are, at least:

Fundamental first steps

• Enable circleci for this repo
• Create a circle branch that adds a .circleci/config.yml
• Modify the .circleci/config.yml to test the mason core unit tests on both OS X and linux and ensure they work (could steal from Add circleci 2.0 support mapbox-tile-copy#121 and mapbox/mason-js@cbf39d8)
• Compare start times/build times against travis for a few recent builds
• After the mason core unit tests are passing, then start looking into running package builds on circleci
  • Research if circle has an API similar to travis's trigger API such that we can continue allowing developers (with commit to this repo) be able to publish packages by triggering a build locally.
  • Focus first on header-only publishing, since that is platform-independent
  • Then figure out how we can run binary builds for all platforms for precompiled packages. Use a job per platform, disabled by default in the main config.yml, then trigger a build for each based on some per-package config?
    • Start with just Linux, and OS X
    • Use a docker image of ubuntu:precise to ensure forward compatibility of all libs built against libc symbols
  • Update the README.md to document any differences in publishing/auth with circleci.

Gochas include:

• Currently our travis usage depends on a "parent" travis file in the master branch root and a per-package .travis.yml child. The parent sets up the compiler environment and runs the unit tests under normal builds. The per-package child .travis.yml controls what platforms we build a given package for (using travis matrix jobs that override the parent). When we "trigger" builds on travis the per-package config is used and inherits the compiler environment from the main .travis.yml parent in a custom way. Ideally we can do away with the per-package .travis.yml because these have become hard to maintain. I would be nice to come up with a single circleci config that would be used, somehow and a cleaner way to declare what platforms a package should be built for. How this would work and be designed is an open question and depends on what is possible and clean with circleci.
• The travis compiler setup depends on source'ing a bash script. However on circleci each line is an independent shell so source'ing does not change the environment for other commands. So we may want/need to change how the compiler toolchain is set up to make this more easy/robust. Circleci has a special BASH_ENV that can be worked with to persist things like CXX and other environment variables. But it would likely be wise to make this compiler toolchain setup not circleci-specific such that running the mason unit tests locally is still easy and consistent with how to test on circleci, if possible.

Once proven, let's aim higher

Assuming the circleci port proves solid, we could piggyback on this (potentially) large build/publishing change to improve the overall developer experience of creating new packages for mason.

Note: The below ideas could likely be done using travis, but I think moving to circleci is the perfect opportunity to attempt them. In particular circleci workflows might be a great way to ensure the right steps happen during the packaging process, as efficiently as possible.

Critical context on this is that two significant limitations of mason currently are:

1. The PR publishing workflow is unintuative 🙅‍♂️

2. We don't test packages 😱

Below I'll detail both of these issues to give a starting sense of how they might be addressed using circleci + some elbow grease/creative thinking.

Improving the PR workflow

Currently, when creating a new mason package (either a totally new package or a new version of an existing package), mapbox devs start with a new branch/PR. However, the PR defaults to testing the mason core unit tests on travis. So, the package is not actually built. This is both unintuitive and dangerous: it's possible for a new developer to create a new package, see travis is green for their branch, and assume everything works fine when actually their package is broken in numerous ways. Most often they have forgotten to update the gitsha for the source download and they only learn this when they go to publish by trigger'ing a build. Oops. Or the package just plain won't compile yet because they do not know to test a build locally or did not have access to all operating systems to test locally first. To scale mason we'd like more developers being able to package themselves in mason, and feeling better supported by the system in doing so.

So, ideally when a user creates a PR we could:

• detect that the PR relates to a new package
• instead of running the mason core tests we would test that ./mason build <package> <version> works on all target operating systems

This could be a different service eventually via a fancy webhook, but I can imagine this being accomplished in the near term by detecting changed files in the ./scripts directory with git diff --name-only master scripts, parsing out the package names and versions, and forming up a command to run the build for all of them.

Then, the PR would fail if the newly added or edited scripts don't build. This would provide the needed feedback to the dev making the package and publishing would happen after all builds work.

Testing packages before and after publishing

Currently we do not test packages. Yes, its true. Mason started with a philosophy of being minimal since we did not have a lot of bandwidth for writing or maintaining a complex and feature rich package manager. Since then mason has grown in usefulness and uptake amongst mapbox projects.

Currently package builds+publish either succeed when a build is trigger'd or they don't. We have no automated way of knowing - before they a package is published - that something might be wrong.

If we started having PRs actually test the package build then we could use circleci workflows to then test that the packages work after they are built. We could even reject new packages unless they had a certain threshold of tests. Mason core/each package would have some starting framework for tests that is extensible so we could add additional tests over time. Examples that could be supported are:

• A header only library is packaged, but its headers are copied to the wrong location due to a cp -r bug (which behaves slightly differently on OS X vs Linux). Instead of learning the headers are missing when using the package downstream, ideally we could know they are not in the right location before publishing.
• What if a build bug in the package script results in a compile without optimization flags? How could we detect that at build time or at least before publishing?
• Say we learn of some narly crash in a downstream library when a symbol is missing - we could ideally have a way to build a test back into the mason package to assert that symbol is not missing to ensure that problem is truly fixed and never again regresses.

/cc @mapbox/core-tech @kkaefer @jfirebaugh

[jfirebaugh]

Research if circle has an API similiar to travis's trigger API such that we can maintain the back compatibility and flow of that publishing process

Looks like it does: https://circleci.com/docs/2.0/api-job-trigger/