Mono-Repo Opportunities

I spent the last couple of weeks doing a deep dive on mono-repos. Following are my thoughts on possible opportunities in the area.

None of these opportunities are mutually exclusive, and all of them could be valuable at least to some of our users. The document is split into a few high level directions:

1. Support the scale of giant mono-repos (GMRs) in order to better cater to the enterprise companies that already run them, and those thinking of making the switch.

   • 1-1. Support virtualization / GVFS
   • 1-2. Explore cloud-based workflows
   • 1-3. Explore “sliced” workflows
   • 1-4. Explore scoped communication

2. Extract the benefits of mono-repos and apply them to multi-repo projects in order to stem the rise of the GMR model, which doesn’t play well with Git or GitHub.

   • 2-1. Make it easier to refactor across multiple repos
   • 2-2. Make it safer to change code that affects multiple repos
   • 2-3. Make it easier for teams to plan across multiple repos
   • 2-4. Help developers avoid duplicated effort
   • 2-5. Help leaders coordinate the big picture
   • 2-6. Enable trunk-based development (TBD)

3. Only semi-related to the previous two directions: Support mono-trees such as the Linux kernel project (or, similarly, extract its benefits and apply them to the GitHub flow).

   • 3-1. Enable the mono-tree model
   • 3-2. Reduce the need for mono-trees

1. Support the scale of giant mono-repos (GMRs)

The first general direction is fairly straightforward: support mono-repos as they exist today.

To be clear, GitHub does support mono-repos. It only becomes an issue when a mono-repo crosses a certain threshold in size. Some of these mono-repos are huge. Microsoft’s repo is over 350 million files and over 270GB in size. Google’s repo is over a billion files and over 86TB in size.

As repos grow to these sizes, Git has more trouble performing basic functions such as status, which can take over ten minutes. checkout can take over 3 hours, and clone can take 12+ hours, if it even works at all. As such, the main issue we need to figure out is not how to support a new workflow, but rather how to deal with scale.

Considering a good deal of tech leaders have adopted GMRs, we should expect many more large companies and projects will follow suit. We should go down this route if we want to keep those companies on GitHub.

Links

• @bryancross’s high-level overview of GMRs and TBD
• GitHub customers who want support for GMRs
• Article: “Why Google Billions of Lines of Code in a Single Repository”

1-1. Support Virtualization / GVFS

Microsoft, Google, Facebook, and Twitter all use mono-repos, and they’ve all created internal tools and processes to work with version control at scale. Microsoft is the first player to open-source its tooling: GVFS (Git Virtual File System).

GVFS virtualizes the entire repo on your file system, which in this case means that the files aren’t really there until you need them. When you or your code try to access a file, it’s downloaded on the spot. What this does is substantially reduce the time it takes to run git commands (if at the cost of some download-time delay as you’re trying to access new files).

To implement this, we would need to implement a GVFS server component that can intelligently send only the right objects as a response to clones and fetches. For users to use this now, they would have to run a slightly modified version of Git and install a new Windows driver.

It’s important to note that GVFS is new, having only been released earlier this year. It only works on Windows (it’s unclear that it’ll ever work on OS X) and still has a lot of bugs to iron out. We can’t assume that it will win over other yet-to-be-released tools for supporting mono-repos. That said, there is some indication that GVFS is being adopted, such as Tower recently adding GVFS support.

GitHub supporting GVFS would considerably increase its (and Git’s, and GitHub’s) chances of becoming the de-facto method of supporting GMRs.

Links

• Customers who want support for GVFS
• 8/24 Summary of our meeting with Microsoft
• 9/26 GVFS architecture review

1-2. Explore cloud-based development workflows

Google’s mono-repo is 1TB. It isn’t feasible to work on it locally. They use their own web-based IDE and builds happen exclusively in the cloud. With the cloud, you don’t have the same limitations on storage or compute resources.

GitHub has the resources (community, adoption) to build a future where any developer can efficiently collaborate on the largest, most complex codebases using a tiny Chromebook and have it feel like local development.

To make this possible, we’d need to make cloud development as appealing as local development, by building things such as:

• A personal cloud staging area that updates as you “save” files online.
  • This should function the way your local development area would.
• A high-quality, integrated code editor on the browser.
• A way to access the repo via local software.
  • For the many developers who’ll never give up vim.
  • For the designers who need to edit binary graphic files.
  • Transmit is a good example of software that lets you make remote edits via local software seamlessly.
• A server-side implementation of git to run things like status for you online.
  • This would have to include scale optimizations similar to GVFS but the nice thing is that we won’t be as beholden to git when it comes to implementing these optimizations, since they only ever run on our servers.
  • That means we also have access to more computing power that could make GMRs run faster. The flip side, of course, is that we’d be using up a whoooole lot of compute which would undoubtedly change our infrastructure economics.

This would be an incredibly technical lift. I have no idea what’d go into building an easy-to-use on-demand “local” environment on the cloud, for instance, but it’s a powerful vision.

The great thing about this direction is how well it fits into our existing plans:

• Our upcoming focus on deployments
  • By shifting existing workflows to the cloud, we could potentially make the transition from “local” to staging to production much more seamless.
• Our current drive to improve the in-browser code editor so it’s no longer a second-class citizen.
  • Add multi-file functionality.
  • Add more sophisticated “local” changes.
  • Implement real-time on CodeMirror so developers can collaborate in the browser.
  • We’ve even discussed a full port of Atom into github.com.

1-3. Explore “sliced” workflows and build tools

Compiling at these sizes is unmanageable and even just cloning these repos can take a frustrating amount of time. Twitter lets their developers torrent the initial repo snapshot instead. At Microsoft, they pair GVFS with a downloadable binary build, so that their developers only ever need to touch and build the files relevant to them. From a GVFS blog post:

In a repo that is this large, no developer builds the entire source tree. Instead, they typically download the build outputs from the most recent official build, and only build a small portion of the sources related to the area they are modifying.

Furthermore, mono-repos don’t often house a single, cohesive app, but rather multiple apps with a web of shared dependencies. As such, you don’t often need to build (or even have access to) the entire codebase to see the results of what you’re working on. Google’s Blaze build tool (open sourced as Bazel) pairs with its version control system (Piper) and lets developers download only the files that their project requires, including all their dependencies.

The problem with Git—unlike Google’s Piper—is that it needs access to the entire repo in order to function properly—even a sparse checkout clones the entire repo before ignoring it. A build tool won’t solve that.

If an engineer only needs to touch the files relevant to them, we could build a system on our end that treats only these files and their dependencies as dynamic, complete, clone-able repo. This “sliced” repo could come with all the relevant commits and omit the others.

Users could work with these repos locally just like they would any other repo. With this approach, there’d be no need to make changes to Git, the git server, or the underlying file system on developers’ machines as GVFS does. We would just need to:

• Be smart about slicing repos up dynamically on our end.
  • We could do this in conjunction with a build tool like Bazel—take a build file and supply a sub-repo based on that.
  • Then these “scoped clones” could work the way any git repo would.
• Be smart about interweaving these commits with the overarching repo’s commits when people push code to GitHub.
• Provide a way to download the rest of the pre-built repo easily if people need it.

This idea is similar to git submodules (which no one seems to like), except there’s an abstracted super-repo rather than a sub-repo. It makes it manageable to work with a GMRs but forgoes a lot of a mono-repo’s inherent benefits. It’s fuzzy and only really the kernel of an idea, but I figured I’d include it in case it sparks some conversation.

1-4. Explore scoped communication

This kind of scale requires more tooling around communication. The same model of repo-wide issues/PRs breaks down at these large sizes, where every developers seeing every issue no longer makes sense.

To make working with monolithic repos more manageable, we need to increase the signal-noise ratio for developers.

Opportunities:

• Build a system that automatically scopes communication.
  • Add an automatic “scope” filter. This could use labels or projects, but an “area” or “maintainer” field might be more fitting. It could potentially be an evolution of Code Owners.
  • Scope PRs, Issues, and Branches based on edits/references to areas, directories, files, or even lines of code.
• Hide irrelevant objects within the UI
  • Display only PR/issues that are relevant to a developer’s owned areas.
  • Display only the branches that touch those areas.
• Design a better triaging system.
  • Let users set a default filter view. We have powerful filters on our PR/Issue pages but the page defaults to show everything that’s open.
  • Let users more clearly switch scopes.

2. Replace mono-repos with better workflows

The second general direction sidesteps supporting GMRs by reducing the need for them.

Mono-repos workflows have tempting benefits—they make it much easier for developers to work holistically across a company’s entire codebase.

What if, instead of spending time supporting the mono-repo workflow, we extract its benefits and apply them to the GitHub flow? This could stem the rise of mono-repos by reducing their relative benefits.

If we’re to compete with a mono-repo workflow, we need to make it just as easy for developers to work holistically with multiple repos as it would be with a single repo. Following are a number of areas we could explore to do just that. None of them are mutually exclusive, and we could choose to work on several.

The main advantage here is that many of these improvements can stand on their own and benefit many development workflows, even if they don’t yet have a need for mono-repos. They can also serve to further differentiate GitHub in the market.

2-1. Make it easier to refactor across multiple repos

With a mono-repo, it’s easy to update or refactor something across a company’s entire codebase with a single PR. If you want to make a similar overarching change on multiple repositories, you need to create multiple, distinct PRs and coordinate their release.

As you’re developing, seeing how changes to one repo might affect another connected repo also becomes more complicated. For example, we’ve invested in tooling for Primer (our css style guide) that lets you publish a temporary version, which you can then use on github/github. Even with this tooling in place, making a simple change can require multiple cycles of making changes and republishing. In contrast, if we were using a mono-repo, changes to Primer would take effect immediately on github/github without any extra work.

Having to coordinate updates this way can discourage engineers from making valuable changes and can increase technical debt. Making this process simpler is one of the biggest benefits of mono-repos. Finding a way to replicate this ease-of-use for multiple repos would go a long way to dissuade people from switching over to the mono-repo model.

Opportunities:

• Multi-repo PRs which can include branches/commits from several repos.
• A “current state” build that uses the current state of multiple local or remote repos in one PR vs. the specific published versions. This could save developers time as they’ll no longer have to formally update packages in order to see their effects.

For their GMR, Google invested in a tool called Rosie. It has sophisticated pattern matching, which helps developers search and replace existing code with new code. For instance, they may want to search for all instances of a function across the entire codebase, and add an argument to them. Rosie also automates splitting the changes up into multiple “patches,” submitting them for testing and code reviews, and committing/merging them when they pass.

Opportunities

• A multi-repo code refactoring tool similar to Rosie
  • This could follow a pattern similar to our conflict resolution tool.
  • It could empower developers to make breaking changes to dependencies when they need to, thus avoiding technical debt.
  • It could interface w/ our existing work on Bumblebee’s dependency graph.
    • Maybe this functions differently when dependents are in the same org?
• Improve org-wide code-search, make it easier to deprecate obsolete code

2-2. Make it safer to change code that affects multiple repos.

Mono-repos have solved most dependency-related issues by including the dependency code within the repo itself, and only using a single version (which solves the diamond dependency problem). When code changes, developers solve any related issues in a single, atomic commit.

Some companies have built tooling around this idea. Any change a developer makes on Google’s codebase automatically run tens of thousands of unit tests across every product to make sure nothing is breaking.

This kind of holistic development process is not as straightforward when a codebase is broken up into multiple repos. That’s not to say it’s impossible. From Dan Luu’s post on mono-repos:

It’s theoretically possible to create a build system that makes building anything, with any dependencies, simple without having a monorepo, but it’s more effort, enough effort that I’ve never seen a system that does it seamlessly.

Opportunities

• Cross-repo CI
  • Add the ability to trigger build/CI on related repos with a new build of the current repo.
• A dependency version tracker and warnings to help engineers keep a single version.
  • This could possibly be a marketplace integration rather than our own efforts
  • Alternatively, it could also take advantage of Bumblebee’s dependency graph.
• A better build system for cross-repo dependencies.
  • Look into Google’s Bazel.

2-3. Make it easier for teams to plan across repos

Teams that work on multiple repos are already running into these issues. GitHub projects can’t include issues from different repos. Issues can’t apply to several related repos.

Opportunities:

• Tools and models that are scoped to an organization.
  • Org-wide projects
  • Multi-repo issues
• A new umbrella between an org and a repo to which issues and projects can be scoped (say, a super-repo, or, you know, a better name).

2-4. Help developers avoid duplicated efforts

In a mono-repo, developers have access to all existing code and can easily re-use it. This is more difficult when code is scattered between various repositories.

Opportunities:

• Improve ease-of-use
  • Develop a build tool that can easily mix and match code between different repos.
    • You may not want the entire repo, just parts of it. Is this possible?
    • Look into Google’s Bazel.
  • Explore single-namespace “GitHub pointers” that a build tool can point to directly rather than reference a specific repository.
• Improve communication and transparency
  • Automatically signal-boost the effort that’s currently happening to raise awareness.
  • Build an integrated way to ask other contributors whether something exists (“inner-sourcing”).
• Improve findability
  • Build a smart, searchable code registry for existing packages and methods.
  • Employ machine learning to figure out whether you’re writing code that might already exist (“this code looks familiar”).

2-5. Help leaders coordinate the big picture

Managing multiple, distinct repos is more taxing than managing a single repo. From Dan Luu, technical leaders need to deal with:

the hassle of tons of repos either managed through some script, or git submodules, or something worse, plus they get drowned in pull requests and issues by being subscribed to everything.

It is also more difficult to make sense of what is happening across what, at a system level, seems like many distinct projects.

Opportunities

• High-level rollups of repo activity (PRs, issues)
  • Either automatic, or a process to manually report up within GitHub
• Org-level insights to help leaders understand their engineering org.
  • Repos
    • What’s the traffic on various repos?
    • Make it easier to see where code is being added on various repos.
    • How many issues and PRs are open on each repo?
    • Org-wide dependency graph. Which repos are related, and how?
  • Teams
    • Which teams exist in the org?
    • How many issues is each team currently dealing with?
    • How well staffed is each team?
    • Which repos do various teams work on?
    • Where are the overlaps?

Acting on insights is also easier with a mono-repo, where reorganizing teams and responsibilities can be as simple as updating an OWNERS file. If you want to actually move files around the codebase, their history remains intact. Both of these things are much more difficult on a multi-repo setup with formal boundaries.

Opportunities:

• Create a method to easily transfer files/directories from one repo to another.
  • Including file history.
• Automated Code Owners
  • Changing an OWNERS file could automatically subscribe/unsubscribe people and teams from related PRs and issues.
• Encourage the use of org-wide teams as owners.

2-6. Enable trunk-based development (TBD)

TBD is important to a mono-repo model because it maintains the mono-repo’s HEAD as an up-to-date source of truth.

Google uses Piper, their implementation of Perforce. Piper doesn’t have the concept of a commit. All changes are instead tracked in a single changelist, which is then submitted for review and merge. Developers are encouraged to keep their changelists small.

TBD is already possible on GitHub, by using short-lived branches instead of changelists, and PRs for review/CI. The only real difference is that branching can encourage long-lived branches, which contradicts TBD principles, where branches are only used for releases.

To empower TBD workflows, we could implement optional admin features such as:

• Branch life reminders
  • When a branch is open for more than a day, for instance, send a reminder to merge the branch back into the trunk.
• Branch size warning
  • When a branch is too large (in # of commits or # of lines changed) warn developers.

These additional settings could potentially benefit non-TBD organizations as well.

3. Explore mono-tree workflows

I’ve seen this article (“Why Github can't host the Linux Kernel Community”) pop up a few times during this research phase, and I wanted to clear some things up.

The Linux kernel is a mono-repo and enjoys all of the aforementioned benefits, but it isn’t a giant mono-repo. With 16+ million lines of code, it isn’t small, but it hasn’t reached a scale where working with Git and GitHub would be technically difficult. As such, the reason why GitHub can’t host the Linux kernel is not scale-related—it’s workflow-related.

Due to the number of contributors, Linux needs a sub-project structure (meaning sub-project-specific issues, PRs, etc). To do this, they’ve adopted a mono-tree model. This is different from a mono-repo and the two shouldn’t be conflated.

The Linux kernel mono-tree consists of many forks of the upstream mono-repo, each belonging to a sub-project maintainer. This way, maintainers can consider the entire code base while being able to focus on their own subset of PRs and issues. Contributors can see which fork in the mono-tree to contribute to by looking at the MAINTAINERS file.

Contributors then submit changes as patches to the maintainers’ mailing list, and cc any other relevant mailing lists. There are several benefits to this:

• The email chain serves as a single location for discussion.
  • There is no need to open multiple PRs/issues for different audiences and coordinate between them.
  • Sometimes you aren’t sure which sub-project will be accepting your changes. With this model, the maintainers can easily agree amongst themselves where to merge the code.
• It’s really easy to reassign PRs/issues. You just alter the CC list.

The contributors and maintainers I’ve spoken to have all echoed the article’s appreciation of the model. It makes a lot of sense for how the Linux project functions, and I think we can find more opportunities here.

As with mono-repos, we can either support mono-trees as they exist today, or we can extract what makes them great and make those things possible as part of the GitHub flow.

3-1. Enable the mono-tree model

The existing mono-tree model simply doesn’t work on GitHub. To enable it, the article’s author asks for two structural changes:

• Allow forks of the same repo to exist side by side in the same organization.
• Make it possible to establish a fork relationship after the fact.

To replicate the benefits of a mailing list, that author asks that we:

• Allow PRs and Issues to apply to multiple org/repos.
  • The main benefit here is to create a single place for discussing a change throughout an entire community that spans many repos.
  • Note: this is different from the multi-repo PRs in 2-1, which apply different commits to different repos. The PRs mentioned here would instead apply the same commits to all repos.
• Allow different repos on the same PR/issue to have different statuses.
  • For instance, maintainers of one repo may decide to WONTFIX an issue, while others decide to work on it.
  • A PR could pass tests and become mergeable on a per-repo basis.
  • Sometimes, PRs might even have slightly different merge commits for different repos.

To make cross-org communication more obvious, we could also implement “public” org teams, which are surfaced to contributors outside the org.

• i.e. you could cc. a linux team such as @linux/kvm on any repo that’s a linux org repo fork.

Note: GitHub’s fork network view is already a pretty amazing benefit to mono-tree projects.

3-2. Reduce the need for mono-trees by splitting up repos

I think that a lot of the aforementioned requests will substantially increase GitHub’s complexity, and make it less useable for the bulk of our users. More specifically, I don’t think GitHub can or should be all things to all people.

I also don’t think implementing these changes will actually result in the Linux kernel or other legacy open-source project migrating over to GitHub:

• Some people with clout on these projects simply dislike GitHub.
• Many contributors refuse to leave their terminal. They use vim and an email CLI for pull requests, and have no interest in a web interface. In the last all-hands, there was a mention of building tooling for accessing GitHub via a CLI, but in this context it feels more like a distraction.

Finally, the article treats mono-trees as a best practice for large open source projects, and I don’t think that’s true. It’s just that there isn’t anything with the same benefits—yet.

Instead of supporting mono-trees as they exist today, it might make more sense to get at why people use them to begin with, and solve for that. So let’s say the Linux kernel project existed in a single repo, rather than hundreds of them. What would we need to do in order to make that as enticing as a sub-tree?

1. The same benefits of a mono-repo, which I’ve already discussed in 2.

2. Scoped/focused communications, which I’ve already discussed in 1-4.

3. Email chains: the ability to include multiple maintainers from different areas—no longer an issue if the project lives in a single organization.

4. Email chains: the flexibility of adding and removing mailing lists from the email chain.

   • To replicate this, we could build a way to update PR/issue subscriptions.
     • Right now developers can add a new team or user by mentioning them, but there’s no way to remove them or switch them out when they’re no longer needed.
     • This replicates the ability to change your cc list on mailing lists.
     • As with 2-5, this could be done automatically using a MAINTAINERS/OWNERS file.

5. The ability to make changes in a relative vacuum (without other maintainers’ work interfering). All the work comes together upstream, merged by the project’s main stewards.

   • To replicate this, we could create maintainer branches.
     • These are basically just… branches, created dynamically via the MAINTAINERS file.
     • They’re protected branches. Changes are only mergeable by the maintainers.
     • Since this follows the same mono-tree model, contributors should be able to open one PR against multiple branches.
       • These branches could be selected automatically, based on the MAINTAINERS file.
       • Since relevant teams are already in the same org and can be cc’ed on the PR, you only need to open the PR against branches you want to merge into.
     • Only maintainers can submit PRs from their maintainer branch to master. The project’s main stewards control master.

If we want to enable this kind of workflow on GitHub, implementing these features should do it.