On ongoing work in the Rust compiler team

Or how to answer "what are the compiler team working on?"..

👋 Hello!

This my first writing about the Rust project, so a little about me and this newsletter:

You might know me as @davidtwco on GitHub or Zulip or anywhere else, I’m currently co-lead of Rust’s compiler team. For as long as I’ve been involved in the project, Rust has had a healthy culture of blog posts being a medium for testing out ideas about the language and project, and I want to contribute to that with some of my own ideas for how we might work better. I’ve also got some (very) vague plans to try and use this newsletter to teach myself about parts of the Rust compiler that I don’t know very well, and to experiment with ways of regularly summarising goings-on in the compiler for the benefit of other contributors.

Before I get started, some caveats about everything I’ll write:

• All opinions expressed in this newsletter are my own, not the Rust project, Rust compiler team, or my employer, unless I tell you otherwise.

• Reviewers of earlier drafts don’t necessarily agree with anything I’ve written.

• Just because I’ve proposed something in a blog post doesn’t mean it’ll happen - depending on the feedback I get from other contributors and the wider community, I may or may not go forward with some of these ideas.

All of the above is also on this newsletter’s about page - which I’ll keep updated.

---

I’ve been thinking a lot lately about how to answer one specific question - “what are the compiler team working on right now?” - and it’s a surprisingly tricky one. I don’t have a good answer to it.

It isn’t that the compiler team aren’t up to much - we’ve always got lots to do and the team are always working hard. It’s more that we’ve never really found a way to track or direct ongoing work within the team over the long-term. It would be great if there would be a page on the project’s website that listed everything the compiler team were working on.

I think the compiler team work pretty well despite the fact that I can’t really answer that question. It’s an important question though - what are we working on? - because being able to answer that helps keep the community informed, can direct investment and attracts contributors.

We can’t communicate what we’re working on to Rust’s users if we don’t know ourselves. If it’s hard for me to come up with a list of the compiler team’s ongoing projects, then it’s a challenge for the Rust user who isn’t involved in the project.

Knowing what projects are ongoing can help direct investment into the project. Rust is being adopted throughout the industry, and some of those adopters will be looking to contribute back to the project.

One way to do that is to become a member of the Foundation, where that money funds resources like the cloud compute program and the community grants program. Another way is to hire contributors from project teams and sponsor their ongoing contributions and to start teams which contribute to the project. If you’re in that latter camp, you need to work out what to contribute to, and you (hopefully) need to justify that as being in the interests of the Rust project.

Team roadmaps, like the compiler team, language team and library team’s roadmaps from last year, are great for this. If you’re just starting out with contributions, then the roadmap directs you to the areas that are a priority; and when reporting to higher-ups about what you’ve contributed, being able to point at team roadmaps makes it easy to show those contributions are aligned with the project’s goals and priorities.

Ongoing projects are great opportunities to attract and retain contributors, but only if the project’s working group or team has momentum behind it. I started contributing during the “impl period” back in 2017 - a few months where the project spun up a bunch of working groups all across the project to try and complete the 2017 roadmap.

For a new contributor, the impl period was great because we had a directory of projects that we could get involved in and had other new and experienced contributors to collaborate with. It wasn’t without its issues, and Niko wrote a great blog post on his lessons learned: some of which we’ve made great progress on - compiler documentation is in a much better place than it was back in 2018, thanks to the rustc developer guide.

One of the things I’ve always taken away from the impl period was the power of working groups to attract and retain contributors. I’m not sure if there has been any other single event which has attracted and retained as many contributors to the compiler team as the impl period did: @wesleywiser’s contributions ramped up during the impl period and Wesley now co-leads the compiler team and contributes to the compiler from Microsoft; @spastorino and @lqd have both been strong, consistent contributors for years and got their start with me in the NLL working group (both now contributing from Futurewei); @matthewjasper also started with the NLL working group and now contributes from my team at Huawei; and obviously there’s also me. We always have contributors come-and-go, that’s the nature of open source, but I don’t think there is anything else we’ve tried that has had five contributors stick around for years afterwards.

We’ve had working groups in the compiler team for years since the impl period, but they haven’t always been that successful, and not every working group during the impl period was that successful either. The NLL working group succeeded where other groups didn’t because it had a clear goal, lots of work to do, enough expertise to support the newcomers (thanks @pnkfelix and @nikomatsakis!), and it was very active.

One of the ways that I can answer the question “what are we working on?” is by looking at the team’s working groups. Working groups are the way we try to structure and direct (most of) the compiler team’s efforts today - so on the face of it, this seems like a reasonable way to answer the question.

In practice, it won’t be a very useful answer, because not all of those groups are very active. Some working groups are - the parallel compiler working group are making great progress and achieving really exciting performance improvements. Some others aren’t - the polymorphisation working group? Not so much (I lead the polymorphisation group, so I don’t mind using it as an example here, there are good reasons we’ve not done much).

If we look at the polymorphisation working group from the perspective of our motivations above - its existence doesn’t help the community know what we’re working on (because there isn’t much work happening), it wouldn’t be a good place to direct investment (but how would you know that?), and it wouldn’t be a good place to direct new contributors (there’s nothing going on to get involved with).

One of the things that we’ve tried to do with working groups (and similar structures) is direct effort into areas that we think need it. I’m increasingly skeptical that this is something we’re able to do.

As co-lead of the compiler team, I’m not really in a position to be able to direct the work that compiler team members or contributors do. Everyone in the team might even agree that we need to work on our support for debuggers, and we might even create a working group to focus on that, but it doesn’t mean that enough of us will take the time to contribute to that group’s goals so that the group succeeds.

This shouldn’t be too surprising - we’re an open source project and people contribute based on their interests and needs. By creating working groups that have been prescriptive (trying to Field of Dreams our way to achieving our goals), we haven’t actually achieved those goals.

Instead, we should be descriptive and focus on trying to describe what is happening. Working groups are a great force multiplier - when there’s a natural upswell of interest in an area, we should capitalise on that and create structures which help organise that work and attract new contributors - this is exactly what successful working groups like NLL and the parallel compiler group have been able to do.

It’s still a worthwhile goal to try and direct work into parts of the project that need it, but we should separate that from tracking what we are working on. If we try and conflate the two (as we’ve been doing), then we succeed at neither.

Instead of taking a top-down approach - creating working groups around areas where some work is happening or where we’d like to see work happening - take a bottom-up approach - regularly ask contributors what they’re working on, and group related work together.

I imagine this working something like the following:

1. Regularly, every two weeks or month or so, ask regular contributors what they’re working on. It’s okay if there aren’t any updates. Just a couple sentences or short paragraph describing their ongoing work/goals/projects.

2. Team leads or an operations team will group related reports together - if there are a handful of people working on diagnostics, then we can group those reports together; if there are lots of people working on the type system, then we can group those reports together.

   1. Once you’ve got some groups, we can write a brief overview of the goal that’s being worked towards.

   2. As contributions shift and move to different areas, groups come and go too - so we don’t need to make sure our current set of working groups reflect what people are actually doing.

   3. If a group is consistently appearing, then there might be some momentum there, and we can think about creating structures to leverage that momentum, such as directing new contributors towards it.

   4. We could use the reports or the groups in the compiler team’s weekly meetings, in regular update blogs or as a part of the project/compiler team’s website.

A bottom-up approach like this leverages the dynamics of the open source project that we currently have, rather than trying to impose a structure on that work which doesn’t always exist.

tl;dr:

• Be descriptive rather than prescriptive

• Take a bottom-up approach to tracking work rather than a top-down approach

• Separate tracking ongoing work from future work

I’ve been thinking about this from the perspective of the compiler team because that’s the team I’ve fairly recently started to co-lead, but it probably applies to other teams - whether they have working groups or project groups or initiatives or whatever else - and the project as a whole. Reach out if you want to chat about this more :)

Or, if you work on big open source projects that aren’t Rust, I’d be interested in learning how you approach these problems - get in touch!

Thanks to lqd, Wesley Wiser, Niko Matsakis and Federico Pizzuti for reading earlier drafts of this.