When you have a lot of code it’s always hard to find a proper approach to organize your code.
Typically engineers choose between monorepo and multirepo layouts. Both approaches have well-known advantages and disadvantages which significantly affect team productivity.
It’s possible to establish a combined workflow, keeping all the advantages of a multirepo layout but not giving up any positive traits of monorepo layout.
We created a draft of a tool implementing our approach for Scala projects using SBT.
Good points of a monorepo layout are:
But there are significant shortcomings:
Multirepo layout is always considered as a first answer to any monorepo issues because:
Though multirepo is a disaster:
These things are especially bad when you have some explicit or implicit dependencies between your components which is a typical case, usually we have at least one shared library (aka SDK) and many or all our components (aka microservices) depend on it.
Let’s assume that we have a product (an online auction platform, for example) consisting of several software components:
iam
: Identity and Account Management Servicebilling
: Billing Serviceanalytics
: Analytics Servicebidding
: Bidding Servicecatalog
: Item Catalog ServiceAll these projects use one shared SDK named sdk
.
We may also assume that there would be several teams working on these projects.
For example we may assign sdk
, iam
and catalog
projects to “infrastructure” team,
billing
and analytics
to “finance” team and bidding
to “store” team.
Imagine that you have a magic tool project
allowing us to choose which
projects we want to work on and set up the corresponding environment:
# Prepares workspace for all our components
project prepare *
# Prepares workspace to work on `billing` and `analytics`
# Pulls in `sdk` as well
project prepare +billing +analytics
# Prepares workspace to work on `sdk`, `iam` and `catalog`
project prepare :infrastructure
# Prepares cross-build project
project prepare --platforms js,jvm,native :infrastructure
This tool would need some kind of declarative description of our product stored in a repository. The rest can be as flexible as we wish. For example, in case we don’t want to keep all our source code in one repository, the tool may pull the components from different repositories, take care of commits, etc, etc.
We may say that our repository has roles and at any time we may choose which roles we wish to activate. So, we may call this approach “Role-Based Repository”, or RBR.
Such a tool would solve most of the problems. When we need to perform a global refactoring we may generate all-in-one project. When we wish to implement a quick patch we may generate a project with just one component. When we need to integrate several components we may choose what exactly we need. Etc, etc.
Unfortunately, there is no such a tool which is polyglot, convenient and easy to use. Something can be done with Bazel, but as far as I know there are no good solutions at this moment (October 2019).
And things become bad when we need this for Scala. And especially bad when we need to work with cross-platform Scala environments (ScalaJS and Scala Native).
There is no sane way to exclude some projects from an SBT build according to some criteria. You may write something like
lazy val conditionalProject = if (condition) {
project.in(...)
} else {
null
}
But it’s ugly, inconvenient and hard to compose.
And cross-platform projects were always a pain. It takes at least twice more time to build a cross-project. And there is no way to, for example, omit all the ScalaJS projects from a build.
For example, IDEA frequently fails to compile any project
if sbt-crossproject
plugin is on. IDEA cannot run tests
in cross-projects. And so on.
SBT builds become very verbose and hard to maintain when you use cross-projects. Usually you have to write at least 3 redundant expressions per artifact.
We’ve created our own dirty tool which prototypes the approach we wish to have. Essentially, it’s a library intended to be used in an ammonite script which takes declarative project definitions and emits SBT build files.
You may find a real project using it here. In case you want to play with it you would need Coursier installed. After you clone the project you may try the following commands:
# generates pure JVM project
./sbtgen.sc
# generates JVM/JS cross-project
./sbtgen.sc --js
# generates pure JVM project for just one of our components
./sbtgen.sc -u distage
Currently sbtgen
is a very simple and dirty prototype but it made our team happy.
Now it’s easy to release, when we need it we may choose what to work on, what to build and what to test.
Also, surprisingly, SBT startup time is a lot shorter when we generate our projects instead of using
sophisticated plugins to avoid settings duplication.
I don’t encourage you to use sbtgen
, but next time you think about organizing your code try to consider RBR flow even
if you would have to write your very own code generator.
I may say for sure that you will not be disappointed.
sbtgen
needs to support multi-repository layouts. At this point all the source code needs to be kept together with the build descriptor,The idea of roles is very useful in many different domains. For example we may fuse microservices into “flexible monoliths”, check our slides You may also read about our project, distage, a module system with an automatic solver for Scala. It allows you to build multi-role applications.
You may follow me on twitter.
The author of a similar tool for .NET/C# and JS projects approached me recently. Here you may find his tool. I think it’s a good proof showing that the idea is viable. And I think that we need new flexible polyglot build tools supporting role-based approach (Bazel?). I have a computational model suitable for such tools, and one day I’ll make another post about it.