Upgrading to Scala 3.9 and sbt 2

Scala 3.9 was released in September and is now the new LTS version of Scala 3, replacing Scala 3.3.

We've also had the first stable release of sbt 2 this year, after a long development period.

For anyone starting a new Scala project, I think the decision is fairly straightforward. I'd use Scala 3.9 and, assuming the libraries and plugins I needed supported it, I'd probably use sbt 2 as well.

Existing applications are more interesting.

If I had a number of services running perfectly well on Scala 3.3 and sbt 1, I wouldn't upgrade them just because there are newer versions available. There needs to be a reason to spend the time and take the risk.

So what do we actually get from upgrading, and how much work is involved?

Why consider upgrading now?

The main reason for me is the move to a new LTS.

Scala 3.9 is now the recommended version for teams that favour stability and longer-term support. Scala 3.3 is still supported, so there's no immediate pressure to move everything.

That gives teams time to plan the upgrade rather than wait until another dependency or platform change forces it.

There's quite a lot between 3.3 and 3.9. You're crossing five minor releases, with changes to the compiler, language, tooling and standard library. Scala 3.8 also moved the minimum supported JDK to 17.

I wouldn't base the decision on any individual language feature though.

For an established application, keeping reasonably close to the supported versions of Scala, Java and the libraries around them is enough of a reason. The further behind an application gets, the more dependencies and changes tend to get bundled into the eventual upgrade.

sbt 2 has a slightly different case.

There are some changes that could have a noticeable effect on development: faster startup using sbtn, incremental testing and caching of compilation and test tasks.

Whether that makes much difference depends on the project.

For a large build that developers are running throughout the day, I'd start by measuring how long the current build and test cycle takes. If build performance is already good, I wouldn't use that as a reason to move to sbt 2. If developers are regularly waiting on builds and tests, the caching and incremental testing improvements could make the upgrade worthwhile. I'd measure the same build again afterwards to see what difference it actually made.

How difficult are people finding it?

The Scala 3.9 migration looks fairly encouraging.

Before releasing it, Scala tested close to 2,000 open-source projects using the Open Community Build. Around 1,780 built with no changes or relatively small changes.

That doesn't mean every production application will upgrade easily, but it's a useful indication of how much compatibility has been retained between the two LTS versions.

I was more interested in the reaction to sbt 2.

It went through a long release-candidate period and plugin support was an obvious concern. That seems to have improved quite a bit. By September, Scaladex listed 172 plugins available for sbt 2.

There are also some early reports from developers who have tried it on existing projects. One developer on Reddit reported moving a Play application to an sbt 2 release candidate and Scala 3.8 with very little difficulty. Others have raised issues around particular plugins, custom builds and some of the changed sbt behaviour.

A normal application using commonly maintained libraries and plugins should be a different proposition from a large multi-project build with years of custom sbt code.

So I wouldn't assume the upgrade will be difficult, but I'd check the project before deciding how much work is involved.

Start with one application

If I was looking at an estate of Scala services, I'd start with one.

Not the smallest service, because it probably won't tell us much, and not the most critical one either.

I'd pick something reasonably representative.

For example:

Scala 3.3
sbt 1.x
JDK 17
Play / Pekko

ScalaTest
sbt-scoverage
sbt-native-packager

Internal libraries
Some custom build configuration
A normal CI/CD pipeline

Before changing anything, I'd check:

Current JDK version
Scala and sbt versions

Framework versions
Scala dependencies
sbt plugins

Custom sbt tasks
Cross-built modules

Internal libraries being published
Other services consuming those libraries

CI build commands
Test reporting
Packaging and deployment

The internal libraries are worth checking early.

If an internal library is published using Scala 3.9, applications still running Scala 3.3 can't consume that version.

If several services share the same internal libraries, the order they are upgraded matters.

Don't upgrade everything together

I'd keep the Scala, Java, dependency and sbt changes separate where possible.

So I wouldn't start with a pull request that changes all of this:

JDK
Scala
sbt
Play
Pekko
ScalaTest
sbt plugins
application dependencies

If the application is still running Java 11, I'd move it to Java 17 first.

Build it and run the tests without changing Scala.

Once that's working, move on to Scala.

The Scala migration guidance recommends moving through the main parts of the stack in roughly this order:

Build tooling, where needed
        ↓
JDK
        ↓
Scala.js, if used
        ↓
Scala compiler
        ↓
Libraries

There's a practical reason for leaving the library upgrades until later. A newer Scala compiler should generally continue working with the libraries you already have, while upgrading a library first can introduce a requirement for a newer compiler.

I'd also keep the changes in separate commits. If something starts failing after a dependency upgrade, you still know that the compiler migration itself was working.

Moving from Scala 3.3 to 3.9

For a small application, the first thing I'd probably try is simply changing the Scala version and compiling it.

You may find there isn't much more to do.

For a larger application, Scala provides migration modes that can make the move more controlled.

Moving from 3.3 to 3.9 crosses the migration rules introduced in 3.4, 3.5, 3.6, 3.7, 3.8 and 3.9.

For example:

scalacOptions ++= Seq(
  "-source:3.4-migration",
  "-rewrite"
)

The compiler can then rewrite code affected by changes introduced in that version.

For a project starting on 3.3, the migration modes are:

3.4-migration
3.5-migration
3.6-migration
3.7-migration
3.8-migration
3.9-migration

They need to be run in order. Running the 3.9 migration mode doesn't include all the earlier rewrites.

After each useful set of changes I'd review the diff, compile and run the tests before moving on.

There's one part of the 3.8 move to watch

Scala 3.8 changed how the standard library is built. It is now compiled with Scala 3 rather than Scala 2.13.

That affects some code where context parameters are passed explicitly.

For example:

Array.empty(ClassTag.Int)

may need to become:

Array.empty(using ClassTag.Int)

Scala 3.7.4 can perform most of these changes automatically.

Before moving beyond 3.7, run:

-source:3.7-migration -rewrite

and review the changes it makes.

There's also a compatibility issue with some Scala 2.13 libraries that use scala-reflect. Apache Spark is one example mentioned in the Scala migration guidance.

If the application uses one of those libraries, I'd check its current Scala 3 support before moving past this point.

Then look at sbt 2

I wouldn't make the sbt upgrade part of the Scala upgrade unless the project was small enough that there was little value in separating them.

Scala 3.9 can still be built with sbt 1, so there's no requirement to move both together.

Once the application is working on Scala 3.9, the sbt version can be changed separately.

In project/build.properties:

sbt.version=2.0.9

There are a few changes I'd check straight away.

Bare settings have changed

In sbt 1, a setting written directly in build.sbt applies to the root project.

For example:

publish / skip := true

In sbt 2, bare settings are common settings and are applied to subprojects as well.

So the example above could result in all of the subprojects being skipped when publishing.

If it should only apply to the root project:

LocalRootProject / publish / skip := true

For a multi-project build I'd review the bare settings in build.sbt before running the upgraded build.

Custom tasks need checking

sbt 2 caches tasks by default.

For normal compile and test tasks, that's one of the benefits of the upgrade.

Custom tasks may need some work.

If a task has a side effect that needs to happen every time it runs, restoring its result from cache won't run the task body again.

Those tasks can be excluded from caching:

myTask := Def.uncached {
  // task implementation
}

There's another case to look for. Cached task results need a JsonFormat. If a custom task returns something that can't be serialised by the cache, sbt can fail while loading the build.

For an existing project, I'd search the build for custom task definitions and deal with those before looking at anything more complicated.

Check the CI pipeline

Some sbt 1 commands need changing as well.

For example:

sbt clean compile test

becomes:

sbt "clean ; compile ; test"

There's also a change in where test output is written. Test reports now sit below target/out, so a CI pipeline with hard-coded paths for JUnit XML or other test results may need updating.

The sbt server is another difference.

With sbt 2, later sbt commands in the same CI job can reuse the server started by an earlier command. They'll also continue using the environment that was present when that server started.

If different stages deliberately use different environment variables, either put them at job level or shut the server down between stages:

sbt "clean ; compile ; test ; shutdown"

Check the plugins before upgrading sbt

I'd go through project/plugins.sbt before changing the sbt version.

For each plugin:

Is there an sbt 2 version?
Is the plugin still maintained?
Does the sbt 2 version require configuration changes?
Do we still need the plugin?

The ecosystem is in a much better position than it was during the early sbt 2 release candidates. There are now more than 170 compatible plugins and the sbt2-compat work makes it easier for plugin authors to support both sbt versions.

There will still be older projects where one abandoned plugin holds the upgrade back.

At that point I'd decide whether to replace the plugin, remove it, update it ourselves or stay on sbt 1 for the time being.

How I'd approach a larger Scala estate

Once the first application has been upgraded, I'd use what we learned from it for the rest.

Something along these lines:

1. Check JDK, dependencies and sbt plugins
2. Move to JDK 17 if required
3. Run the Scala migration rewrites
4. Deal with the Scala 3.8 compatibility changes
5. Move to Scala 3.9
6. Test and deploy
7. Upgrade sbt separately
8. Review custom tasks and settings
9. Update CI
10. Measure build and test performance

If the estate uses the same frameworks, internal libraries and build plugins, most of the investigation has already been done by the time you reach the second or third service.

I'd also keep the Scala 3.9 and sbt 2 changes as separate pull requests for the larger services. There's no real benefit in making them dependent on each other.

Would I upgrade now?

For a new project, I'd use Scala 3.9.

For an existing Scala 3.3 application, I'd start looking at the move now. There's no need to rush because 3.3 is still supported, but I wouldn't wait until the end of that support period either.

sbt 2 is a slightly different decision.

For a straightforward build using supported plugins, I'd try it. The caching and incremental testing improvements are useful, and plugin support is now much broader than it was earlier in the year.

For a complicated build with a lot of custom sbt code, I'd upgrade Scala first and look at sbt separately.

And if the application is still on Scala 2, I wouldn't use this process. Moving from Scala 2 to Scala 3 has a different set of compatibility issues and needs its own migration plan.

For an established Scala 3 application, though, the move from 3.3 to 3.9 looks manageable. The main work is likely to be understanding the dependencies and build around the application rather than changing large amounts of application code.