Happy to report that I managed to fix the problem, at least for my setup!
TL;DR:
Replaced MariaDB with MySQL 8 and applied these changes, now pages load in 450ms and the REST API responds in 120ms. 
Had to do work on an older project and noticed that it loaded much, much faster than my current one. After lots of debugging and updating the environment of the newer project to match the old one, I determined that the database was the problem – the newer project used MariaDB to more closely align with the production setup, the old one used MySQL 8. Unfortunately you can’t hotswap databases, so I needed to zip my project folder and reimport it into Local, this time choosing MySQL as database.
And indeed: switching from MariaDB to MySQL 8 reduced the loading times from 3.5s down to a bearable 1.2s!
Next, I applied the changes from the “Optimizing Local “Lightning” performance on Windows 10” thread to my project. Not sure if the opcache changes improved anything (if so, then by a couple ms at most), but disabling the xdebug extension further improved loading times and got them down to 450ms! (sometimes 900ms on first load, I guess because there’s no cache yet, but oh my…)
Even better, responses from the REST API now only take 120ms – previously they took 3-4s like the slow page loads, which really affected my project because it takes advantage of this API a lot.