​After a year of building infrastructure, it seemed worth finding out whether it actually worked.
The test was simple: create a million rests and see what happened. One operation –

​– repeated until either something broke or the numbers settled.

First experiment: one client, one thread, a million sequential calls. This was on a 2017 MacBook Pro, so not exactly cutting-edge hardware.

Every call succeeded. Average execution time was 35 microseconds. Memory stayed at 112 MB without drifting upwards. Garbage collection remained in the young generation. Thread count didn't move. Zero errors.
​
Nothing dramatic, which was the point.

Second test: four threads on one client, 250,000 calls each. Another million operations, but with some concurrency this time.

Same result. All calls succeeded, memory stayed flat, garbage collection behaved, threads were stable. Still zero errors.
​
This level of concurrency should be unremarkable for any serious system. It was.

Third experiment: ten clients, four threads each. Forty threads total – enough to make the laptop work.

Average call times went up to 246 microseconds, but everything else held steady. Memory stable, garbage collection well-behaved, threads disciplined. Error count still zero.
​
Same 2017 laptop that started the tests.

From one thread to forty, from 35 microseconds to 246, the pattern was consistent. The system handled load without breaking.
​
These were only raw API calls, not STM transactions with musical logic and collaboration on top. Those will be next. For now, the figures suggest the foundations are fast, steady, and ready to carry more.

Three tests left on the visibility work. After that, this part of the system is done.

It’s taken a few weeks to get the final bits of the gRPC server/client interaction implemented, but it now feels solid. The server reliably and efficiently handles all the complex aspects of live clients connecting, disconnecting, and communicating locally or over the inherently chaotic internet. 

With that done, the focus is now on what is called observability: being able to see what's going on inside. In other words, server statistics. We’re now tracking server-wide and per-client metrics: connections, API call rates, message sizes, event streaming, queue overflows. Endpoints can return JSON during development or Prometheus format in production. Content negotiation is automatic, or you can force a format with a query parameter. No new tools to learn. Grafana dashboards work straight away.

Why this matters for musicians: research has shown that in collaborative settings, users prefer a notation program that lets them work together over one with the most features. That says everything about how poorly existing software supports collaboration. It’s treated as an optional extra, if it exists at all.

Ooloi is built differently. Collaboration is not bolted on afterwards but designed into the system from the start. If event queues slow or connections drop, collaboration breaks down. A slur added on one screen never appears on another, or two edits collide and the score diverges. The statistics make these problems visible. They show when the system is stressed, when clients are falling behind, when the score risks drifting out of alignment. In short, they make collaboration trustworthy.

And unlike the current trade-offs, Ooloi isn’t choosing collaboration instead of notational depth. It is designed to be both: a serious, advanced notation platform that also makes shared editing reliable.

The same architecture also opens up another possibility. Because Ooloi is client/server, it can stream smoothly scrolling scores in real time. That makes digital music stands for ensembles or orchestras feasible. It’s not the main focus, but it’s there: the professional counterpart to the school-level collaboration studies.

Three tests left. Once they pass, the intricate server machinery will be visible. And visibility is the first condition for trust.

Deep in the implementation of the gRPC layer I fell into the same foxhole as so many gRPC developers seem to do: implementing streaming between gRPC server and client using in-process communication. Everything works swimmingly: high speed, nothing gets lost, advanced async patterns just work.

And then they try doing this using network transport. Suddenly HTTP/2 changes everything, and that equivalence between simple request-response and asynchronous streaming just explodes.
​
I spent a couple of days cursing the universe, but in the end I managed to crawl up into the daylight again with a solid solution. Since this seems to be a common snag, I decided to publish a little guide on the considerations involved, so others may have to do less cursing.

Here it is: GRPC_STREAMING_THREADING_GUIDE.md

​Nearing the end of the gRPC implementation phase now: coming blog posts should soon become more musical again.

The Ooloi server must of course also have comprehensive introspective statistics, as it can be deployed in several ways and has a structure which is perhaps a bit unusual. The combination of STM-wrapped gRPC API calls with server-to-client event streaming creates some operational challenges: things like per-client queue overflow behaviour and collaborative editing session patterns.

Hence this new ADR on server statistics, which takes a two-level approach: server-wide aggregates that survive client churn, plus detailed per-client metrics for operational visibility. Oh, and it interfaces with Grafana, Prometheus, and friends straight out of the box via simple HTTP endpoints.

Halfway through the stats implementation now. It's been a productive weekend.

I've just published ADR-0024 and, based on it, a new gRPC Communication Guide documenting when flow control helps and when it interferes.

The decision was straightforward once the analysis was done: implement flow control for event streaming (where slow clients can block fast ones) but avoid it entirely for API requests (where gRPC's threading model and STM transactions already coordinate beautifully).

Sometimes the most sophisticated solution is knowing where not to add sophistication.

The guide covers the technical reasoning and includes practical examples for anyone implementing gRPC clients, regardless of language choice.