Five minutes of some trivial testing shows that this is just not true.

Noise is still affected by input gain

Here’s a composed recording of four 3-second clips recording the room tone in my office. They are (in order): Auto gain, 57dB, 35dB, 0dB:

As you can hear, auto and maximum input gain in this case have very similar noise levels (which is to say, perceptually none), but as you reduce the recorder’s input gain (and instead apply the gain in post) the noise increases substantially and becomes very noticeable.

Granted this is a very big gain application – 57dB – which you would hopefully never need to apply to a real recording, but nonetheless it demonstrates that Tascam’s claims are exaggerations at best; if you actually had the Portacapture X8’s input gain set to 0dB and recorded quiet sounds, you would in fact have problems with noise – problems that would be avoided with a correct input gain setting.

Clipping still happens if input gain is too high

The recorder clearly applies actual analog amplification and can still saturate its ADCs, as shown in this composite of three gain levels. They are (in order): Auto gain, 35dB, 57dB.

Warning: annoying, distorted sound.

It only went over by about 3dB (at 57dB input gain), but that was enough to destroy the input signal and make the recording unusable.

Auto Gain still affects the recording

In the above tests I included the Auto Gain setting, even though it doesn’t exhibit particularly high noise nor does it clip in these simple sound environments (basically constant sound levels). And it worked pretty well (not optimal input gain levels, but close enough for my taste).

But, I was curious if it had any effect at all – again, reading about 32-bit floating-point recording online, you’d be forgiven for thinking Auto Gain has no impact on the actual recorded data. Many people liken the format to camera raw files, and some explicitly state that Auto Gain has no impact on the bits that get written to disk.

This is completely false, at least in the case of this Tascam Portacapture X8. It’s trivial to see why:

Auto Gain still does exactly what it always does – it changes the gain in response to the input. That change is baked into the recorded audio track(s).

So in a nutshell, 32-bit floating-point recording might provide slightly more flexibility in some situations, but it does not mean you can ignore your input level settings, it does not mean you can use Auto Gain in every scenario, and it does not mean you cannot clip.

Addendum: Technical details

I tested post-production gain changes in Final Cut Pro, Logic Pro, & Audacity. All produced the exact same results (to my ears). I had read that Final Cut Pro sometimes ‘bakes in’ clipping with 32-bit float inputs, as if it’s pre-rendering them down to some smaller dynamic range, so I wanted to rule out some Final Cut Pro-specific stupidity. It’s possible that all these editors are doing that, but I’d be flabbergasted if that’s true.

The “industrial noise” sample I used was my dehumidifier, which is about 66dB according to DecibelX on my iPhone 14 Pro. My office room tone is about 42dB according to the same app.

I used Sennheiser K6 modules with an ME65 and ME66 attached, plugged into the Tascam Portacapture X8 via 3′ Cable Matters XLR cables.

I recorded at 96kHz because that’s what I’ll use most often. I like the aliasing headroom above 48kHz (even though of course my final outputs are almost always 44.1kHz or 48kHz), but don’t see evidence that 192kHz provides meaningful additional benefit (and it also hurts the frequency response significantly, compared with 48kHz and 96kHz, according to Tascam).

This is kind of a sister function to hard links, which similarly avoid copying the file’s contents but where modifications apply to all copies. See also this article on the differences, including versus aliases and symlinks.

Copy-on-write is beneficial for several reasons:

• Copies don’t take up any significant space (just whatever tiny amount is necessary for their metadata).
• The initial copy operation is practically instantaneous (just a few small metadata writes & updates).
• Deferring (if not entirely avoiding) the actual disk I/O reduces wear on the disk.
• The copies can share common segments, saving disk space even when they’re not ultimately identical copies.

It does have some potential downsides:

• You don’t get the increased data redundancy and error resilience that actual copies provide (although if you’re aiming for data redundancy or backup, you should be using separate physical disks anyway).
• It can make subsequent modifications of the file slower, as even just modifying a single byte can trigger the actual copy to be performed.

And some basic limitations:

• It’s only supported on APFS (and maybe additional file systems added by 3rd party extensions, but I haven’t tested nor can I find any accounts of this).
• It only works within individual volumes (it doesn’t work even between two volumes in the same APFS container, or sharing the same physical disk).

Contrary to what I saw online in a few places, copy-on-write works on all APFS volumes, irrespective of whether they are backed by SSDs, HDDs, or some other type of storage.

Should I use it?

Yes!

For most purposes those downsides aren’t an issue, and the limitations merely mean that it’s wise to have a fallback option (of just copying the actual file contents) whenever copy-on-write isn’t available. And many of the tools & APIs fallback automatically (unless you explicitly require them not to, such as with COPYFILE_CLONE_FORCE to copyfile).

How do I use it?

Testing method

Not that this is interesting, just for posterity and to show my work a bit, in case I made a mistake.

I created large-enough files on each of my test volumes (APFS on SSD, APFS on HDD, HFS+ on SSD) that an actual copy would take tens of seconds at least, using dd e.g.:

I then ran the various command line tools on these files, attempting to clone the file to a different name in the same folder, and observed disk I/O activity with Activity Monitor and iStat Menus.

For actual copy-on-writes I would observe that the program successfully concluded practically instantly, and there’d be at most a small blip of disk writes (for metadata modifications).

For failed copy-on-writes I would observe that the program would not exit promptly and I’d see voluminous disk writes (hundreds of megabytes to gigabytes per second, depending on the disk, sustained for many seconds until I was satisfied with the results and killed the test).

For API tests the overall approach was the same, but the APIs were invoked from inside swift repl where possible, and from a throw-away Swift script otherwise³.

1. I haven’t tested it, but as far as I’ve heard APFS does not actually check if the modifications actually diverge the files. e.g. if you “modify” a byte of the file to the value it already has – a pointless but technically possible operation that leaves both copies still identical – APFS will still copy the modified block.
   
   Furthermore, APFS & Apple’s operating systems appear to have no tools (nor APIs) to deduplicate files – e.g. to detect full or partial copies and deduplicate their actual storage on disk. Not even tools that you could invoke manually if you do the hard work of first determining that two files’ contents are identical.
   
   APFS / Apple’s operating systems rely entirely on user applications using copy-on-write explicitly and upfront. ↩︎
2. Curiously, clonefile first shipped in macOS 10.12 (Sierra), according to its man page, which is the release preceding the introduction of APFS in macOS 10.13 (High Sierra). Yet, as far as I can tell HFS+ doesn’t and never did support cloning – nor do any of the other file systems supported by macOS 10.12 (e.g. FAT16 & FAT32, exFAT, NFS). It’s possible it was just convenient for Apple to include it in the prior release as they were probably testing APFS with it internally, during APFS’s development.
   
   Update: Michael Tsai pointed out that Sierra included APFS support in beta form (e.g. you could create disk images with it, but not use it for a boot disk). Thus why clonefile (and other APFS-related tools) were included in Sierra. ↩︎
3. While the C APIs worked just fine inside swift repl irrespective of what volumes I was targeting, the Foundation APIs oddly refused to work whenever the files were not on the boot volume, throwing “Operation not permitted” errors (NSCocoaErrorDomain 513, with no user info). If it weren’t for the C APIs working just fine I’d think it’s a sandboxing issue, but clearly it’s not (and swift repl -disable-sandbox makes no difference). 🤔 ↩︎

Tangentially, I want to highlight that Axel’s comparison is notable because he is interested in efficiency, not mere brute performance. The two are usually correlated but not always the same. He correctly noted that electricity is a major and increasingly large part of server costs (see my prior post for why it’s even worse than you likely realise). That said, while he did take RAM and power measurements, his benchmark and analysis didn’t go into detail about energy efficiency.

Benchmark method & apparatus

Axel wanted to see how a very simple web server performed in:

• FPM w/ NGINX (PHP).
• Helidon (Kotlin / Java¹).
• Node.js (JavaScript).
• Vapor (Swift).

He was particularly interested in throughput & latency vs RAM & power usage. All are important metrics in their own right, but are most useful in light of each other.

He chose to use Fibonacci sequence calculation as the load. Choosing a load for any web server benchmark is always highly contentious, and not the focus of this post. Whether you think Fibonacci’s a good choice or not, read on to see why really it didn’t matter.

He did use very old hardware, though – an Intel Core i3-550 from over a decade ago. Fortunately it didn’t turn out to materially impact the relative results nor behaviours of the benchmark, but it’s usually unwise to add unnecessary [potential] variables to your setup, like unusual hardware.

In my own debugging and profiling, I used my also very old 10-core iMac Pro. It’s at least a Xeon? 😅

Benchmark results

In words:

• Helidon (Kotlin / Java) had the highest throughput and lowest latency at low (and arguably more reasonable) loads, but used by far the most RAM, and the most power. Consequently it handled the most load before requests started failing (timing out).
• Node.js (JavaScript) was qualitatively very similar to Helidon (Kotlin / Java) but less in all metrics – less throughput, less peak load capacity, but also less RAM and very slightly less power used.
• FPM + NGINX (PHP) followed the pattern.
• Vapor (Swift) did not – it had higher throughput than PHP yet requests started failing much sooner as load increased. It used the least RAM and least power, though, and kept on trucking irrespective of the load.

Many people would have left it at that – obviously the results make sense for the first three (“everyone knows” that Kotlin / Java’s faster than JavaScript that’s faster than PHP) and Vapor / Swift apparently just isn’t fast and has weird reliability behaviours. QED, right?

To his credit, Axel wasn’t so sure – he felt that the results he was seeing were suspicious, and he sought help from the Swift Forums in explaining or correcting them.

While Axel did suspect something was wrong – noting the oddly small but persistent failure rate – he missed the most obvious proof of wrongness – logically impossible results. Doing 80,000 continuous concurrent streams of requests with ~98% of those requests completing within the two second time limit means the server must have a throughput of at least 39,000 requests per second. Yet the benchmark tool reported a mere ~8,000 requests per second.

Sadly, though I pointed this out as the very first response to the thread, it seemed to be overlooked by everyone (even myself!), even though it clearly fingered the benchmark tool itself as the problem (which is only partially correct, as we’ll see later, but in any case was the exact right place to start looking).

Debugging the benchmark

Domain experts weigh in

The Swift Forum post immediately attracted relevant people: folks that work on Vapor and NIO, and folks that have experience using them. However, ironically this didn’t initially help – they tended to assume the problem was in Vapor (or its networking library, SwiftNIO) or how Vapor was being configured. It turned out none of this was really true – there was a small optimisation made to Vapor as a result of all this, which did marginally improve performance (in specific circumstances), but ultimately Vapor & NIO were not the problem, nor was the benchmark’s configuration and use of them.

There were some assertions that the results were plausible and just how Vapor performs, and that the “problem” was the choice of Vapor rather than some other web server framework (e.g. Hummingbird).

Others similarly asserted that the results were plausible because Swift uses reference-counting for memory management whereas PHP, JavaScript, and Kotlin / Java use garbage collection. It was presented as “common knowledge” that garbage collection has inherent benefits for some programs, like web servers, because it makes memory allocation super cheap.

Examining the load

Even though it was clear that something was wrong with the actual measurements, a lot of the early discussion revolved around the load used (Fibonacci sequence calculation), particularly regarding whether it was:

The “right” load

A few folks asserted that the CPU-heavy nature of calculating Fibonacci numbers isn’t representative of web servers generally. Multiple people noted that – in the Swift implementation, at least – the majority of the CPU time was spent doing the Fibonacci calculation. Some felt this was therefore not a useful benchmark of Vapor itself.

A lot of this boiled down to the “no true Scotsman” problem, which is very common in benchmarking, with a bit of perfect world logical fallacy peppered in, trying to identify the One True Representative Benchmark. See the earlier point about fixating on such matters rather than whether the benchmark is useful.

A “fair” load

Accusations were made pretty quickly that the benchmark is “unfair” to Swift because Swift doesn’t – it was asserted – have a properly-optimised “BigInt” implementation, unlike all the other languages tested.

No real evidence was given for this. Even if it were true, it doesn’t invalidate the benchmark – in fact, it just makes the benchmark more successful because it’s then highlighted an area where Swift is lacking.

The BigInt library that Axel used, attaswift/BigInt, is by far the most popular available for Swift, as judged by things like GitHub stars, forks, & contributor counts, ranking in web & GitHub searches, etc. There are quite a few others, though.

It wasn’t until actual evidence was presented, that the discussion made progress.

It was shown that in fact the BigInt implementation in question was significantly slower than it could be, because JavaScript’s implementation of addition was much faster. Some additional simple tests showed even wider performance gaps regarding the other key operation: rendering to strings. It was that data that turned out to be critical – I myself happened to have implemented BigInt string rendering for Apple’s new Foundation, and then saw it dramatically optimised by Oscar Byström Ericsson, whom has his own BigInt package for Swift, Numberick. So I had a pretty darn good idea of where I might find a faster package… 😆

You can read more about that specific bit of serendipity in the Swift Forums thread.

It was trivial to do the package switch, and it quickly improved Vapor/Swift’s showing in the benchmark manyfold – in combination with some other simple and reasonable tweaks, it was five times faster!

a += b
swap(&a, &b)

let c = a + b
a = b
b = c

Axel posted a follow-up with additional data (with the aforementioned changes and optimisations). That showed Swift now beating out the other three frameworks / languages, with the highest throughput and lowest latency (and still the lowest RAM and power usage).

So, all done, right? Turns out, Vapor/Swift wins, yeah?

Well, maybe.

Do these improvements apply to the other cases too?

That is yet to be examined. Because only Swift seemed to be producing odd results, Axel only put the benchmark to the Swift community for deeper analysis. It’s quite possible that doing the same with the other web frameworks & languages would similarly reveal potential improvements.

Still, the results are useful as they stand. Some simple and very plausible – even for a Swift beginner – optimisations made a big difference, though of course the biggest difference was simply using a different 3rd party package. There are a lot of useful lessons in that, both in the specifics as already covered and as general best practices.

…but… why is the success rate still weird?

Despite the improved performance, a fundamental problem remained: the numbers still didn’t make sense.

The success rates are slightly different but not materially – as concurrent requests go up, the throughput plateaus very quickly, yet success rate remains about the same. It’s exactly the same problem as at the outset – these results cannot possibly be correct.

Despite all the community’s efforts, we hadn’t actually figured out the real problem. We’d merely made Swift look better, without actually providing confidence in the accuracy of the results.

In fairness to myself, I was well aware that we weren’t done, I was just struggling to understand what was really going on, as I noted here.

Examining the benchmark tool

While there’d been some tangential questions about wrk, the benchmarking tool Axel used, it had largely been ignored thus far.

Ironically (as you’ll soon see) Axel chose wrk specifically because he didn’t like the behaviour he saw with ApacheBench. Mostly its lack of HTTP/1.1 connection reuse (a subjective but valid methodology choice on Axel’s part) but also because it sounds like he saw some inexplicable results from it too. In hindsight, that might have been a clue that something more pervasive was wrong.

In retrospect there were a few tangential comments in the Swift Forums thread that were on the right track, e.g.:

…when a new connection comes in, the server needs to make a decision: It can

• Either accept the new connection immediately, slowing the existing connections down a little (because now there are more connections to service with the same resources as before)
• Or it can prioritise the existing connections and slow the connection acceptance (increasing the latency of the first request in the new connection which now has to wait).

Johannes Weiss, Swift Forums post

As a little spoiler, it seems apparent that the other three web frameworks all accept incoming connections virtually immediately with priority over any existing connections & request handling (even though they don’t necessarily attempt to serve all those connections’ requests simultaneously). Vapor does not.

Suspicions did [correctly] develop around the opening of the connections themselves, which triggered testing with longer timeouts in a somewhat blind attempt to cover-up the “spurious” first moments of the test.

Characterising the failure mode(s)

Though admittedly I wasn’t fully conscious of what I was doing at the time, the next breakthrough came from simply gathering more data and analysing it qualitatively. This helped in two key ways:

• It better defined and pinned down the circumstances in which things appear to go wrong with the benchmark itself.
  
  It separated out a whole bunch of test configurations that seemingly weren’t interesting (as they behaved in line with intuition / expectations, and similarly across all four web servers).

• It provided hints and potential insight into the nature of the problem.
  
  It showed that there was some kind of variability (in time) in the benchmark’s behaviour, with three very distinct modes (including one which was basically the benchmark actually working as expected, the existence of which had been unknown until that point!).

Interestingly although ultimately only tangentially, this modality finding prompted quite a few “me too!” responses from other folks, about a variety of use-cases involving Vapor or NIO. I took that as affirmation that I was onto something real, but in retrospect that should have been an even better clue: the fact that some people had seen this issue without Vapor involved – the only common denominator was NIO. Even though it turns out NIO itself wasn’t doing any wrong, it was on the right path to answers. This was specifically pointed out to everyone, even.

Overlooked clues

At this point there were a bunch of discussions about benchmark tool configuration, hardware arrangement, whether TLS should be used, etc. I’m going to skim over it, because there’s not much to ultimately say about it – it turned out to not be on the right track in this case, or purely tangential, but it was entirely reasonable to investigate & discuss those aspects. Such is debug life.

What’s interesting is that yet another key clue was mentioned in the Swift Forums thread, yet was overlooked because it was attributed incorrectly and the mechanics miscategorised:

Don’t test with more than 128 connections. You will get read errors. This is due to the file descriptor limit applied to each process on macOS. As @johannesweiss mentioned earlier the default for this is 256. You can change this but it involves disabling the System Integrity Protection.

Adam Fowler, Swift Forums thread

The 128 connections & read errors parts were spot on, in hindsight. But the rest was incorrect (it’s not about the file descriptor ulimit) and in particular the incorrect statement about having to disable SIP perhaps further distracted readers (corrections were posted in reply, which perhaps steered the thread away from what actually mattered).

I’m not sure what precisely the lesson is here… if Adam had better understood the behaviour he’d seen previously (re. 128 connections being the apparent limit) he might have been able to immediately point out one of the key problems. But who can say why he didn’t quite understand that limit correctly, or whether he should have. This sort of thing happens, and maybe it suggests a failure to properly diagnose problems previously, but mostly I’d just point out that the discrepancy here – between 128 and 256 – should have been noticed, and had it been questioned it would have accelerated progress towards the root cause.

Speaking just for myself, I think I (erroneously) dismissed Adam’s comment because I already knew that the default file descriptor limit is not actually 256 (it’s 2,560 on macOS, mostly) and so I assumed the whole comment was wrong and irrelevant.

Another clue was put forth, yet again essentially by accident (without understanding its significance, at the time):

Yes, the reason I used wrk, is that it uses pipelining. That’s why ab (apachebench) had such terrible performance: it opened a new socket for each request. And then it overloaded the system by throwing

• socket: Too many open files
• apr_socket_recv: Connection reset by peer 

errors.

I raised the ulimit -n to 10240, but still apr_socket_recv: Connection reset by peer (104) occurred occasionally.

Axel Roest, Swift Forums thread

This hinted very directly at the second major problem, but it seems nobody in the forum thread realised it. I think there was still a pre-occupation with the file descriptor ulimit.

A little logic applied at the time of Axel’s comment should have revealed its mistaken presumption: that opening new TCP connections for each HTTP request will inevitably cause connection failures. Sure, it will if you give it enough concurrent connection attempts, but real-world web servers operate at huge loads that are basically one HTTP request per connection, without any significant reliability problems. In hindsight, it’s clear that Axel’s dismissal of this behaviour as in any way normal was a mistake – as was everyone else in the thread going along with that dismissal.

A misunderstood workaround

In parallel to all of the above discussion in the Swift Forums thread, I’d been diving into wrk to see what it was really doing. I discovered a way to eliminate the errors: by opening all the TCP connections in advance in a way that happened to limit how many were attempted concurrently by wrk thread count which happened to be low enough in my use of wrk to not hit the magic 128 limit (more on that later). As you can see in my forum post on this, I initially misunderstood how wrk functioned and misattributed the root cause as bugs / bad design in wrk.

In my defence, wrk isn’t written very well, eschewing such outrageous and bourgeois software engineering practices as, you know, actually checking for errors. So it wasn’t unreasonable to believe it was ultimately just broken, given plenty of evidence that it was at least partly broken (which it was & is), but it was ultimately a mistake to let that cloud my judgement of each individual behaviour.

Then again, if I hadn’t been so appalled by the bad code in wrk, and taken it upon myself to rewrite key parts of it, I might not have stumbled onto the above “fix” and therefore also not found the true cause, later.

It’s never the compiler or the kernel… except when it is

At the time I did think I’d actually fixed wrk; I didn’t realise I’d merely found an imperfect workaround. I’d solved the connection errors (not really)! But, I was still curious about one thing – something pretty much everyone had kinda ignored this whole time:

Though those lingering few read/write errors still bother me. I might look into them later.

Me, Swift Forums thread

Tracing those reported errors to their cause was quite a challenge. The only known way to reproduce the errors was to use a very high number of concurrent TCP connections (several thousand), which made it hard to follow any single connection through its lifecycle using any low-brow methods (printf debugging etc). I eventually managed using System Trace² (lamenting, the entire time I used Instruments, that it would have been so much easier in Shark).

Unfortunately, what I was seeing – while in fact correct – did not make sense to me, so I was hesitant to take it on face value.

The lack of any error reporting on the server side, because Vapor lacks it completely, was also both a known problem at the time and also a problem in hindsight. Had Vapor/NIO actually reported the errors they were encountering, it would have partially validated what I was seeing in the system traces – in fact, it would probably have saved me from having to capture & analyse system traces.

Alas I don’t actually remember now precisely what led me to the final answers and root causes. I know it involved many hours of experimenting, exploring hypotheses, and in generally fiddling with everything I could think of.

Somehow or other, I did finally cotton on to a key configuration parameter: kern.ipc.somaxconn.

That controls how many connection requests can be pending (not formally accepted by the server) at one time. It defaults to 128 on macOS. Remember that number, 128?

Once I had figured out that kern.ipc.somaxconn directly controlled the problematic behaviour, the rest followed pretty naturally and quickly – I realised that what I saw in the system traces was in fact accurate, and that in turn revealed that the macOS kernel contains multiple surprisingly blatant and serious bugs (or at the very least dubious design choices, and lying documentation) regarding TCP sockets in non-blocking mode. I wrote that up in some detail in the second half of this Swift Forums post.

As a sidenote, that darn magic number that everyone kept ignoring – 128 – cropped up yet again, in the listen man page, though by the time I saw it there it was merely a confirmation of what I’d already discovered, than a helpful clue. Still, perhaps there’s a lesson there: read the man page. 😆

Conclusion

All told, the major problems identified by the benchmark were (and not all of these were mentioned above, but you can find all the details in the Swift Forums thread):

• The particular 3rd party library used for BigInt support in Swift, attaswift/BigInt, performs quite poorly.
• Vapor would accept too few connections per cycle of its event loop (promptly fixed, in 4.96.0).
• The benchmark tool used, wrk, has numerous bugs:
  • It doesn’t always use the configured number of concurrent connections.
  • It doesn’t measure latency correctly.
  • It doesn’t report errors correctly (in the sense both that it miscategorises them, e.g. connect vs read/write, and that it doesn’t provide enough detail to understand what they are, such as by including the errno).
• The macOS kernel (and seemingly Linux kernel likewise) has multiple bugs:
  • Connection errors are reported incorrectly (as ECONNRESET or EBADF, instead of ECONNREFUSED).
  • kqueue (kevents) behaves as if all connections are always accepted, even when they are not. Put another way, you cannot actually tell if a connection was successful when using non-blocking sockets on macOS.
• Key network configuration on macOS & Linux is way too restrictive:
  • Maximum file descriptors per process is only 2,560 generally on macOS, and even less (256) in GUI apps. It may vary on Linux, but on Axel’s particular server it was 1,024.
  • Maximum number of unaccepted connection requests (the kern.ipc.somaxconn sysctl on macOS, /proc/sys/net/core/somaxconn on Linux) is only 128 on macOS. It may vary on Linux.

It appears that the kernel bugs apply to Linux as well (although it’s not known if kqueue was in use there, as wrk also supports epoll and select), as the behaviour seems to be the same between macOS and Linux.

With the above issues fixed or worked around, his benchmark produces more explicable results (but keep in mind that the difference in connection acceptance behaviour is real and reflects a different design trade-off in Vapor, which may be a problem for real-world use if you don’t raise somaxconn and the listen backlog limit enough).

And that’s all just the problems Axel’s benchmark surfaced – there was a whole host of other interesting lessons taken away from all this (only a fraction of which were highlighted in this post – many more can be found in Axel’s posts and the Swift Forums thread).

Nominally the end result is also a benchmark that shows Vapor (Swift) out-performing other popular web frameworks in other languages. Hugely out-performing them, if you factor in not just throughput & latency but RAM & power usage. But, to reiterate what I pointed out earlier, take that with a grain of salt.

So, for a benchmark that many initially decried as unrealistic or plain poorly conceived, it turned out to be pretty darn useful, I think. And if that doesn’t make it a successful benchmark, I don’t know what does.

Addendum: post title

Looking at the comments about this post on HackerNews etc, I feel like I have to explain the title a little. I was quite pleased with myself when I came up with it (admittedly by accident), because it’s subtle and I think kinda clever, but perhaps too subtle.

“Swift sucks at web serving… or does it?” is a [platonic] double entendre.

On face value it’s alluding to the more typical type of post that is both (a) click-baity and (b) a standard “turns out” story where actually Swift is awesome at web server and haha to all those who doubted it. (where one can replace the word “Swift” with basically any programming technology, because benchmarking brings out some ugly competitiveness from the community)

But really what it means here, if you read the whole post, is that actually we still don’t know. It’s alluding to the oft-overlooked fact that benchmarks are rarely as conclusive as they’re presented. Which I thought was quite clever because it reiterates, at a meta level, my whole point about learning being more important than competing.

At least, that was the idea. 😆

1. Helidon itself is written in Java, but Axel used Kotlin for his little web server implementation – including most crucially the Fibonacci calculations. Both interoperate atop the JVM and plenty of “Java” libraries are partly written in Kotlin, or have dependencies written in Kotlin – and vice versa. A little like Objective-C and Swift interoperate such that many Mac / iDevice apps use a rich mix of both and you don’t typically need to care which language is used for any particular piece. ↩︎
2. I always endeavour to link to the things I mention, but in this case there’s nothing to link to – Apple don’t provide any actual documentation of the System Trace tool in Instruments, and there’s not even any usable 3rd party guide to it, that I can find. It’s a sad demonstration of Apple’s general indifference to performance tools. 😔
   
   Apple don’t even have a proper product page for Instruments itself – the closest you can find is merely its Help. ↩︎

var previous = UIntXL(0)
var current = UIntXL(1)

print(previous)
print(current)

while true {
    let next = previous + current
    previous = current
    current = next

    print(next)
}

Short of using a different algorithm entirely (there are much smarter ways to calculate Fibonacci numbers), you’d think it’d be optimally fast – I mean, how can the above example get any better, really?

Well, by doing this:

var previous = UIntXL(0)
var current = UIntXL(1)

print(previous)
print(current)

while true {
    previous += current
    swap(&previous, &current)

    print(next)
}

Sometimes this version is significantly faster (e.g. 30% in a recent example, and I’ve seen up to 14x in some of my own, similar cases).

Admittedly, sometimes it’s merely as fast – sometimes the Swift compiler optimises the first version into the second for us. But the compiler’s optimiser is unreliable and unpredictable. So if performance is important to you, it’s best to use swap explicitly rather than hang your hopes on the compiler.

What does swap do?

Its purpose is pretty obvious – it swaps the contents of two variables. Logically it’s equivalent to¹:

func swap(_ a: inout T, _ b: inout T) {
    let tmp = a
    a = b
    b = tmp
}

But that’s not how it’s actually implemented. The implementation starts here, though the pertinent details are hidden behind compiler built-ins. Suffice to know that it boils down to simply swapping the raw bytes.

Why is that better?

It avoids unnecessary work: temporary object initialisation and deinitialisation, memory allocation (and frees), reference-counting, etc. That can be from the more efficient swap itself, and/or from mutating one of the values in place rather than creating a temporary third value.

It can also prevent unnecessary copy-on-write behaviour, if you’re mixing the swap in amongst other mutations on value types that implement reference semantics² (like the standard Array, Set, etc). That usually saves time but can also, in some relatively rare circumstances, save memory too (by not leaving duplicate copies of internal buffers in memory indefinitely).

It also works for ~Copyable types without any extra effort.

Is it always optimal?

For actually swapping two values in RAM, yes.

Well, maybe. Mostly? It turns out – after I initially published this post – that it’s a bit more complicated than that. Sometimes an alternative method is slightly faster: the “tuple swap”. e.g.:

var previous = UIntXL(0)
var current = UIntXL(1)

print(previous)
print(current)

while true {
    previous += current
    (previous, current) = (current, previous)

    print(next)
}

In some benchmarks that’s faster, in some it’s slower³. It seems to always be faster than using a named temporary, just like swap, although I don’t believe that’s guaranteed behaviour. Furthermore, it relies on the compiler successfully recognising that particular expression of the swap intent and optimising it appropriately. Plus, using swap is shorter, clearer, and less error-prone. So I think an explicit call to swap is still the best option.

In any case, there are sometimes faster methods which eliminate the need to actually swap the bytes around. e.g. using a pointer to toggle between the two values. To my knowledge, Swift doesn’t provide any way to actually do that for value types, since Swift tries to pretend that pointers don’t exist for them. 😔

Hypothetically the optimiser could perform that optimisation for any code like the above examples, and others, although I’ve never seen it do that.

1. Ignoring support for ~Copyable types, which can be done but requires a more complicated implementation than we need to worry about here. ↩︎
2. Many value types in Swift are not actually just simple sequences of bytes. They often contain pointers to potentially-shared reference objects (e.g. classes or actors), or raw memory buffers. When you copy the value – which includes simply assigning it to a new variable – there can be a lot of work involved in incrementing reference counts for these now-shared internal objects & buffers (which also means time spent decrementing those counts some time later).
   
   And that’s assuming a type that uses copy-on-write (or similar) optimisations – otherwise, you’re going to have to copy all those internal memory buffers as well, which can be tremendously expensive.
   
   Even mere reference counting operations are actually pretty expensive – tens to hundreds of instructions each, typically. Usually in Swift we don’t notice them because the optimiser works very hard to actually remove them, as much as possible. ↩︎
3. I put together some benchmarks in researching & writing this post, with the intent of using them as specific examples of the performance differences. Unfortunately they are apparently too simple, and the optimiser is generally able to optimise the named temporary method into using swap or the “tuple swap” method (even though in real-world code it usually fails to do so, in my experience). ↩︎

Usage is hidden

In leasing / colo arrangements electricity cost is often flat-rated, and/or bundled up with many other costs.

Many hosts don’t provide electricity usage reporting at all. And those that do sometimes provide it only obliquely as the nominal dollar cost, not in terms of actual Joules used (typically expressed as kWh or MWh, which are just different units for the same measure).

If you build & operate your own datacentre, you do get to see your actual usage – at least in terms of power (Watts) and energy (Joules) if not more holistic cost – but very few people have any actual experience with that.

The sticker price is not the cost

Most of us have no idea what electricity actually costs in any context, even for our own residential use at home, or for electric car charging. Practically nobody pays the actual cost of electricity up front (more details). The full cost is usually many times larger than what is claimed on the bill from the utility provider. It’s largely disaggregated (mainly as air pollution & greenhouse gas emissions) and often grossly unfair (e.g. loss of lives due dam failures & nuclear accidents, health harms geographically localised around power stations or around rivers downstream, etc).

This is broadly true for all sources of electricity, “green” or not, although it is dramatically less-so for some forms of renewable energy (particularly photovoltaics and wind).

Electrical power is mostly not an operating expense

It’s often presumed that electricity is only an operating expense, i.e. you pay a power company per kWh and that’s it. That’s actually a small fraction of the cost – most of the cost is in capital expenses. Even factoring in the true cost of power, i.e. minus subsidies and including externalities, electricity operating expenses are still only a fraction of the cost.

Every bit of electricity you want to put into a datacentre requires infrastructure to handle it. The more energy you want to put in, the more it costs to build that capacity (economies of scale help dampen this impact only a little). Transmission lines, substations, transformers, power distribution throughout the building, rectifiers, backup systems (batteries, fly-wheels, emergency generators), etc.

And every bit of power in means heat out, which has to be removed, which takes yet more energy & money. Baffles and cabinets, fans, radiators, plumbing, heat exchangers, heat pumps (air conditioners) and evaporative coolers, etc. It’s not quite as bad as the tyranny of the rocket equation, but in the same vein.

If you look at a datacentre, physically, most of it is actually about power (and the flip-side, heat). The servers themselves take up a minority of the space and actually cost less than the infrastructure that supports them.

Power costs are dominated by provisioned power, not actual usage

You pay for power infrastructure whether you use it or not.

If you operate at your power capacity all the time, then you at least have a technically efficient system in that respect.

Almost no datacentres (nor servers individually) operate that way. Most are actually operating at way below full load, pretty much all the time. And that means you’ve paid a pretty penny for your power capacity and yet you’re wasting most of it (and therefore most of your money).

The reasons they usually operate well below capacity are myriad, but in short the keys are:

• Bad software (encompassing everything from top level architecture to implementation details, and fundamental decisions like choice of programming languages).
  
  This is (in my personal estimation) by far the dominant factor.
  
  This is a huge subtopic in its own right, which I’ll expand upon in a future post.
• Unused space. Whether space not yet filled with servers (because installation and growth takes time, and you usually want some spare capacity at all times), or space temporarily bereft of operating servers due to equipment upgrades and the like.
• Poor architecture amounting to (in a nutshell) insufficient over-provisioning.
  
  Server workloads typically aren’t stable but usually are predictable – e.g. a diurnal cycle because most people are asleep at night rather than using your services, weekends differing from weekdays due to the typical western work week, etc. So even with good software it can be impractical to achieve complete utilisation of hardware. In that case, you can still save money by recognising that you don’t need every server in your datacentre to operate at peak power simultaneously – you can essentially move your power capacity around based on moment-by-moment need.
  
  In a simplistic homogenous environment – where every server is identical – this is somewhat pointless, but most non-trivial datacentres are not like that; they have not just many generations of servers, but also different kinds of servers (web servers vs storage servers etc). During peak traffic you might focus your power on your web servers, but when traffic ebbs you can shift that power to your batch-job servers.
  
  Arguably you don’t need to over-provision your hardware if you have a good software architecture, e.g. if you can run any job on any machine and move jobs fluidly, and thus can always efficiently bin-pack your machines. That’s practically never perfectly the case, although you can get usefully close – nonetheless, which approach is fundamentally the best remains a contentious topic and a subject of reasonable debate within server architecture circles.

And yet, when trying to screenshot things with a pixel-perfect even border around them, I’ve been using trial-and-error in order to get the start point right. Like a caveman.

In taking a screenshot just now, I accidentally hit the option key while dragging out my target area, and discovered that it lets you change the bounds in all four dimensions, not just the bottom & right!

🤯

You can also hold down shift to constraint changes to one of the cardinal directions (horizontal or vertical). Which of course combines neatly with holding down option.

These are very standard drag modifiers in graphics applications, which I’ve known about and used for decades. Yet I had no idea they work with macOS’s built-in screenshotting system.

How on earth did I not know this existed?! Was it only recently added?

$file:t → Work.zip

$file:h → /Users/me/Documents

$file:r → /Users/me/Documents/Work

$file:e → zip

You can also compose these arbitrarily, e.g.:

$file:t:r → Work

A full list of expansion modifiers is in the Zsh Modifiers documentation.

Kudos to Zaiste Programming for clueing me into this.

I studied them simultaneously, receiving degrees in both after five years (some classes counted towards both, thus why it wasn’t seven or eight years).

I grew up playing on, dabbling with, and programming computers. From BASIC II to Hypercard to RealBasic and onwards. I don’t recall ever questioning the apparent inevitability of going to university to study computer science, and then into a permanent career in software development. It was as if I were born to it.

In contrast, I chose to do electronic engineering basically on a whim, while I was choosing my university course. I had effectively zero background in it. It just looked interesting.

Though I observed the curious results – the unexpected inversion in my academic grades – it took me quite a while to learn the lesson. I did better in EE despite my inexperience because I paid attention. It was interesting and I knew I was starting from scratch there, so I worked hard at it. I gave it more time.

I’ve increasingly appreciated the importance of this as I’ve gathered other life anecdotes. Why does the new hire straight out of school often do a better job than the senior engineer? Why do “fresh eyes” on a long troublesome area of the project suddenly find all sorts of bugs and flaws that had been overlooked?

I was originally doing something like:

ffmpeg -ss 00:07 -t 00:03 -i "IMG_0164.MP4" -async 1 -c copy "IMG_0164_TRIMMED.MP4"

The resulting trimmed MP4s play just fine in Quicktime, the Finder – anywhere that uses Apple’s decoding libraries (though I didn’t test iOS).

However, in VLC, or Lightroom, the audio is completely corrupt – just incoherent noise.  In Lightroom the video doesn’t even play correctly, because of Lightroom’s stupid habit of re-encoding the video & audio into internal caches – apparently their video decoder is somehow thrown off by the audio channel issues, too.

After much trial and error and many dead-ends (thank you completely bogus & wrong Stack Overflow threads… sigh) I eventually realised that the problem is apparently simply that Lightroom, VLC, etc get offended when you include pcm_s16le audio in an MP4.  ffmpeg itself says that’s not a valid audio codec for the MP4 container, iff you explicitly tell it to use that as the codec.  If you’re just copying from an existing audio / video file, however, it makes no mention at all of the concern.  Sigh.

So the apparent solution is simply to switch to the MOV container format instead.

ffmpeg -ss 00:07 -t 00:03 -i "IMG_0164.MP4" -async 1 -c copy "IMG_0164_TRIMMED.MOV"

The encoded bits remain identical, but the MOV container apparently accepts PCM audio where MP4 does not.  VLC, Lightroom, etc are now happy (and Quicktime et al remain happy).

(another possibility is that the ‘incompatibility’ is related to MP4 levels or some other such junk… I didn’t try deciphering or exploring that)

It’s frustrating that VLC & Lightroom can’t handle this when clearly it’s technically possible (witness Quicktime), and worse they don’t even properly recognise that they’re not handling it properly – they just play completely corrupt audio that’s literally painful on the ears.

It’s also very curious that the trail camera uses PCM audio if that’s not valid in an MP4 container.  It’s downright bizarre that VLC & Lightroom can play the unmodified MP4s straight from the trail camera, even though they use the same purportedly invalid audio codec… somehow something ffmpeg is doing during its transmutation is making them angry.  I was unable to determine what that might be, though, through trial-and-error with ffmpeg command line options & rudimentary examination of the input & output files.

P.S.  An alternative is to bitwise-copy only the video stream (i.e. change -c copy to -c:v copy), and let VLC transcode the audio into its default AAC for the MP4 container.  That probably wouldn’t be a problem for me in my case – the audio from trail cameras is pretty crappy to begin with – but at the same time the audio tracks in these files are insignificant in size, so re-encoding them (and lossy as AAC) is pointless to saving disk space.

Process: Mail [19884]
Path: /Applications/Mail.app/Contents/MacOS/Mail
Identifier: com.apple.mail
Version: 11.3 (3445.6.18)
Build Info: Mail-3445006018000000~4
Code Type: X86-64 (Native)
Parent Process: ??? [1]
Responsible: Mail [19884]
User ID: …

Date/Time: 2018-04-19 16:44:45.717 -0700
OS Version: Mac OS X 10.13.4 (17E199)
Report Version: 12
Anonymous UUID: …

Sleep/Wake UUID: …

Time Awake Since Boot: 94000 seconds
Time Since Wake: 530 seconds

System Integrity Protection: enabled

Crashed Thread: 13 Dispatch queue: Task Manager Serialization Queue (QOS: UNSPECIFIED)

Exception Type: EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY

Application Specific Information:
*** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'This method should only be called on the serialization queue'
terminating with uncaught exception of type NSException
abort() called

Application Specific Backtrace 1:
0 CoreFoundation 0x00007fff55da96bb __exceptionPreprocess + 171
1 libobjc.A.dylib 0x00007fff7d4c1942 objc_exception_throw + 48
2 CoreFoundation 0x00007fff55daf2a2 +[NSException raise:format:arguments:] + 98
3 Foundation 0x00007fff57ee7340 -[NSAssertionHandler handleFailureInMethod:object:file:lineNumber:description:] + 193
4 IMAP 0x00007fff6fe88959 -[IMAPTaskManager secondaryIdleMailboxName] + 216
5 IMAP 0x00007fff6fe880cb -[IMAPTask mailboxIsUserVisibleUsingDataSource:] + 180
6 IMAP 0x00007fff6fe6ab68 -[IMAPMailboxSyncTask _nextNetworkPriorityAndOperation:reservedNetworkPriority:] + 164
7 IMAP 0x00007fff6fe6c4ba -[IMAPMailboxSyncTask recalculatePriorities] + 398
8 IMAP 0x00007fff6fe67dd9 -[IMAPMailboxSyncTask initWithDataSource:taskManager:imapMailbox:fromStatus:forceFullSync:] + 766
9 IMAP 0x00007fff6fe931f7 -[IMAPTaskManager _syncMailboxWithDataSource:withIMAPMailbox:fromStatus:forceFullSync:userInitiated:] + 370
10 IMAP 0x00007fff6fe92e68 -[IMAPTaskManager syncMailboxWithDataSource:withIMAPMailbox:fromStatus:forceFullSync:userInitiated:] + 240
11 IMAP 0x00007fff6fe9631a -[IMAPTaskManager didAddMessagesWithUnknownUID:toDataSource:] + 872
12 Foundation 0x00007fff57e4a5df __NSBLOCKOPERATION_IS_CALLING_OUT_TO_A_BLOCK__ + 7
13 Foundation 0x00007fff57e4a441 -[NSBlockOperation main] + 68
14 Foundation 0x00007fff57e488ee -[__NSOperationInternal _start:] + 778
15 Foundation 0x00007fff57e44917 __NSOQSchedule_f + 369
16 libdispatch.dylib 0x00007fff7e09ee08 _dispatch_client_callout + 8
17 libdispatch.dylib 0x00007fff7e0b1ed1 _dispatch_continuation_pop + 472
18 libdispatch.dylib 0x00007fff7e0a9783 _dispatch_async_redirect_invoke + 703
19 libdispatch.dylib 0x00007fff7e0a09f9 _dispatch_root_queue_drain + 515
20 libdispatch.dylib 0x00007fff7e0a07a5 _dispatch_worker_thread3 + 101
21 libsystem_pthread.dylib 0x00007fff7e3f0169 _pthread_wqthread + 1387
22 libsystem_pthread.dylib 0x00007fff7e3efbe9 start_wqthread + 13

Long story short, the issue turns out to be having connection logging enabled.  That’d been turned on many months before in order to debug a different stupid Mail bug, and had been simply left on (deliberately IIRC, since Mail tends to bug-out quite often, so why not have logs already available when it comes time to debug it yet again?).

Connection logging is enabled or disabled by opening the “Connection Doctor” (Window menu > Connection Doctor) and using the checkbox titled “Log Connection Activity”.

So how do you get to that checkbox when Mail crashes on launch?  Well, in this specific instance I was able to disable all mail accounts via System Preference’s Accounts pane, launch Mail, disable logging, quit Mail, re-enable all mail accounts via System Preferences, and then relaunch Mail to have it finally actually work.

From even just brief web searching, it’s clear that this issue has been present and well-known in Mail for a really long time.  Sigh.  Apple’s protestations that they care about software quality, or the Mac, are relentlessly undermined by their actual actions.

• The Hotline File Transfer Protocol v1.1.1.  I presume I was interested in, or actively doing, a third party Hotline client.  I did tend to make lots of data transfer clients back then (e.g. HTTP, FTP, even POP3 & SMTP).
• My little gallery of childhood toys & memorabilia.  I hadn’t forgotten about this per se, but it’d certainly been a long time since I’d look at it.  In hindsight I’m really glad I took the time to take these photos – most of this stuff is long gone now, so the photos are all the remains to pique my nostalgia.
• Apparently I discovered some ‘fix’ for Apple breaking dial-up modems in a Mac OS X Jaguar (10.2.4) update.  I vaguely recall that, though I’m pretty sure that’s not the only time I’ve had to fix stupid regressions in Mac OS X by reinstalling frameworks, kexts, etc from a prior OS build.  Sigh.
  • Here’s a second reference to the same thing.
• I’m listed in the National Park Foundation’s 2015 donor list.  I’m also in the 2016 one too, though it doesn’t show up in a web search [yet], and expect to be in the 2017 one as well.
  • If you’re interested, here’s the many ways you can contribute to the NPF too.  Not listed prominently, but relevant for others in the tech industry, is that you can donate stock without paying capital gains on it, while still receiving the tax benefits of the stock’s full, current market value.  I wish I’d known this years ago when I started doing non-trivial charitable giving. 😔
• Steven Troughton-Smith gives a call out to me (among others) in his little SceneKit demo program and his pretty impressive “OpenWorldTest” (i.e. Minecraft tech demo clone).  I don’t recall have any specific contribution, but way back when Steven & I did chat a bunch about SceneKit, and other topics.
  • It tickles me now, though I haven’t talked to Steven much in years, that his name appears increasingly often in pretty high profile places (e.g. the ATP podcast mentions him almost every episode lately).  He’s certainly made a name for himself.
  • Someone called “Taka Taka” has also run with OpenWorldTest a bit, to add VR support among other things.  Nothing to do with me any more than Steven’s original version – I mention it just because it’s cool.
• I’m cited in the USF Maritime Law Journal (alas a broken link as I write).  Yep, I’m a big mover & shaker in the legal circle… which is to say, Marisa had an article published there and thanked me for supporting her as she wrote it. 😉
  • And I get a mention on every single page on her website too, for taking one of the photos of her on it.  Booyah!
• In reference to my hacking on CoreGraphicsServices, Nat! (no other name given, as far as I can see) wrote a journal entry about how some people in the Apple community can be, well, wet towels.
• A bajillion years ago I wrote the Keychain Framework, a relatively clean ‘Cocoa’ (Objective-C & Foundation) framework that wrapped Apple’s C-based libraries for keychain access & basic security functionality (which is in turn an implementation & based on the CDSA ‘standard’ that nobody but Apple appears to have ever actually implemented).  Anyway, apparently it’s used in SQLEditor.  I’m pretty stoked – I sunk a spectacular amount of time into that framework, for very dubious benefit in hindsight. 😕
• And another old pet project that shows up is my Mailcash plug-in for Apple Mail.  This was an implementation of Hashcash, which I still think is a neat idea for reducing email spam, by making senders ‘pay’ for every email like a virtual stamp.  I have no idea if anyone ever really adopted it, nor if my Apple Mail plug-in still works at all.  Presumably not – I recall Apple Mail breaking plug-ins repeatedly over the last ten years or so, since I wrote Mailcash.
  • Oh how easy it is to date an open-source project when it’s still hosted on Sourceforge.  Poor, sad, lost-its-way Sourceforge.

Curiously, there’s several other open-source projects of mine on Sourceforge that don’t show up in a web search for my name:

• Build Installer, a template for software installers that build the software from source on the user’s machine – assuming they’ve installed Apple’s developer tools, that is.  From memory this was inspired by the tedious and error-prone nature of various open-source projects, w.r.t. how they distributed & installed their software on Mac OS X.  Sadly, nothing much has changed – there’s now brew & such package managers, but in my experience they make just as a big a mess as doing it manually & ad-hoc did.
• DePC, a tool for stripping files of those nasty DOS-based file extensions, and instead setting the files’ type & creator codes correctly.  Sigh… I lost that battle.  To this day I still think that was a bad choice, that Apple made, to cave in to file extensions.  They’re still ugly and error-prone.
• Mission to the ISS, the horrible group project I did with various folks in university, for the CSE32PRO (3rd-year Computer Science Project) class.  Back in 2004, I’d guess, based on the open-source release being in January 2005.  That was a fun project in many ways, though the end result was a bit embarrassing – we ran out of time, during the class, to actually finish it properly, so for example the underlying algorithms that control the simulation are fundamentally step-based, but are stepped every time the user provides any input, so high user interaction makes the simulation run faster than intended, with silly and sometimes outright broken results.  Sad panda.

Luckily I’m far from the first person to need to create RAID and/or encrypted CoreStorage volumes, in El Capitan.  Florian Knapp has a concise summary of how to set up an encrypted RAID volume.  Tom Nelson (of About.com) has a slightly more detailed tutorial for managing the RAID part.

Now I just wish the hard drive industry would actually push capacities up, like they once did, so that I don’t have to resort to striped RAID sets just to make a disk big enough for Time Machine backups.  It feels like we’ve been effectively stuck at 6 TB for many years now, and affordable 8+ TB drives aren’t really on the horizon (Seagate & Western Digital have offerings, but historically have been bad brands for drive reliability, e.g. Backblaze’s data, plus my own personal experience with their drives).

An internal state error occurred
Operation failed…

I guess my reasons for signing up include:

• A curiosity as to whether I can still handle a traditional academic learning environment.  I have warm, fuzzy nostalgia for my uni days, for the most part.  And when I was at uni I was nostalgic over high school…. ad infinitum, I suspect.
• Determining whether I have any meaningful interest in furthering my academic credentials in future (I’ve toyed with the idea of getting a Masters or PhD from Stanford for a while; if it weren’t so damn expensive I probably have started by now… maybe this will convince me it’s worth the cost).
• A desire for betterment, and a realisation that a formal schedule and a familiar paradigm will be more likely to stick, long-term, than haphazard or irregular approaches.

At first glance I was tempted to sign up for a whole lot of the courses – basically all that looked at all interesting – but I decided against it in the end, reasoning that it wouldn’t do any real harm to just take one to begin with, and see what the workload really is, and how much I enjoy it.