For the price of one Apple Pro Display XDR you can get four Asus ProArt 6k displays.

I suspect there’s only three 6k panel models in existence – the one used by Apple, the AUO one used by Asus², and the LG one used by LG, Dell, & Kuycon.

It’s strange to me that Dell haven’t dropped the price of their 6k display given that LG are offering the same panel in a much svelter package for 15% less (and you can get the very similar Asus display for 45% less!).

1. Apple don’t appear to have ever published a brightness spec for the original Retina iMac, but Tom’s Guide measured their review model at 382 lumens.
   
   The comparison to the original Retina iMac is apt because it was the first of the retina [desktop] displays, of which these 6k displays are all members. It also had the exact same pixel density – 218 PPI – as the Apple and Asus displays – i.e. each individual pixel is the exact same size – so it’s a very fair point of comparison despite the overall differences in resolution. ↩︎
2. Speculation is that the Asus uses an AUO panel (the same one that’s popular with Chinese display manufacturers, because it’s very cheap). I wouldn’t normally give this much weight, except it is interesting that both the Asus and the Chinese displays are frequently reported as making an annoying whining noise at certain brightness levels, and have identical panel specifications, and were released at similar times (notably, many years after the Pro Display XDR). ↩︎
3. This is what LG states in the monitor’s specifications. Yet, LG also states that this monitor is DisplayHDR 600 certified, which means it’s required to have a static contrast ratio of at least 8,000 : 1. ↩︎
4. As with the LG, Dell states a 2,000 : 1 contrast ratio even though DisplayHDR 600 conformance requires at least 8,000 : 1. ↩︎
5. Asus states a 1,500 : 1 contrast ratio even though DisplayHDR 600 conformance requires at least 8,000 : 1. Though they also describe the 1,500 : 1 as “typical” while also listing 3,000 : 1 as the maximum. ↩︎
6. LG says “typical” brightness is 450, with minimum being 360, without explaining the difference – e.g. whether that’s for a white patch vs full-screen white, or perhaps depending on ambient temperature. I’m choosing to be generous and assume it’s merely a temperature thing, and not likely to be a concern in a typical indoor environment, because 360 is ridiculously dim. ↩︎
7. LG don’t explicitly state this, but it’s a requirement of the DisplayHDR 600 conformance. ↩︎
8. Dell don’t explicitly state this, but it’s a requirement of the DisplayHDR 600 conformance. ↩︎
9. Asus don’t explicitly state this, but it’s a requirement of the DisplayHDR 600 conformance. ↩︎
10. Apple don’t state the actual coverage – just vaguely reference the various colour gamut standards – so these are the figures as actually tested by PCMag. ↩︎
11. Only when used with Macs which support DSC (Display Stream Compression), otherwise the USB-C ports are limited to USB 2.0 (400 Mb/s). ↩︎
12. The tech specs claim it has a built-in KVM, but there’s no mention of that anywhere else in the marketing materials nor the user manual. It does have a USB hub, which you can manually switch between the Thunderbolt or USB upstream ports by diving into the on-screen display, but there’s no apparent support for (a) doing this automatically when the input source changes nor (b) switching input sources & USB routing via keypress. ↩︎
13. Meaning the simple product of the three maximal dimensions. Some of these displays have curved backs, so their actual volume will be substantially less. ↩︎

The externalities cost estimates might be a little tongue-in-cheek, but honestly, are they all that wrong? One small irritation at the wrong moment can ricochet my happy mood off into the doldrums, and Apple’s products produce a hundred “small” irritations every day – which compound in their irritation when you see them software update after software update, year after year, product after product. It’s hard not to take it personally. Like Apple is deliberately being cruel.

Daniel Kennett wrote, in his memorial to Aperture, about how the Mac-using experience wasn’t actually rainbows and perfection even back in whatever you personally feel was the golden age (I’m with him that circa System 7.1 was glorious, though the early days of Mac OS X were also very special to me). Which is true – fire up your favourite old Mac on Infinite Mac and see the strength of your rose-tinted nostalgia glasses.

But, the big difference is that Apple back then was a relatively tiny company struggling just to survive in a brutal industry dominated by soulless, greedy monsters.

There is a point at which mere indifference or incompetence transitions into negligence, and it’s long before you become one of the wealthiest companies on the planet with a veritable army of engineers.

Having worked at Apple – among other big tech companies – I can say with confidence that there’s no valid reason why they cannot fix long-standing, infamous bugs. It’s. Not. That. Hard. One half-decent engineer could fix everything listed on Bugs Apple Loves in six months, single-handed.

It’s not apparent why that doesn’t happen, but it’s not that Apple are technically incapable of it, and it cannot be that they’re unaware, so it must be that they’re choosing not to.

But, looking over other people’s shoulders, and listening to the groans and cries, I’m struck by something. We’ve been here before. Icon jail, I mean (“Squircle” jail specifically in Tahoe’s case):

This isn’t the first time Apple’s put icons in grey jail. Mac OS 9 already did this:

This mode made the icons activate with a single click rather than a double-click. It was a window-specific (and desktop-specific) setting. I vaguely recall it being sold as a simpler, more efficient way of using the Finder – fewer clicks! It did not catch on, though possibly there were some fans. I vaguely recall encountering Macs in the wild that had this mode enabled.

It wasn’t new in Mac OS 9 either, really – it was actually introduced with At Ease circa System 7.1:

In a way, Apple is just returning to its roots; continuing to try to dumb-down the Mac.

Of course, a big difference between all these prior incarnations and macOS 26 Tahoe’s is that previously the user controlled it, and could turn it off. You didn’t have to install At Ease at all. Mac OS 9 didn’t use the Buttons view mode by default, you had to enable it, and anyone could disable it again at any time.

That’s where today’s Apple actually differs from yesteryear’s – they used to respect the user, and now they don’t.

• Permission errors with the destination folder (e.g. not writable by the current user).
• Conversion failures if you have the iPhone set to export images in “compatible” formats rather than their originals¹
• Unsupported image formats (e.g. HEIC) on very old versions of macOS (macOS 10.12 Sierra and earlier).
• The destination folder not existing.
• Etc.

Basically it seems like this “-9934” error is a catch-all for any kind of I/O error. Good job, Apple, in providing useful information to your users.

But add to the aforementioned list one other possibility, that I didn’t see mentioned anywhere: the destination volume is full.

I’ve been hitting this error message in Image Capture sporadically for months and been baffled by it – especially since it seemed to occur for seemingly random sets of images and videos, that were sometimes consistent between attempts and sometimes changed unpredictably. Which is of course the result of the destination volume being perpetually close to full but varying slightly over time as random bits & pieces are added or removed from it.

It’s especially tricky because the destination volume won’t typically show as actually full, in the Finder – it could have virtually any amount of space left, even hundreds of gigabytes, because the particular image or video that hits the limit could be huge, and Image Capture either tries to pre-allocate the entire file’s space up front (failing immediately if there’s insufficient space) or retroactively zeroes out the file if it encounters a write error. Either way, it leaves a bunch of zero-byte files (the failed transfers) behind, rather than incompletely-written ones, further hiding the fact that it ran out out of disk space.

1. Please don’t ever do that. It means transcoding your original images from DNG or HEIC into JPEGs, increasing their file size and/or reducing their quality in the process. Don’t do it. DNG and HEIC are old formats that are widely supported – not just on Apple devices, but Windows as well (and probably most Linux distros). ↩︎

Using their Explainfile tool to dig into the log files, the clue seems to be:

  - line 288 - 2024-12-16 16:16:17 0000000646 - ERROR: UpdateBzDoneRegardingFlsToBeExp - Z_B_TOO_MANY_CHUNKS bz_done_ line chunk related, numBytesInLargeFile=1099512156951, totNumChunks=104858, bz_done_line_is: 5	! …
  - line 522 - 2024-12-16 16:18:46 0000000646 - ERROR - bz_done_ INCONSISTENCY_FOUND - 20241216161846 - BadBadBadChunkRecord hexAsciiVal=0x78 - AfterBzdoneLargeFileAnalysis: chunkSeq=100001, highestChunkSeqSeen=104857, fileIdOfLargeFile=00000000002c53cd, dateTimeOfLargeFile=20231217091843, XYXBXXX_FILE_NAME: …

Admittedly I’m guessing somewhat, since that’s a rather reader-hostile log message, but the combination of the Z_B_TOO_MANY_CHUNKS error mnemonic and chunkSeq=100001 (because of its proximity to the arbitrary round number 100,000) strongly suggests that Backblaze is imposing a 100,000 chunk limit. Since chunks are 10 MB each, that’s exactly 1 TB.

This is unequivocally at odds with what they claim repeatedly on their website, on pages like What Backblaze Backs Up and File Sizes.

It’s not clear to me why this is suddenly a problem; is this a newly-imposed limit? It’s possible that a month ago I removed some exclusion on the file, but I don’t remembering doing that and I can see no reason why I would have excluded it to begin with. If it is newly imposed, that would imply it’s also retroactive – that Backblaze actually deleted the existing backup of the file from their servers, thus causing the client app to try uploading it again.

I reached out to their technical support, of course, but thus far have only received mindless responses – restart your computer, reinstall Backblaze, etc.

Update

I received no further response from Backblaze’s technical support. They asked me to send them the log files, which I did on January 2nd, 2025, and they never responded again.

As of this update (February 4th 2025) their website still falsely advertises support for files of any size.

Addendum

I was surprised to see that many folks on HackerNews were surprised by the idea of a 1 TiB file. I certainly agree that’s large, but it doesn’t seem unusual or inexplicable to me. In my case, this particular “problem” file is an encrypted, compressed disk image of the boot drive of a prior computer, that I saved when I upgraded to my current computer.

It’s true that I could probably throw it out at this point – it was just a precaution in case I forgot to migrate something over, so now (eight years later) it seems that either I made no such mistake or whatever I forgot to migrate doesn’t matter anyway. For now I’ve just manually excluded it from the backup, to work around Backblaze’s bugs.

There are other cases in which I’ve had files over 1 TiB, though – e.g. video files:

• With some cameras and recording modes (e.g. documentarian, interviews) it’s in principle easy to exceed 1 TiB per file. e.g. the Nikon Z9 & Z8 record around 700 MB/s for 8k60 N-RAW, which is about 24 minutes per TiB.
  
  Note that I don’t recall if I personally have ever actually exceeded 1 TiB this way. I mention it mainly for illustration. It’s also possible that the Z9 & Z8 shard large recordings into multiple files (I don’t recall seeing this in years – not since the 4 GiB per-file limit of cameras a decade ago – but perhaps I’ve just not had a single recording large enough).
• Usually (for me) it’s output files that are largest, since they can combine many clips. I use Final Cut Pro and its video compression capabilities aren’t great, so I export essentially lossless ProRes and then use ffmpeg or Handbrake for the real compression. ProRes 422 HQ is nearly a gigabyte per second for 8k60, so it takes less than twenty minutes of video to exceed 1 TiB. Fortunately these large intermediaries only have to live as long as the final compression takes (though that can be days, especially with the latest formats like AV1).

I’m very glad Apple’s Messages team somehow snuck this one little button past the corporate bureaucracy (Apple infamously likes to pretend their software is infallible and therefore doesn’t need things like useful error messages or rectification tools).

It’s of course not great that iMessages don’t reliably sync between Apple devices to begin with, but at least there’s a simple way to fix that when it occurs.

In case you’re unfamiliar with it, this button is in the “iMessage” tab of the Messages Settings – although placed dangerously close to the “Sign Out” button. 🤨

Nice gal Amanda

I submitted the following feedback, not really expecting much of it. In fact, I fully expected some useless, boilerplate response that completely ignored the point and directed me to the upload requirements page (despite it being the very first thing I myself pointed to) or was nothing but mail-merge platitudes.

And in retrospect I could have phrased this more respectfully, too. So I wasn’t even going in as a great customer.

What I got surprised me greatly, in the best way:

Holy shit. A humane response from a real human. Better yet, a couple of hours later that was followed by (unprompted by me):

Even though these responses made no conclusive promises (regarding support for these file formats), I was thrilled. Beyond the pleasant tone and clear demonstration that Amanda actually read and comprehended my feedback, just knowing that my feedback actually got to the intended recipients (the dev team) is heartening; it made me feel good about choosing to use Flickr for all these years.

But then…

Bad guy Marc

Coincidentally, while Amanda was getting back to me I ran into a bug in Flickr’s file upload API. It was apparent from the logs that Flickr was the cause of the issue, but nonetheless I took the time to first confirm that with the Photo Upload plug-in’s author. Then, I submitted the bug report to Flickr:

HTTP/1.1 504 Gateway Time-out
Content-Type: text/html
Content-Length: 1033
Connection: keep-alive
Server: CloudFront
Date: Thu, 18 Jul 2024 01:42:43 GMT
X-Cache: Error from cloudfront
Via: 1.1 CENSORED.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: CENSORED
X-Amz-Cf-Id: CENSORED

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>504 ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
CloudFront attempted to establish a connection with the origin, but either the attempt failed or the origin closed the connection.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
<BR clear="all">
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
<BR clear="all">
<HR noshade size="1px">
<PRE>
Generated by cloudfront (CloudFront)
Request ID: CENSORED
</PRE>
<ADDRESS>
</ADDRESS>
</BODY></HTML>

* Host up.flickr.com:443 was resolved.
* IPv6: (none)
* IPv4: 13.227.78.140
*   Trying 13.227.78.140:443...
* Connected to up.flickr.com (13.227.78.140) port 443
* ALPN: curl offers http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
} [315 bytes data]
* (304) (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* (304) (IN), TLS handshake, Unknown (8):
{ [25 bytes data]
* (304) (IN), TLS handshake, Certificate (11):
{ [4967 bytes data]
* (304) (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* (304) (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* (304) (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / AEAD-AES128-GCM-SHA256 / [blank] / UNDEF
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: CN=flickr.com
*  start date: Feb  5 00:00:00 2024 GMT
*  expire date: Mar  4 23:59:59 2025 GMT
*  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M02
*  SSL certificate verify ok.
* using HTTP/1.x
> POST /services/upload/ HTTP/1.1
> Host: up.flickr.com
> User-Agent: curl/8.6.0
> Accept: */*
> Authorization: OAuth oauth_signature="CENSORED",oauth_token="CENSORED",oauth_version="1.0",oauth_nonce="CENSORED",oauth_timestamp="CENSORED",oauth_signature_method="HMAC-SHA1",oauth_consumer_key="CENSORED"
> Content-Length: 123113221
> Content-Type: multipart/form-data; boundary=------------------------cbLPLbBocYE2oLsFDbSmbX
> 
} [233 bytes data]

  0  117M    0     0    0  128k      0   453k  0:04:25 --:--:--  0:04:25  452k
  5  117M    0     0    5 6848k      0  5670k  0:00:21  0:00:01  0:00:20 5669k
  6  117M    0     0    6 7360k      0  3068k  0:00:39  0:00:02  0:00:37 3068k
  6  117M    0     0    6 7360k      0  2165k  0:00:55  0:00:03  0:00:52 2165k
  6  117M    0     0    6 7488k      0  1780k  0:01:07  0:00:04  0:01:03 1780k
  7  117M    0     0    7 9280k      0  1773k  0:01:07  0:00:05  0:01:02 1848k
  9  117M    0     0    9 11.5M      0  1906k  0:01:03  0:00:06  0:00:57  997k
 12  117M    0     0   12 14.3M      0  2025k  0:00:59  0:00:07  0:00:52 1508k
 14  117M    0     0   14 17.4M      0  2175k  0:00:55  0:00:08  0:00:47 2182k
 17  117M    0     0   17 20.4M      0  2270k  0:00:52  0:00:09  0:00:43 2682k
 19  117M    0     0   19 22.8M      0  2295k  0:00:52  0:00:10  0:00:42 2843k
 21  117M    0     0   21 24.9M      0  2274k  0:00:52  0:00:11  0:00:41 2730k
 23  117M    0     0   23 27.1M      0  2279k  0:00:52  0:00:12  0:00:40 2648k
 25  117M    0     0   25 30.0M      0  2324k  0:00:51  0:00:13  0:00:38 2567k
 27  117M    0     0   27 32.6M      0  2342k  0:00:51  0:00:14  0:00:37 2472k
 29  117M    0     0   29 34.9M      0  2350k  0:00:51  0:00:15  0:00:36 2462k
 32  117M    0     0   32 38.0M      0  2399k  0:00:50  0:00:16  0:00:34 2679k
 34  117M    0     0   34 40.5M      0  2403k  0:00:50  0:00:17  0:00:33 2705k
 36  117M    0     0   36 42.6M      0  2400k  0:00:50  0:00:18  0:00:32 2601k
 38  117M    0     0   38 45.6M      0  2432k  0:00:49  0:00:19  0:00:30 2691k
 41  117M    0     0   41 48.5M      0  2456k  0:00:48  0:00:20  0:00:28 2780k
 43  117M    0     0   43 51.5M      0  2487k  0:00:48  0:00:21  0:00:27 2772k
 46  117M    0     0   46 54.2M      0  2499k  0:00:48  0:00:22  0:00:26 2830k
 48  117M    0     0   48 56.5M      0  2489k  0:00:48  0:00:23  0:00:25 2810k
 50  117M    0     0   50 59.2M      0  2504k  0:00:48  0:00:24  0:00:24 2782k
 52  117M    0     0   52 62.1M      0  2526k  0:00:47  0:00:25  0:00:22 2811k
 55  117M    0     0   55 65.1M      0  2547k  0:00:47  0:00:26  0:00:21 2803k
 58  117M    0     0   58 68.8M      0  2589k  0:00:46  0:00:27  0:00:19 2990k
 61  117M    0     0   61 72.1M      0  2617k  0:00:45  0:00:28  0:00:17 3221k
 62  117M    0     0   62 73.8M      0  2556k  0:00:47  0:00:29  0:00:18 2791k
 65  117M    0     0   65 77.3M      0  2620k  0:00:45  0:00:30  0:00:15 3091k
 67  117M    0     0   67 79.5M      0  2600k  0:00:46  0:00:31  0:00:15 2873k
 71  117M    0     0   71 83.3M      0  2651k  0:00:45  0:00:32  0:00:13 2987k
 72  117M    0     0   72 85.5M      0  2638k  0:00:45  0:00:33  0:00:12 2753k
 73  117M    0     0   73 85.7M      0  2557k  0:00:47  0:00:34  0:00:13 2565k
 73  117M    0     0   73 85.8M      0  2494k  0:00:48  0:00:35  0:00:13 1728k
 76  117M    0     0   76 89.3M      0  2506k  0:00:47  0:00:36  0:00:11 1934k
 76  117M    0     0   76 89.3M      0  2439k  0:00:49  0:00:37  0:00:12 1150k
 76  117M    0     0   76 89.3M      0  2375k  0:00:50  0:00:38  0:00:12  726k
 76  117M    0     0   76 89.3M      0  2315k  0:00:51  0:00:39  0:00:12  705k
 76  117M    0     0   76 89.3M      0  2258k  0:00:53  0:00:40  0:00:13  680k
 76  117M    0     0   76 89.3M      0  2203k  0:00:54  0:00:41  0:00:13     0
 76  117M    0     0   76 89.3M      0  2151k  0:00:55  0:00:42  0:00:13     0
 76  117M    0     0   76 89.3M      0  2102k  0:00:57  0:00:43  0:00:14     0
 76  117M    0     0   76 89.3M      0  2054k  0:00:58  0:00:44  0:00:14     0
 76  117M    0     0   76 89.3M      0  2009k  0:00:59  0:00:45  0:00:14     0
 76  117M    0     0   76 89.3M      0  1966k  0:01:01  0:00:46  0:00:15     0
 76  117M    0     0   76 89.3M      0  1924k  0:01:02  0:00:47  0:00:15     0
 76  117M    0     0   76 89.3M      0  1885k  0:01:03  0:00:48  0:00:15     0
 76  117M    0     0   76 89.3M      0  1846k  0:01:05  0:00:49  0:00:16     0
 76  117M    0     0   76 89.3M      0  1810k  0:01:06  0:00:50  0:00:16     0
 76  117M    0     0   76 89.3M      0  1775k  0:01:07  0:00:51  0:00:16     0
 76  117M    0     0   76 89.3M      0  1741k  0:01:09  0:00:52  0:00:17     0
 76  117M    0     0   76 90.3M      0  1739k  0:01:09  0:00:53  0:00:16  231k
 77  117M    0     0   77 91.3M      0  1724k  0:01:09  0:00:54  0:00:15  436k
 78  117M    0     0   78 92.3M      0  1711k  0:01:10  0:00:55  0:00:15  652k
 79  117M    0     0   79 93.3M      0  1700k  0:01:10  0:00:56  0:00:14  886k
 80  117M    0     0   80 94.7M      0  1695k  0:01:10  0:00:57  0:00:13 1186k
 81  117M    0     0   81 96.0M      0  1689k  0:01:11  0:00:58  0:00:13 1166k
 83  117M    0     0   83 97.7M      0  1690k  0:01:11  0:00:59  0:00:12 1319k
 84  117M    0     0   84 99.5M      0  1692k  0:01:11  0:01:00  0:00:11 1478k
 87  117M    0     0   87  102M      0  1714k  0:01:10  0:01:01  0:00:09 1870k
 89  117M    0     0   89  104M      0  1719k  0:01:09  0:01:02  0:00:07 1995k
 90  117M    0     0   90  106M      0  1725k  0:01:09  0:01:03  0:00:06 2137k
 91  117M    0     0   91  107M      0  1718k  0:01:09  0:01:04  0:00:05 2042k
 92  117M    0     0   92  108M      0  1705k  0:01:10  0:01:05  0:00:05 1861k
 93  117M    0     0   93  109M      0  1700k  0:01:10  0:01:06  0:00:04 1521k
 94  117M    0     0   94  110M      0  1683k  0:01:11  0:01:07  0:00:04 1237k
 94  117M    0     0   94  110M      0  1664k  0:01:12  0:01:08  0:00:04  896k
 95  117M    0     0   95  111M      0  1651k  0:01:12  0:01:09  0:00:03  803k
 95  117M    0     0   95  112M      0  1621k  0:01:14  0:01:10  0:00:04  633k
 95  117M    0     0   95  112M      0  1615k  0:01:14  0:01:11  0:00:03  497k
 95  117M    0     0   95  112M      0  1595k  0:01:15  0:01:12  0:00:03  419k
 96  117M    0     0   96  113M      0  1585k  0:01:15  0:01:13  0:00:02  510k
 97  117M    0     0   97  114M      0  1574k  0:01:16  0:01:14  0:00:02  498k
 97  117M    0     0   97  114M      0  1541k  0:01:18  0:01:16  0:00:02  462k
 97  117M    0     0   97  114M      0  1536k  0:01:18  0:01:16  0:00:02  419k
 98  117M    0     0   98  115M      0  1530k  0:01:18  0:01:17  0:00:01  581k
 98  117M    0     0   98  116M      0  1520k  0:01:19  0:01:18  0:00:01  564k
 99  117M    0     0   99  117M      0  1513k  0:01:19  0:01:19 --:--:--  614k
* We are completely uploaded and fine
100  117M    0     0  100  117M      0  1495k  0:01:20  0:01:20 --:--:--  695k
100  117M    0     0  100  117M      0  1476k  0:01:21  0:01:21 --:--:--  589k
100  117M    0     0  100  117M      0  1458k  0:01:22  0:01:22 --:--:--  388k
100  117M    0     0  100  117M      0  1441k  0:01:23  0:01:23 --:--:--  252k
100  117M    0     0  100  117M      0  1424k  0:01:24  0:01:24 --:--:-- 57489
100  117M    0     0  100  117M      0  1407k  0:01:25  0:01:25 --:--:--     0
100  117M    0     0  100  117M      0  1391k  0:01:26  0:01:26 --:--:--     0
100  117M    0     0  100  117M      0  1375k  0:01:27  0:01:27 --:--:--     0
100  117M    0     0  100  117M      0  1359k  0:01:28  0:01:28 --:--:--     0
100  117M    0     0  100  117M      0  1344k  0:01:29  0:01:29 --:--:--     0
100  117M    0     0  100  117M      0  1329k  0:01:30  0:01:30 --:--:--     0
100  117M    0     0  100  117M      0  1314k  0:01:31  0:01:31 --:--:--     0
100  117M    0     0  100  117M      0  1300k  0:01:32  0:01:32 --:--:--     0
100  117M    0     0  100  117M      0  1286k  0:01:33  0:01:33 --:--:--     0
100  117M    0     0  100  117M      0  1273k  0:01:34  0:01:34 --:--:--     0
100  117M    0     0  100  117M      0  1259k  0:01:35  0:01:35 --:--:--     0
100  117M    0     0  100  117M      0  1246k  0:01:36  0:01:36 --:--:--     0
100  117M    0     0  100  117M      0  1233k  0:01:37  0:01:37 --:--:--     0
100  117M    0     0  100  117M      0  1221k  0:01:38  0:01:38 --:--:--     0
100  117M    0     0  100  117M      0  1209k  0:01:39  0:01:39 --:--:--     0
100  117M    0     0  100  117M      0  1196k  0:01:40  0:01:40 --:--:--     0
100  117M    0     0  100  117M      0  1185k  0:01:41  0:01:41 --:--:--     0
100  117M    0     0  100  117M      0  1173k  0:01:42  0:01:42 --:--:--     0
100  117M    0     0  100  117M      0  1162k  0:01:43  0:01:43 --:--:--     0
100  117M    0     0  100  117M      0  1151k  0:01:44  0:01:44 --:--:--     0
100  117M    0     0  100  117M      0  1140k  0:01:45  0:01:45 --:--:--     0
100  117M    0     0  100  117M      0  1129k  0:01:46  0:01:46 --:--:--     0
100  117M    0     0  100  117M      0  1118k  0:01:47  0:01:47 --:--:--     0
100  117M    0     0  100  117M      0  1108k  0:01:48  0:01:48 --:--:--     0
100  117M    0     0  100  117M      0  1098k  0:01:49  0:01:49 --:--:--     0
100  117M    0     0  100  117M      0  1088k  0:01:50  0:01:50 --:--:--     0
100  117M    0     0  100  117M      0  1078k  0:01:51  0:01:51 --:--:--     0
100  117M    0     0  100  117M      0  1069k  0:01:52  0:01:52 --:--:--     0
100  117M    0     0  100  117M      0  1059k  0:01:53  0:01:53 --:--:--     0
100  117M    0     0  100  117M      0  1050k  0:01:54  0:01:54 --:--:--     0
100  117M    0     0  100  117M      0  1041k  0:01:55  0:01:55 --:--:--     0
100  117M    0     0  100  117M      0  1032k  0:01:56  0:01:56 --:--:--     0
100  117M    0     0  100  117M      0  1023k  0:01:57  0:01:57 --:--:--     0
100  117M    0     0  100  117M      0  1014k  0:01:58  0:01:58 --:--:--     0
100  117M    0     0  100  117M      0  1006k  0:01:59  0:01:59 --:--:--     0
100  117M    0     0  100  117M      0   997k  0:02:00  0:02:00 --:--:--     0
100  117M    0     0  100  117M      0   989k  0:02:01  0:02:01 --:--:--     0
100  117M    0     0  100  117M      0   981k  0:02:02  0:02:02 --:--:--     0
100  117M    0     0  100  117M      0   973k  0:02:03  0:02:03 --:--:--     0
100  117M    0     0  100  117M      0   965k  0:02:04  0:02:04 --:--:--     0
100  117M    0     0  100  117M      0   958k  0:02:05  0:02:05 --:--:--     0
100  117M    0     0  100  117M      0   950k  0:02:06  0:02:06 --:--:--     0
100  117M    0     0  100  117M      0   943k  0:02:07  0:02:07 --:--:--     0
100  117M    0     0  100  117M      0   935k  0:02:08  0:02:08 --:--:--     0
100  117M    0     0  100  117M      0   928k  0:02:09  0:02:09 --:--:--     0
100  117M    0     0  100  117M      0   921k  0:02:10  0:02:10 --:--:--     0
100  117M    0     0  100  117M      0   914k  0:02:11  0:02:11 --:--:--     0
100  117M    0     0  100  117M      0   907k  0:02:12  0:02:12 --:--:--     0
100  117M    0     0  100  117M      0   900k  0:02:13  0:02:13 --:--:--     0
100  117M    0     0  100  117M      0   893k  0:02:14  0:02:14 --:--:--     0
100  117M    0     0  100  117M      0   887k  0:02:15  0:02:15 --:--:--     0
100  117M    0     0  100  117M      0   880k  0:02:16  0:02:16 --:--:--     0
100  117M    0     0  100  117M      0   874k  0:02:17  0:02:17 --:--:--     0
100  117M    0     0  100  117M      0   868k  0:02:18  0:02:18 --:--:--     0
100  117M    0     0  100  117M      0   861k  0:02:19  0:02:19 --:--:--     0
100  117M    0     0  100  117M      0   855k  0:02:20  0:02:20 --:--:--     0
< HTTP/1.1 504 Gateway Time-out
< Content-Type: text/html
< Content-Length: 1033
< Connection: keep-alive
< Server: CloudFront
< Date: Thu, 18 Jul 2024 01:42:43 GMT
< X-Cache: Error from cloudfront
< Via: 1.1 CENSORED.cloudfront.net (CloudFront)
< X-Amz-Cf-Pop: CENSORED
< X-Amz-Cf-Id: CENSORED
< 
{ [1033 bytes data]

100  117M  100  1033  100  117M      7   855k  0:02:27  0:02:20  0:00:07   253
* Connection #0 to host up.flickr.com left intact

And the response was:

What the fuck happened? You were doing so well, Flickr. Here I am going out of my way to diagnose and report – with all pertinent details available to me – a bug in your API servers, and I get a robotic, bullshit response that’s purely about refusing to accept responsibility. This is the kind of response which makes me think your support staff are paid based solely on how fast they close tickets.

And what the hell is Marc ‘appreciating’ here? Amanda’s “Warmly” valediction actually feels genuine, in context (and is human even out of context). “Appreciatively” feels like corporate innuendo.

We humans are weird

Admittedly I started writing this out of catharsis, but it really got me thinking.

Why should I care that Flickr delivered miserable customer support in this second case? Shouldn’t I just quietly move on, like I would if it were from most other companies? Shouldn’t I be thrilled it’s at least not a bug in Lightroom itself, since Adobe’s customer support is a hundred times worse in every case; among the most aggressively evasive and pre-emptively hostile I’ve ever encountered?

I think we’re all pretty conditioned to expect terrible experiences with so-called customer support from large software companies, like Adobe, or Google. We expect it and tolerate it, against all justice and our own interests.

Instead, it’s often inconsistency in a single company’s behaviour that’s the most infuriating and raises our ire. This makes no logical nor rational sense – and is very bad from a game theory perspective, as it encourages companies to be consistently dumb and evil.

It seems akin to how most folks pay little attention to e.g. Facebook doing yet another horrible thing – the headlines might as well all be “Facebook acts like Facebook yet again” – but if e.g. Apple does something a bit clueless, the world gets up in arms.

I guess it boils down to hope, ironically. If we see a company – or a person – demonstrate that they can do better, then we raise the bar for them. And are then ripe to be disappointed if they merely behave like most of their peers, subsequently.

Which is stupid, really.

I’ve attempted the multi-display lifestyle a few times over the last thirty years. At first as a bourgeois indulgence, back when even having a single 17″ display was considered luxurious¹, let-alone two displays. Just because I could. Then sporadically over the years in more practical attempts to expand my screen real estate.

The debut of 5k displays in 2014 relieved that pressure for a long while. It’s hard to overstate how profound that first 5k iMac was.

But in the last year or so I’ve felt a bit cramped – particularly as I’ve spent increasing amounts of time in Xcode, which is okay on a 5k display but not great. Especially if you try to do GUI design, whether AppKit or SwiftUI. Let-alone if you start working with iPad Pro simulators and the like.

So I’ve been considering, yet again, my options.

Sadly:

1. I have an iMac Pro. Even if I were willing to waste the built-in display, I cannot – Apple does not allow iMacs to disable their internal displays.
   
   I wouldn’t necessarily mind replacing the iMac Pro (it’s been a pretty shitty computer since the day it was made) but Apple keep conspiring to discourage me from doing so².
2. There aren’t any great options for larger displays.
   
   Available 6k displays aren’t great³, although I’m glad there’s at least a couple of half-decent options. I never thought I’d be thankful that Dell, of all hardware companies, is still around and making better products than Apple.
   
   8k displays practically don’t exist – I don’t have the physical space to use an 8k TV⁴, which is otherwise actually the best way to go, and the one 8k computer display I know of is far too physically small for its resolution⁵.

As it happened, I had an extra 5k display left over from working at LinkedIn (they wouldn’t provide decent displays to employees while I was there, so I had to buy my own out of my own pocket⁶). So far as multi-display setups go it’s arguably ideal, since it’s basically the exact same panel as the iMac Pro’s display, so it should match quite well⁷. So a few months ago I went to some effort to rearrange my working space to accomodate it, and set to work with it.

And not a lot has changed in thirty years.

The Mac operating system still Just. Fucking. Sucks. at remembering where windows were, when relaunching apps.

For the first month or so it was mostly fine – a couple of apps would be occasionally problematic, but it was tolerable. Mostly. Xcode in particularly pisses me off in that it always forgets where the documentation window was, every time you open it⁸.

But then, something invisible to me apparently happened, and since then every single boot every single window resets its position onto the built-in display. Every single boot I have to spend a bunch of time digging buried windows out from the over-crowded main display and shifting them back to their correct positions on the second display.

I might dismiss this as particular bad luck, except I know from decades of experience that this is in fact normal. It always happens, sooner or later. And at some point it will unfuck itself and start [mostly] remembering where windows go. Only to inevitably repeat the vicious cycle.

Full-screen apps (mostly games) don’t handle the additional display(s) correctly.

This cuts both ways – some apps erroneously ignore the extra displays, while some erroneously don’t and do stupid things like put windows on the wrong display:

• Games are the most common offenders in the former category, as they usually fail to dim or turn off unused displays. It’s very distracting having a bright display right next to one on which you’re trying to play a game. Every time I launch or exit such an app I have to manually open System Settings, go to Display settings, click the secondary display, and manually changes its brightness.
  
  My secondary display (LG 5k) doesn’t have a power button, and I found out real quick that unplugging the Thunderbolt cable confuses the hell out of macOS, resulting in windows going wild and some apps outright crashing.
• In the latter, “WTF are you doing?!” subcategory are applications like Adobe Lightroom Classic, which will spontaneously move the entire window to a different display sometimes, and the only way to get it back is to quit & relaunch Lightroom⁹.

Side Docks aren’t practical.

When the Dock is on the side, it only appears on the display that is farthest to that side. Having to move the mouse across multiple displays just to get to the Dock is untenably slow and awkward.

I have the Dock set to auto-hide, because I’m not a monster. But that does mean I have to be careful about mouse movements near its edge of the screen.

I prefer having the Dock on the left-hand side because it’s the least intrusive place – there’s relatively little you have to mouse to or interact with on the left edge (the right side is second-best, but notably is the home of scrollbars). It’s also slightly easier to mouse left (or right) than down, because moving the mouse ‘down’ (towards me) has extra tension from the USB cable.

With multiple monitors (in a typical horizontal arrangement), I’m forced to use a bottom Dock, which is the worst place to have the Dock because so much stuff is placed right at the bottom of the screen – window resize widgets, toolbars & their buttons, horizontal scrollbars, etc. It’s immensely irritating to have to finesse minute mouse movements in order to avoid triggering the Dock while somehow still getting the cursor within a few pixels of the edge of the screen. Sometimes it’s necessary to move the entire window just to interact with its bottom parts – which can in turn require first resizing it from the top, in order to allow it to be moved upwards.

The menu bar is particularly petulant when multiple displays are in use.

I have the menu bar set to auto-hide, partly for reasons of visual cleanliness and to save a little screen space, but also because the menubar ruined my previous iMac’s display by burning in, hard.

I also use the menubar a lot more than for just activating menus with the mouse – I have iStat Menus and a host of other monitoring utilities in there as well. So I’m frequently mousing up to the top of the screen to reveal the menubar. Which is not a burden in itself – flicking the mouse to the top of the screen is a completely trivial operation.

But it doesn’t work half the time, when you have multiple displays, because the menubar will only deign to reveal itself on the “active” display, which is often at a glance visually indistinguishable from the “inactive” display (thanks to Apple’s long-running war against clarity).

That defeats the instinct and muscle-memory to just flick the cursor to the top of the display in order to see the menubar. Worse, since macOS has some design flaws regarding menu bar reveal (e.g. it refuses to reveal if the frontmost application’s main thread runloop isn’t idle, and whenever an application is launching), you can’t immediately tell if it’s refusing to reveal itself because the cursor’s on the “wrong” display. So inevitably there’s wasted, distracting moments of pause and investigation, and possibly movement to a whole different display. For an operation that happens hundreds of times a day, wasted seconds add up.

Windows appear on the wrong display sometimes.

Enough said, really. I have no idea why they appear in the wrong places, but then so much about macOS’s window management is inexplicably buggy like this, such as why some modal dialogs appear behind all existing windows.

Multi-display ergonomics tend to suck.

If – in a typical horizontal arrangement – you give the two displays equal priority in placement, you end up with one to your left and one to your right, with a very irritating gap between them in the one place it’s natural to look – straight ahead of you. Neither display is aligned with the keyboard or mouse, which I find impairs typing and mousing accuracy (especially with a treadmill desk where I’m constantly moving). And I find I get a sore neck no matter what, because I’m spending all my time looking anywhere but centre.

If you keep a “primary” display front and centre, and put a second one off to a side, the second display becomes much less useful because it’s even more uncomfortable to actually look at. Which couples with the fact that…

Splitting work across multiple displays is hard.

More so than it seems it should be. In practice there’s a big difference between two windows side-by-side on one display, and two windows on separate displays (especially if the displays aren’t identical, or aren’t aligned on horizontal or vertical axes). Maybe it’s as a simple as the difference between moving your eyes and moving your head.

Whatever it is, I just cannot get comfortable moving between displays frequently. That then forces me to not put anything important on the second display (especially if it has an ergonomically inferior position compared to the primary display), so it ends up being kind of a dumping ground for stuff I don’t actually use that often – e.g. calendar, clipboard viewers, calculators, etc.

And since I don’t actually use them that often, there’s really not much advantage to having them on a second display, versus just bringing them to the front on the main display and then hiding them again when I’m done. In fact, sometimes it’s counter-productive as irrelevant animations on the second display distract me, whereas they’d be safely out of sight behind other windows on the main display.

The only meaningful use-case I have so far for genuine simultaneous use is moving information references – e.g. Xcode’s documentation window – to the second display. In fact that was the specific use-case that made me go to the trouble of hooking up the second display, since it’s practically impossible to fit a non-trivial Xcode project into just half a 5k display, and having to constantly toggle between code & documentation is irritating and a little disruptive – there’s often a lot of rapid back-and-forth between code & docs, which is of course much easier if they’re simultaneously visible.

Follow-up

There’s a surprisingly long thread on HackerNews about this post. And I was pleasantly surprised to find mostly commiseration and honest attempts to help, as opposed to vitriol. It’s worth looking over if you want to dive even deeper into this topic.

Addressing a select few points raised in the HackerNews comments:

• I’ve been a Mac user for about thirty-five years, and always focused on the Mac. So assertions that I’m “doing it wrong” because I’m secretly a Wintel or Linux stooge are misguided. 🙂
• A lot of folks also brought up problems with even getting external displays to work reliably, which I can definitely relate to – that LG 5k I’m using was a real bitch, frankly, before I replaced what turned out to be a faulty Thunderbolt cable that it came with. Even then, it never worked reliably with my work MacBook Pro, for some reason. Seems to work just fine with my iMac Pro. 🤷‍♂️
  
  I chose not to go there in this post because somehow I’m not having those sorts of issues right now (🤞) and because I feel like that’s a whole other can of worms. But yes, it’s definitely a big problem for the Mac, too.
• One or two folks suggested that my problem is that I should just get a bigger single display. To which I agree, that is my problem. That was covered in the opening of the post. 🙂
  
  Quite a few people suggested or at least evangelised using ultra-wide displays. I’m open to it, if one is ever released with decent pixel density. e.g. 5k is a decent number of pixels in 23.5″ (the horizontal width of a conventional 27″ display) but is way too few on a display that’s like 50″ wide. Abysmally few. I touched on this topic tangentially in the footnotes, but to reiterate: a decent display has a pixel density of at least 100 PPD (pixels per degree) at a reasonable working distance. Apple’s “Retina” displays all have this. To my knowledge no ultra-wide display ever made has this – not even close.
• The specific setup I’ve been using recently is with my iMac Pro in its traditional place front and centre, and the LG 5k off to the left side by about 45°.
  
  In prior attempts at the multi-display lifestyle I’ve used many other setups, including symmetrical horizontal arrangements, stacked vertical arrangements, and even ad hoc “diagonal” arrangements with things like my iPad Pro or laptops. I don’t think there’s a reasonable layout that I have not tried.
• I don’t use Spaces. I’ve used it in the past (mostly way back when it was new to Mac OS X, and earlier on Linux) but it just never really felt nor worked right. It also has its own family of window management problems & bugs, with things like new windows (particularly modal dialogs) silently opening in the wrong space. 😔
• I don’t full-screen apps (except in very limited circumstances, such as some games which only run full-screen).
  
  For years I did, on my work MacBook Pros, but only because I felt forced to by the small laptop screen – there are so many bugs around full-screen mode in macOS. And it makes very little sense for most apps when you have a decent-sized screen, as you end up with massive areas of wasted space in margins etc. In fact I got so sick of the bugs that even on my little 12″ MacBook Air I don’t use full-screen mode anymore, and just suffer through the loss of screen real estate. And, more importantly, I don’t do any real software development or photo/video work on laptops anymore – mostly just writing and web browsing.
• A few folks seemed surprised that I use the Dock at all, which I found surprising in turn. I’d never really considered not using the Dock.
  
  Reflecting on it, I mostly use the Dock for:
  • Switching applications.
    
    I use ⌘⇥ too, of course, but with multiple displays it’s unpredictable as to which screen the app palette appears on, making it hard to efficiently use the mouse with it (and merely hitting tab over and over again is far too slow when you have tens of apps open).
    
    I do use Spotlight to launch & switch apps, but I try to avoid it for commonly-used apps as Spotlight is very unreliable:
    • Sometimes it takes a non-trivial amount of time to show the floating text input window, and keystokes in the interim may be silently ignored or go to the wrong place.
    • Sometimes it just doesn’t work. I type some letters and nothing happens, or it just shows me that infuriating “indexing” indeterminate progress bar. Try again later, apparently.
    • It randomly changes what frequently-used character sequences mean. e.g. I might have been using “te” for TextEdit for months, and then suddenly it decides that “te” means some random fifteen year old PDF file. It’s absolutely infuriating and inexplicable, and there’s no apparent way to fix it when this happens.
  • Accessing minimised windows. I don’t know of any other way to get at these.
  • Accessing the contextual menu for an app’s Dock icon – most often to force quit it, but also some apps put important functionality there which isn’t readily accessible through any other means. And I know that I can ⌥⌘⎋, which I use sometime too, but for whatever reason I prefer going via the Dock.
  • Monitoring (e.g. unread message counts).
• A lot of people tangentially critiqued the Mac’s lack of window tiling and gesture functionality, particularly those coming from Windows it seems. These are fair and broadly accurate critiques, but they did seem to overlook that while macOS doesn’t have a lot of such gestures & tiling features, it does have some (e.g. what Apple calls “Split View“).
  
  But, “Split View” is essentially a specialisation of full screen mode, with many of the same design problems and bugs.
• There were a lot more display and window management utilities than I thought – many people come forth with suggestions and recommendations I’d not yet encountered (e.g. Stay, Display Maid, Phoenix, Rectangle Pro). I might look into some of those, and will perhaps do a follow-up post if they substantially improve things (but, keep in mind that problems like Dock and menubar accessibility, let-alone fundamentals like the ergonomics, aren’t addressed by these utilities).
• A few people responded with essentially “huh, none of this happens to / bothers me”. Which is fair to say, but not surprising – different strokes for different folks. I was pleased to see that most weren’t trying to dismiss my opinion out of hand, merely noting that there are other opinions. Which is totally fine and important to remember.

1. Apple’s first so-called 17″ display (it was actually only 16.5″) was the Apple Multiple Scan 17, released in 1994 for USD$1,100 (not counting taxes). That’s equivalent to about $2,500 today. For a display that could do just 1024×768 – but, in a time when 640×480 was the vastly dominant standard, so that was a huge leap up in display resolution – 156%! It’s the same as going from a 4k to 6k display. ↩︎
2. I say that with some sense of humility and humour, of course. But the effect is real, even if arguably irrational. Nobody likes buying something that they know is about to be replaced, nor something that’s a bit outdated even at its start, irrespective of whether it’s functionally sufficient for their needs. ↩︎
3. Apple’s 6k display is stupidly over-priced, even by Apple standards, and uses a severely compromised design with its coarse backlight zoning system. That results in significant blooming, which is annoying even for casual use but particularly problematic for photography and videography. I’d rather have a uniform display with lower contrast – like existing 5k displays – as the lesser of the two evils, since that more faithfully reproduces images.
   
   Dell’s 6k display is a better option, having made better decisions on key design compromises, but it’s still a bit pricey and it’s a bit dim at just 450 cd/m². Good enough for coding in a not too brightly lit room, but it’s pretty useless for working with HDR imagery. ↩︎
4. The smallest I can find is 65″, which would mean sitting at least a metre away from it to avoid seeing its individual pixels. Not impossible, but deeper than most people’s desks. And a corner desk (like mine) has a lot less usable depth due to the angle of the walls, making it completely impossible to fit a 65″ display at all, let-alone while maintaining a practical viewing distance. ↩︎
5. Apple pretty much nailed it with their 5k displays, right from the start; 27″ is a good size for them. At a healthy viewing distance of about 70cm, it’s about 111 PPD (pixels per degree) which is close enough to the limit of even excellent human vision (~130 PPD) to make it genuinely a ‘retina’ display (where you can’t typically distinguish individual pixels) for most people without wasting a lot of pixels. And if you have exceptionally good vision, situating yourself a little farther away is plausible in most setups.
   
   Dell’s 8k display, on the other hand, is a mere 32″, yielding a PPD of 143 – way higher than necessary and meaning you can’t actually utilise its full resolution at typical viewing distances – it’d require being within 50cm, which might not sound all that bad but try it; 50cm is really close to any computer display, let-alone a relatively large one at 32″.
   
   8k displays should be about 44″, to be optimal. ↩︎
6. And lest you think I was just a fussy princess, know that quite a lot of my colleagues did the same. You could look around the office and get a pretty good idea about who really cared about their work (or at least their eyes), by looking at whether they had the company-issued displays or not. ↩︎
7. In practice it doesn’t – they differ substantially in temperature, with the LG display being too warm, but I could probably fix that by properly calibrating the LG display… I just haven’t bothered. I used to obsess over monitor calibration, years ago, until I became more aware of just how incredibly situational and subjective human colour perception is anyway (and how well-calibrated Apple’s displays usually are already, out of the box).
   
   It’s largely pointless stressing over small errors in colour when you have no control over the viewing context anyway – let-alone the fact that most people view content on tiny little phone displays that are comically not colour-accurate, and can’t see anything anyway. ↩︎
8. But then, it essentially forgets the windows contents themselves, anyway. It preserves only the first open tab, not any of the others. When I’m in the middle of development and I accidentally close the documentation window – losing a dozen open tabs that were very precisely tied to what I was working at that exact moment – I want to throw my whole god damn computer through a wall.
   
   So, Xcode is an infuriating piece of shit even on one display, in that respect. Somehow that’s little consolation. ↩︎
9. Yes, Lightroom’s Window menu has a “Move to <other display>…” item, but it doesn’t do anything. ↩︎

If you’re too young to have personally lived that time, just know this key point: back then, buying or streaming music online was basically not a thing¹. You bought CDs in physical stores, or maybe by mail order from an online retailer if you were on the bleeding edge. Or you listened to the radio. Or, you traded pirated copies of CD rips via a plethora of file sharing apps, most famously Napster.

It was the opinion of quite a few people – my younger, more ignorant self among them – that if the music industry wasn’t willing to provide us the songs we wanted by the means we wanted and at the price we wanted, then it was their fault that we pirated their product.

Talk about entitlement.

Computer game pirates had – and seemingly still have – a similar attitude. “$70 for a game? Outrageous! I have no choice but to pirate it!”

Bullshit. You have plenty of choice: you can pay for it, or you can not have it. You’re not entitled to the work of others. That used to be believed, but not anymore.

A similar sense of entitlement seems to underpin many critiques of Apple’s business practices. While I’m no fan of some of Apple’s business practices – because it’s always a shame to see flaws in an otherwise good or promising product – I’m under no delusion that I’m somehow entitled to have Apple’s products at all, let-alone on purely my own terms.

1. I think it’s now largely forgotten just how big a deal it was when the iTunes Music Store was announced in April 2003, with five of the biggest western music labels onboard. ↩︎

Even so, there are things which just scream “I’m actually an iOS app”, like when you right-click a message or attachment while the Messages window is on a secondary display, and the contextual menu appears in the top-left corner of the main display. And disappears when you click on any item in it, without activating the item. Because it’s reacting to the mouse location as if it were in the correct place on the secondary display. But if you click on the secondary display, it just dismisses the pop-up menu without activating the selected item – I don’t know how they managed to get that discontinuity between mouse move events and mouse down events, but there you go, the miracle that is Catalyst.

Sigh.

Also, keyboard navigation doesn’t work correctly in the contextual menu – it’s fine for the plain text items, but it’s impossible to select any of the reactions.

Fortunately, for reactions specifically, there’s a workaround! ⌘T brings up a special contextual menu just for reactions, and you can then just hit a number key, from 1 through 6, to apply the corresponding reaction. It only works on the most recently received message in the selected conversation, but then that’s usually the one you want.

Of course, the GUI for that special contextual menu is ripped directly out of iOS, further emphasising Message’s unnatural nature.

…after I’d filled out a form PDF that was emailed to me. It had exactly the option I wanted first and foremost, to send the completed PDF back to the sender.

Sure, manually digging up the completed PDF from disk and dragging it into a Mail Compose [Reply] window isn’t hard, but it just feels so thoughtful when the system saves me the effort. Knowing that someone, somewhere, actually thought through how Mail & Preview might be used, and thought enough of their users to go to the trouble of implementing this.

Kudos to whomever was behind this sweet little feature.

A veiled proxy for the speaker’s laziness. Used as in “this isn’t actionable” to mean one or more of:

• I don’t understand and I don’t want to.
• This looks like it takes effort and I don’t want to.
• I don’t like this and I don’t want to.

Cannot Reproduce, n.

A versatile term, often found as a formal state in bug / ticket tracking systems, which can mean various things:

• I tried vaguely, once, to reproduce this, carefully ignoring any detailed instructions you provided, and I didn’t immediately see a problem.
• I successfully ignored this for long enough that the reporter gave up (or died of old age) so now I can pretend it magically went away by itself.
• I couldn’t be bothered even trying to reproduce this, which is a form of “cannot”, right?

Crowdsourcing, n.

The practice of making other people do your work for you without compensation or credit.

See also the observation that none of us are as dumb as all of us are.

Free to play / Freemium, a.

A sales method used when presenting the true cost up front would result in no sales.

Commonly used in some industries outside of tech as well, such as drug dealing.

Full Stack Developer, n.

Someone who programs exclusively in JavaScript and once wrote a query for MySQL.

Gamify, v.

To make a product less enjoyable by inserting unnecessary casino mechanics and condescending interstitial animations.

GPL, n.

A passive-aggressive way of open-sourcing your software without actually making it free (as in freedom).

(also, AGPL, even more so)

Levandowski, n.

A unit of money equal to approximately $80M USD. Used especially in contexts where a purchase has deeply negative actual value. Derives from the hiring bonuses and other promised compensation required to recruit a single Anthony Levandowski.

• Often used in hiring negotiations to establish compensation expectations, e.g. “I don’t switch companies for less than three Levandowskis.”
• Can be used to refer cryptically to a huge future cost, e.g. “That’s only one Levandowski now, but also in future.”

MVP (Minimum Viable Product), n.

A trump card in design discussions to unilaterally reject anything that is inconvenient, hard, or from someone other than one’s self. e.g.:

Mohammed: For our banking app, let’s add the ability to generate photos of people with AI.

Mary: We should put some safeguards in place to prevent deepfakes.

Mohammed: Mary, we’re working on the MVP right now, we need to stay focused.

Offline, a.

As in, “let’s talk about that offline”.

May be used with any of several meanings, the ambiguity of which is usually intentional:

• To shift communication from the real world to the online world, without any apparent sense of irony.
• To dismiss something with prejudice, under the curious social contract that no one is allowed to call you out on it.
• To relatively tacitly state that you do not like someone’s input and want them to shut up.

Prototype, n.

Software created by ignoring all the development best practices that are inconvenient, such as unit testing and accepting input from others.

Technical Debt, n.

Liabilities in software design and implementation due to poor choices and unrealistic deadlines, that managers inexplicably believe – in the face of all evidence from every software project ever – is a limitless line of free credit.

UX Designer, n.

Self-selected job title for pretentious UI designers.

Beware of using the term “UI” in front of an afflicted person, as it triggers a compulsory thirty minute monologue on the difference between “UI” and “UX” design, made particularly excruciating by the fact that there is no difference.

Further reading

How to speak Silicon Valley: 53 essential tech-bro terms explained.

The Devil’s Dictionary by Ambrose Bierce (context).

Universal Clipboard basically just means you can copy something on one device and paste it on another, as long as they’re physically near each other and signed into the same Apple ID. There is no step 3.

It’s such a beautiful feature, in concept and execution. Admittedly most of the time I use it merely to shuffle authentication codes from my iPhone to my Mac, but still – I love not having to temporarily memorise arbitrary numbers and then type them in manually.

I think what really elevates it to perfection is that it works. It sounds like such a vapid point, but it really makes all the difference. Nominally Universal Clipboard is just one of a family of similar features, under the Continuity moniker, but it’s the only one that actually works reliably and predictably. The only one!

Many of the other Continuity features are similarly great in concept, but just don’t work reliably. SMSes sometimes don’t sync to my Macs. AirPlay sometimes just won’t connect, or the video quality is comically bad. Unlocking my Macs with my Watch is a guessing game as to whether it feels like working today. AirDrop is so fussy and unreliable that it actually gives me anxiety if I even think of using it.

I wish Apple’s products had more features like Universal Clipboard.

The question was raised whether an official Swift backtracer should try to support code that doesn’t use frame pointers. Which immediately raised the question – in my mind – of if anyone is still using the “optimisation” of omitting frame pointers, anyway. And perhaps more importantly, whether they should still be omitting frame pointers.

What is a frame pointer?

A pointer to a stack frame, held in a well-known location. That location can be in the stack itself (forming a linked-list of the stack frames) or in registers (e.g. the x29 register on AArch64, or RBP register on x86-64).

The controversial part – insofar as there is any controversy – is in dedicating a CPU register to hold a frame pointer (to point to the start of the current stack frame). It’s super convenient for a lot of things, but particularly for debuggers and profilers as it gives them a reliable and very fast way to find the top of the current callstack. But it’s not technically required for the program to function.

No live CPU architectures, that I’m aware of, have a dedicated hardware register for frame pointers. So you nominally have to “give up” a GPR (general-purpose register) in order to have a frame pointer.

Franz Busch pointed out that some notable software still ships with frame pointers omitted, e.g. apparently some major Linux distros. I suspect it’s merely some inertia (or simply oversight) that’s delaying getting people off of that old crutch. I’m not remotely surprised that some big Linux distros are in this bucket – they tend to be absurdly conservative and slow to change¹. And it’s mind-boggling how much vitriol restoring frame pointers generates from the peanut gallery.

From watches to servers these days – and frankly most of the embedded space, since it’s mostly ARM – everything generally has an ISA with sufficiently many GPRs to negate any big benefit from omitting frame pointers. Giving up one of 31 GPRs (for e.g. AArch64, the dominant CPU architecture family today) is pretty insignificant for the vast majority of code, because almost nothing actually uses all 31 GPRs anyway. It only makes a significant difference² when the CPU design is register-starved to begin with, like i386. And those architectures are largely dead, in museums, or restricted to very tiny CPUs as used in some microcontrollers (“embedded” systems).

Even back when i386 et al were still a concern, the proponents of -fomit-frame-pointer often argued not on the potential merits of the trade-off, but rather that it was a “free” performance boost, so even if it was only by a percentage point or two, why not? They of course were either naively or deliberately overlooking the detrimental effects.

There may still be software for which omitting frame pointers is the right trade-off, even on modern CPUs. But I find it hard to believe there’s enough cases like that to warrant accomodation in standard tools.

A brief trip back to Apple circa 2007

Back in the brief window of time when i386 was a thing for the Mac (32-bit Intel, e.g. Core Duos³ as used in the first MacBooks), I was at Apple in the Performance Tools teams (Shark & Instruments), and it was a frustration of ours that -fomit-frame-pointer was a noticeable performance-booster on the register-starved i386⁴ architecture⁵, so it was hard to just bluntly tell people not to use it… yet, by breaking the ability to profile their code, people who used it often left even bigger performance gains on the table (or otherwise had to invest much more labour into identifying & resolving performance problems).

At one point there was even an Apple-internal debate about whether to abandon kernel-based profiling in favour of user-space profiling⁶ because implementing backtracing without frame pointers is possible but very expensive and requires masses of debug metadata (e.g. DWARF), making it highly unpalatable to put in the kernel. Thankfully there were too many obvious problems with user-space profiling, so that notion never really got its legs, and then x86-64 finally arrived⁷ and it was mooted.

1. e.g. Ubuntu still not officially supporting ImageMagick 7 even though it’s been out for nearly a decade. ↩︎
2. Aside from the question of register space, there is additional cost to implementing frame pointers, as additional instructions are required around function entry & exit in order to maintain the frame pointers – to push & pop them off the stack, etc. The cost of those is usually insignificant – especially in superscalar microarchitectures, as is the norm – so that aspect is not typically the focus of the controversy. ↩︎
3. Tangentially, I vaguely recall us Apple engineers kinda hating the Core Duo (Yonah), or more specifically Apple’s choice to use it. Apple used them only for a tiny window of time, from May 2006 to about November 2006 when the Core 2 Duo (Merom) finally replaced them across the line. I don’t recall all the reasons that the Core 2 Duo was superior, but they included that Core 2 Duo corrected the 32-bit regression (for Macs) and performed much better. Anytime Apple releases a Mac with a dud processor in it, like those Core Duos, a lot of Apple engineers die a little inside because they know they’re going to be stuck supporting the damn things for many years even after the last cursed one rolls off the assembly line.
   
   It’s still a mystery to me why Apple rushed the Intel transition in this regard. They only had to wait six more months and they could have had a clean start on Intel, with no 32-bit to burden on them for the next seven years. ↩︎
4. Why do I keep calling it “i386”? Isn’t it officially “IA-32”? Well, yes, but that’s (a) only retroactively and (b) only ever used by Intel. Though I guess “x86” is probably the more common name? Yet “i386” is in my mental muscle memory. Maybe that’s just how we used to refer to it, at Apple? Maybe just because that’s the name used in gcc / clang arch & target flags?
   
   Incidentally, clang -arch i386 -print-supported-cpus on my M2 MacBook Air still lists Yonah (those damn Core Duos) as supported. Gah! They won’t die! 😆 ↩︎
5. It’s funny how the Intel transition is now heralded as being amazing and how much better Intel Macs were than PPC Macs, but for a while there we lost a lot of things, like a 64-bit architecture, an excellent SIMD implementation, and the notion of more than [effectively] six GPRs.  ↩︎
6. There were at the time already some Apple developer tools that did user-space profiling, most notably Sampler (now a niche feature in Activity Monitor) and early versions of Instruments (in fact Instruments still has the Sampler plug-in which does this, although I can’t really fathom why anyone would ever intentionally use it over the Time Profiler plug-in). ↩︎
7. In the sense of all Macs adopting it, not just the Mac Pro. It was easy to ignore i386 at that point because it was then all but officially a dead architecture as far as Apple were concerned. ↩︎

Sometime during each day I get a follow-up notification telling me that Private Relay is now active. Even though by all appearances it already was.

This is frustrating on multiple levels – obviously there’s the question of why this notification is appearing to begin with, and lying¹ about the support status of Private Relay, but it’s also concerning that it’s sending mixed messages about whether Private Relay is working or not.

1. I mean, I assume it’s lying about the status – I suppose it’s possible that Apple, headquartered in the San Francisco bay area, might have revoked support for Private Relay in their own region. Seems like an odd decision, though. ↩︎

At first it was just Google that went to shit, but now even Bing & DuckDuckGo are increasingly filled with SEO spam, garbage results, and ads. Worst of all, they’re increasingly unwilling to address my actual queries, preferring instead to serve up uselessly generic results and insulting clickbait.

Is Kagi any better? Hard to say; I only just started playing with it. It looks promising at the outset, but only time will tell.

Kagi does have at least one incredibly non-novel yet unique feature: the ability to permanently customise the ranking of each website in your results. Personalised search done the right way.

You can give a website a nudge up or down, or take it to the extreme by pinning or blocking the website entirely. This is what actually got me to sign up for Kagi. I can’t tell you how many times I’ve looked at Bing, DuckDuckGo, or Google search results and said “for fuck’s sake, why do you even allow this website in your index?”. Over and over again, the top page of results is clogged with those rip-off merchants that clone e.g. Wikipedia or StackOverflow, and just stuff them full of ads and trackers. I genuinely can’t fathom how these websites, clearly commercialising industrial-scale copyright infringement, can persist.

We’ll come back to this feature in a minute.

Kagi Search Stats

While playing with Kagi, I incidentally found their Search Stats. Which are interesting for several reasons – for a start, as I write they have just 19,468 “members” – which I take to mean paying customers – which seems crazy tiny even for a search engine which has essentially only been around for seven months, given they’ve received some high-profile evangelism (such as the aforementioned by John Gruber). At least they’re growing.

But perhaps the most interesting is what they call “Domain insights”, which is a polite way of saying “best & worst big websites” – at least, as expressed by Kagi’s 19,468 members. This is the summary of all those ranking customisations Kagi’s users have made. As such, I think it’s an interesting insight into what people consider good websites as opposed to popular.

On the positive (most-favoured) side, there’s not a lot of surprises. Wikipedia tops the list by a mile – specifically the English version of Wikipedia, suggesting Kagi might have a very U.S.-centric user base – followed by a variety developer-centric websites, with Reddit being really the only other exception. As one might expect, the early adopters of Kagi are computer geeks.

It’s the ‘negative’ side that’s most interesting. Remember that in order to get on these lists you have to actually be a major website, well known and a frequent-enough occurrence in search results that a large number of people have told you to bugger off. So it’s essentially revealing the sentiment towards the ‘popular’ websites.

It surprised me to see Pinterest at the top. Not just the top position, but the top seven positions, accounting for their various geo-specific site variants.

I myself resent Pinterest’s existence because I find that the dominant search engines prioritise Pinterest over far superior matches, to an absurd degree. I can recall only one time, in my life, where I visited Pinterest and actually got any benefit out of it. One.

But, I didn’t realise so many others felt similarly.

The rest of the results are much less surprising to me. Fox News & Breitbart for their political nature (DailyMail’s presumably in the same bucket). Facebook because it’s a horrible turd on so many levels but most pertinently because it essentially can’t be viewed without being signed in². Instagram apparently gets marginally less hate from the computer geek community, even though it falls into the same bucket as Facebook in that respect (and others).

I’m slightly surprised to see w3schools in the list, as though I’m aware it’s not “cool” to use it as a web developer, frankly I use it all the time because they have better summaries of key CSS features than Mozilla’s developer docs.

1. When my primary search engine fails me. Google has a more extensive index than Bing / DuckDuckGo, so sometimes it’s the only one of the Big Three that can find a relatively niche or unloved page. ↩︎
2. Incidentally, if your restaurant’s menu is only available on a Facebook page, congratulations, you will never count me amongst your customers. ↩︎

It immediately reminded me of the Shark 5 re-implementation of the profiling interface between the kernel and Shark. Shark 5 of course didn’t actually survive to birth – it was snuffed out by politics and, admittedly, a bit of our own hubris in the Shark team – but that underlying infrastructure did, as the guts of Instruments‘ Time Profiler and System Trace features.

In Shark 4 and earlier, userspace would allocate a single big buffer of memory for the profiling data, hand it [back] to the kernel¹, the kernel – specifically the Shark kernel extension – would write into that buffer, then when the buffer was full profiling would end². The userspace driver – Shark – could technically start profiling again immediately, but of course you’d often have a gap in your profiling – potentially a big one, for the more expensive profiling modes (e.g. System Trace) or on heavily-loaded machines. Shark didn’t run with elevated priority, as far as I recall, so it could get crowded off the CPU(s) by other programs. All communication between user & kernel spaces was via Mach messages, which are ultimately syscalls, and was time-sensitive since kernel- & user-space had to be in lock-step all the time.

For Shark 5, we wanted to up the ante and remove limitations. We wanted to be able to record indefinitely, limited only by disk space. More importantly, we also wanted to be able show profiling results live.

So, we – and I use the term loosely, as I think it was mostly Rick Altherr with possibly the help of Ryan du Bois – came up with a mechanism that’s very similar to io_uring, but years earlier, in ~2008. Userspace would allocate multiple smaller buffers – also easier to acquire given the requirement to be physically contiguous – and hand those to the kernel somewhat as needed. Userspace merely needed to stay ahead of the kernel’s use, which didn’t necessarily mean pre-allocating all the buffers. When each of those individual buffers filled up with profiling data from the kernel, they’d be made available back to the userspace side³. Shark (in userspace) would optionally – if in ‘live’ mode or if running short on empty buffers – process those buffers while profiling was still actively occurring, freeing them up to go back into the ring for the kernel’s use. It did mean you slightly increased the probability of perturbing the program(s) under profiling, but that was mostly just a matter of ensuring the buffers were large enough to sufficiently amortise the cost of their processing & handling. I don’t recall the extent of the processing, but I think it was not necessarily much more than copying the data out into non-wired memory (maybe even a memory-mapped file?). That was cheaper than allocating and wiring new buffers.

It worked really well, and is possibly still the implementation used to date – while it appears a lot of the frameworks (e.g. CoreProfile) have disappeared from macOS, presumably rewritten somewhere else, I’d be surprised if the kernel-user interface itself has changed much. It was pretty much perfect.

The creation of this new, superior profiling system also spurred me to create rample, a CLI tool kind of like top which showed you what your CPUs were doing in real time⁴ as a heavy tree, much like you’d get in a Time Profile in Shark (and later Instruments). It was super quick to write – just a day or two, I believe – and yet I unexpectedly found that it was more useful than Shark itself. Being able to see – practically instantly by virtue of how lightweight it is to launch a simple CLI program – what’s chewing on the CPU, including inside the kernel, was incredibly useful. Instruments is the closest you can get today on a Mac, but it’s super slow and clunky in comparison. It’s one of my top regrets, of my time at Apple, that I didn’t get rample into Mac OS X, or at least into the Dev Tools.

1. I don’t recall precisely why, but it was apparently better (or only possible?) to allocate the buffer in userspace, even though that was ultimately done with syscalls serviced by the kernel. I think it had something to do with being easier to handle failures, like being out of memory, in particular contiguous memory – the memory was wired during use to ensure the kernel wouldn’t have to deal with the practical nor performance problems of page faults while recording profiling data. ↩︎
2. Which was a problem because generally profiling sessions were specified to run for a specific time duration, by the end-user. So the buffer had to be sized to avoid filling prematurely, which meant wasting memory. ↩︎
3. Technically they were always available – we didn’t bother changing access permissions – but the userspace app would coordinate with the kernel via an elegant lockfree state machine. That state machine was a superb design – it deserves its own detailed nostalgia trip. ↩︎
4. Well, updated every second or so, like top. ↩︎

There was of course the bug whereby encrypted external drives no longer mounted automatically. That was pretty special, because it reveals that Apple doesn’t test encrypted external drives anywhere in their QA process – possibly not even ad-hoc internally, since it’s hard to image that an Apple employee stumbling on this bug by accident would not bother to report it. Or maybe they did but even Apple’s own employees’ bug reports are ignored. Consistent, at least.

Since virtually every external drive should be encrypted, this suggests Apple either believes nobody actually uses external drives, or that they don’t care. At least that bug was mostly fixed in 14.1, although it took Apple a full month to get around to it.

Radical pink redesign

But perhaps the most glaring bug, in a very literal sense, was the rampant pinkification of the GUI.

As you can see, Sonoma brought with it a penchant for tinting things a gross light pink colour, including the rectangular extents of images that are otherwise invisible because they’re transparent.

WTF?!

I spent a lot of time debugging this. Over days, weeks, and months. Trying out new avenues as they came to me, even repeating some web & forum searches over and over again in frustrated desperation.

I only just today figured out what’s going on. 84 days later.

One of my first instincts – right on day 0 – was that it was some Quartz Debug thing, because it looks a lot like some of the tinting options Quartz Debug has for things like flashing screen updates or showing opaque regions. But the Quartz Debug app insisted everything was disabled. I enabled everything anyway, and then disabled it all again, in case that would jog something free. No bueno.

I then spent the better part of three months trying every superstition, every rumour or snippet of hearsay online, that might in any way related to this problem or conceivably fix it. Nothing helped.

Epiphany

It was last night, when I was fiddling with some images in a gallery here on my website, that I discovered that adjusting the CSS dimensions of an image by just a single pixel could pinkify them. Most importantly, restoring the dimensions to the actual image dimensions would fix them. I’d already observed something similar to that, in various apps and websites, but it felt different to be directly triggering the problem myself.

That kept me up half the night, tossing and turning in bed, highly suspicious that this was a pivotal clue, but unsure what it pointed to.

Eventually, it led me right back to my very first instinct – Quartz Debug – and the tentative conclusion that this was in fact a Quartz Debug problem even if Quartz Debug claimed otherwise. I become suspicious that the app was either not showing me the full range of settings, or just plain wrong about them.

And that was the insight I needed – right on the very first page of search results for “quartz debug defaults” was a link to Testing and Troubleshooting High-Resolution Content. Buried behind a disclosure triangle near the bottom is the magic string:

CGContextHighlight2xScaledImages

That was set to YES, on my system.

Deleting it fixes the issue once affected apps are restarted (something Apple’s own documentation fails to mention – kind of an important step, Apple!).

defaults delete -g CGContextHighlight2xScaledImages

The magic incantation to exorcise Barbie

In theory you can also disable this from the Quartz Debug app itself, it’s just hidden – in the Tools menu, not the actual settings window, is an item labelled “Color 1x Artwork”. Note how the setting applies even when Quartz Debugging is disabled.

So, wait, how did this come about?

I have no idea why this only cropped up when Sonoma was installed. Either Sonoma turned this on (unlikely…?) or it was already on yet wasn’t having any effect in prior OS releases (from 2017 onwards, at least, since my iMac Pro never exhibited this issue before Sonoma).

I know for sure that this setting did work at least going back over a decade, as I remember using this – on a different computer – for doing then-new-fangled Retina development… but I’m pretty sure I’ve never set this on my iMac Pro since it way post-dates the introduction of Retina. Conceivably it was pulled across by Apple’s Migration Assistant. Though, again, why did no earlier version of the OS, going back over a decade, exhibit this behaviour if it was supposedly enabled the whole time?

I know it’s not just me that’s been afflicted by this to date, as I found a sporadic few reports of this going back years, e.g. Highly abnormal graphic glitch on yosemite / retina: all white backgrounds appear pink. It pisses me off immensely that this page has exactly the keywords I was searching for right from the start – right in its title! – yet neither Bing nor Google can find it.

Apple’s response? “Fuck you” in mime.

I actually filed a pretty detailed bug report with Apple about this, the day I discovered it (September 19th, 2023). FB13190068.

Of course, they never responded in the slightest.

I provided a bunch of example images, even a screen recording to show how the pink comes and goes in certain circumstances.

I studiously tested every new macOS update and updated the bug report to note that they each had not fixed it.

The morale of this is, as always, that filing bug reports with Apple is an infuriating waste of time. Mea culpa.

So I didn’t buy an M1 Ultra.

The M2 was announced in July 2022. The M2 Ultra wasn’t announced for another eleven months, in June 2023, and offered only a small performance gain over its predecessor. There were rumours that the M3 was right around the corner (and it was, released just four months later). Worse, the M3 Pro and Max were released simultaneous to the base M3, and the M3 Max actually outperforms the M2 Ultra in many workloads.

So I didn’t buy an M2 Ultra.

So now the question is: when will the M3 Ultra come out? If it takes eleven to sixteen months, like its predecessors, then we’ll be expecting the M4 by then, and this sad cycle will have repeated yet again.

Will I buy an M3 Ultra?

I hope so, but it’s up to Apple.

A recent example: this set of services have poor uptime so we need to put another service in front of them, and this will – apparently, magically – make them have better uptime.

This might be true in specific cases. If the abstraction layer utilises caching, for example, it’s conceivable it’ll be able to continue operating in at least some capacity while an underlying component is (briefly) unavailable. Or maybe the reduced load to the downstream service(s) will alleviate pressure on racey code and dodgy GC. But this is practically never a factor, in the real-world examples I see. It really does seem to be “step 1, collect underpants … step 3, profit”.

The best you could say is that it accidentally arrives at a practical benefit, despite itself.

It’s important to separate the actual causes of improvement from the red herrings. I can only assume that the confusion between the two is what has allowed this blatantly illogical practice to gain some ground. “We implemented better input validation and an extra layer, and things got better”, and bafflingly nobody ever seems to question how “and an extra layer” contributed.

Now, adding an abstraction layer might have other direct benefits – e.g. the opportunity for use-case-specific APIs, better alignment with organisational structure, etc – but reliability is unlikely to be one of them. Particularly if by ‘reliability’ one largely means availability. Again, it’s crucial to understand the actual causality involved, not miscategorise coincidence.

The logical thing to do in the face of an unreliable component is to simply make it reliable. Anything else is just making the situation worse.

I started out as a one-person shop, doing my own things, selling shareware. I had no manager nor technical lead. I had to make all my own decisions, in all aspects, without guidance or assistance.

Subsequently, during my four years at Apple, I did have a manager, but they focused on people, not the technical – myself and/or my colleagues collectively made the technical decisions, and provided technical leadership, and effectively set the product direction. My managers were there to make that as easy as possible for us.

Over my nearly eight years at Google, I observed the tail half of a major cultural transition for Google. Long before I started, Google had explicitly laid down a culture where managers were not product / technical leads. The two roles were physically separated, between different people, and they operated independently. Managers focused on people – career growth, happiness, basic productivity, & skills – while tech leads focused on the technical, the product. In fact the manager role was so principled about focus on people that managers would sometimes help their direct reports leave the company, if that was simply what was best for those people for their own success & growth. And, to be clear, not in a “you aren’t working out” sense, but for engineers that were excellent and simply didn’t have deserved opportunities available to them at Google.

By the time I joined, that culture was half-gone, but still present enough in my division for me to experience it. But by the time I left the culture was heavily weighted towards managers being technical leads.

In my nearly three years now at LinkedIn, I’ve completed that arc. LinkedIn culturally & executively emphasises managers as technical / product leads even moreso than Google ever did. As far as I’ve been told, LinkedIn always has (meaning, this is presumably the culture Yahoo had too, from which LinkedIn forked).

Having experienced most of this spectrum, I finally feel qualified to pass judgement on it.

I immediately, intuitively recognised & appreciated this at Google, but now I’m certain of it.

People management & (technical) product leadership are fundamentally at odds with each other. The needs of individuals are often at odds with the needs of the product. The product might need Natalie to really focus on churning through a bunch of menial tasks, but to evolve, Natalie might really need design experience & leadership opportunities.

Having one person (in authority) try to wear both hats creates conflict, bias, and inefficiency. It discourages dialogue, because you can never really trust where the polymorph stands. The roles require different skillsets, which rarely coexist in a single person and in any case are difficult to keep up to date in parallel. Context-switching between them is burdensome. It creates a power imbalance and perverse incentives.

Even if an individual is exceptionally talented at mitigating those problems, they simply don’t have the time to do both well. Being a product or technical lead is at least a full-time job. Likewise, helping a team of any real size grow as individuals requires way more hands-on, one-on-one attention than most people realise. It’s hard enough being good at either one of them alone – anyone that attempts doing both simultaneously ends up doing neither effectively.

I’ve had the opportunity to be both a technical lead only and a manager only. This is quite rare in the tech industry. I deeply appreciated being able to focus on just one of those roles at a time. I could be consistent, deliberate, and honest. I could, as a manager, tell people exactly what I thought they should or shouldn’t work on, irrespective of what the product(s) need, because I knew the technical lead(s) would worry about those angles. Conversely, when I was a technical lead, I could lay out what was simply, objectively best for the project, uncomplicated by individuals’ interests. In either case, there was real, other human being that could be debated with, as necessary, to find happy mediums.

Yet beyond just being more efficient and effective, the serendipitous consequence was that it gave agency to the individuals – whenever a conflict arose between people and products, it was revealed to them, and the implicit decision about it at least in part theirs to make. Most importantly, they knew that whichever way they leaned they had someone in their corner who had their back.

(Of course, sometimes they didn’t like having to make that decision, but putting it on them forced them to take control and responsibility for themselves, and evolve into more confident, happy, motivated developers.)

I suppose it’s no surprise that companies tends this way – to conflate people with products. These days, for many big tech companies, people literally are the products, and their humanity inevitably stripped away in the process. People are “promoted” into management from technical positions, and often by way of the Peter Principle, are not actually good people managers, nor able to relinquish their former role and ways of thinking. A hierarchy of technical leads in manager’s clothing becomes self-sustaining, self-selecting, and self-enforcing.

The question is: what’s the antidote?

Acknowledgement: I was inspired to pen this post by reading Tanner Wortham‘s Why Manager as Product Owner Will Usually Fail, which is essentially positing the same thing albeit in different terminology.

I was pleasantly surprised.

There are multiple ways to do it, apparently. One would think that there’d be the correct & working packages already available through apt, but just as with Docker, one would be wrong. In the case of Swift, there is the Swift-ARM site & group that does provide a package repo with pre-built binaries (and various tutorials on using them), but oddly they don’t provide the current version of Swift.

An alternative is the buildSwiftOnARM Github repo. They merely provide tarballs, which is slightly suboptimal, but very straightforward and they have tarballs for essentially every version – major and minor – of Swift to date. The git history also indicates that they’re very prompt about building tarballs as new versions are released.

Better yet, they provide a couple of shell scripts to build Swift from scratch from source. Only a couple of dependencies (e.g. clang) need be pre-installed, which can be done quickly & painlessly via apt.

Installing from source is presumably the least reliable approach, but since I had already resigned myself to a miserable experience, I figured I might as well go all in.

However, it works. Perfectly. Sure, it takes some time to pull down the huge source base for Swift and all its core dependencies, and some time to build it (though not that long – hours, not days, contrary to what I read online). But the end result was a working toolchain.

It remains to be seen exactly how good or bad Swift development on Linux is, given the absence of the numerous Apple system libraries which are what actually distinguish macOS development above other platforms, but at least getting Swift itself installed is painless.

Sidenote: SATA to USB

The only real speedbump in the whole process, for me, had nothing actually to do with installing Swift itself, but rather the external storage situation on the Raspberry Pi.

The Raspberry Pi doesn’t offer SATA directly, unfortunately (let-alone any form of pluggable PCIe). MicroSD is a low-performance, low-reliability, and high-cost option. So to attach any significant storage you’re basically going either through Ethernet (e.g. NAS) or USB.

USB to SATA adaptors are a shitshow. I’ve tried at least half a dozen different vendors’ offerings over the years, and every single one has been super buggy. The one I newly acquired for my Raspberry Pi use proved to be no exception.

Long story short on that, the symptoms were I/Os taking incredibly long times to complete (many seconds each, serialised), and generally unusable performance. /var/log/messages contained countless pages of:

Oct 13 11:13:45 applepi kernel: [  234.087294] sd 0:0:0:0: [sda] tag#2 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD IN
Oct 13 11:13:45 applepi kernel: [  234.087306] sd 0:0:0:0: [sda] tag#2 CDB: opcode=0x28 28 00 77 3b ce 00 00 00 d0 00
Oct 13 11:13:45 applepi kernel: [  234.126541] scsi host0: uas_eh_device_reset_handler start
Oct 13 11:13:45 applepi kernel: [  234.277450] usb 2-2: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
Oct 13 11:13:45 applepi kernel: [  234.312541] scsi host0: uas_eh_device_reset_handler success
Oct 13 11:14:15 applepi kernel: [  264.805760] sd 0:0:0:0: [sda] tag#7 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD IN
Oct 13 11:14:15 applepi kernel: [  264.805778] sd 0:0:0:0: [sda] tag#7 CDB: opcode=0x28 28 00 77 3b d1 b8 00 00 48 00

Turns out this is an incredibly common problem with USB to SATA adaptor chipsets, that’s documented as such all over the web. Finding how to solve it was less trivial, because a lot of the advice given is either outright wrong or at least doesn’t work on Raspbian. The solution I found, via this random thread, was to simply add:

usb-storage.quirks=152d:1561:u

…to the end of the existing line in /boot/cmdline.txt (where 152d:1561 is the vendor & device IDs for the particular chipset used in my case). All other variations on this, involving adding similar magic incantations to files in /etc/modprobe.d etc, simply do not do anything on Raspbian.

For a couple of little home projects I need an always-on computer. In an ideal world, perhaps, this would be something like a Mac Mini. Powerful [enough], easy to install & maintain, runs anything & everything (including anything Linux through Docker or at worst a straight VM). Unfortunately, Mac Minis are surprisingly expensive – even nine year old models are a couple of hundred dollars at a minimum.

So, I decided to instead explore this Raspberry Pi thing.

I very quickly started wishing I hadn’t.

The whole process thus far has just been a series of absurd errors & frustration.

Acquiring a Raspberry Pi

Step zero, of merely buying a Raspberry Pi, is stupidly difficult. Virtually all the retailers officially listed on raspberrypi.org did not actually have the Raspberry Pi 4 in stock. Later I discovered that some of these same retailers, that list no stock on their own websites, are actively selling the Pi on Amazon. So I bought one through there, which is fine, but why doesn’t raspberrypi.org just say to use Amazon, if that’s really the only way to get them?

Next up was all the peripherals – the Pi by default doesn’t even come with a power supply, so it’s useless out of the box. A cursory internet search reveals a huge amount of FUD about power supplies for the Pi. I have no idea if it’s accurate or not, but given some relevant, egregious design flaws in the Raspberry Pi 4, it seems plausible.

Plus you need at a minimum some stand-offs, if not a full case, to prevent the Pi damaging the surface it’s placed on, or damaging itself through shorts.

And addressing those bootstrapping problems ended up sending me down a rabbit hole trying to find a cooling solution too, since it turns out the Raspberry Pi 4 is infamous for overheating and suffering severe performance – and presumably reliability – problems as a result.

In the end, I spent several hours just figuring out how & what to buy, and what is nominally “the $35 computer” cost over $100. Still without a case, even.

Sidenote: the Pimoroni Fan Shim for Raspberry Pi, while a little fiddly to assemble, does seem to work very well, and is quite quiet.

Booting a Raspberry Pi

This is the one part of the process thus far that’s actually worked mostly as it should. I downloaded the full Raspbian Buster image, following the installation guide, and using balenaEtcher to plop the image onto an SD card. It all worked, even with the Etcher app being a tad dodgy (e.g. it lets you select non-removable volumes, which you cannot possibly intend to flash Raspbian onto, which is unnecessarily dangerous). The Raspberry Pi 4 booted first time.

I tried to discern whether booting it headless from its birth would work. Officially it does not, but I found that baffling and dug further, reading countless online guides (e.g.), which seemed to suggest it is possible.

I learnt that there exists the raspi-config tool for headless setup, but it was unclear if it would really work, fully. Though I did the GUI set up process to be conservative, I’ve since used raspi-config quite a bit. Turns out, it not only does work just fine, but it’s actually necessary because the GUI install doesn’t do some important things (like resize the root file system to fill the SD card).

One thing which nearly blew the whole enterprise was when it came to join a wifi network. I have multiple wifi networks, all with [different] emoji for names. The GUI set up tool can’t handle emoji, rendering them as octal escape sequences. I don’t happen to have memorised the four-character byte codes of each emoji, so it was a tedious game of trial-and-error in which I tried every permutation of unreadable SSID & password.

Worse, it took multiple attempts, too, before it finally worked – I have no idea why it failed to join the network the first time or two, despite using the right password. To this day it still arbitrarily fails to join one of the networks, yet joins the other just fine – both are in the same frequency bands from the exact same router.

Aside: Raspberry Pi 4 as a desktop computer

Since my intended use is as a headless, touchless server, I played only briefly with it in the GUI, using a makeshift setup involving my TV (the only HDMI viewing device I’ve ever owned – lucky I had that at least!). It’s fine, but very sluggish – it was immediately apparent that nobody with any other options would ever try to actually use a Raspberry Pi 4 as a desktop machine. Just [cold] launching the web browser, before you even navigate to a website, takes up to a minute. And everything is uncomfortably small, with no apparent system configuration options available to adjust render scaling. Clearly Raspbian is not really intended to be operated at UHD resolutions.

Enabling Remote Access

Though I ultimately intended to use only SSH to interact with the Pi, I did want to have VNC available as an option in case I ran into anything which required using the GUI (again, based on the heavy bias in all the official documentation, and the uncertainty created by that as to whether GUI interaction is required or merely an option).

Turns out, VNC doesn’t work out of the box on a Raspberry Pi, unless you buy commercial, proprietary VNC software. A baffling collusion on the part of the Raspberry Pi / Raspbian people. You have to do additional work to make it actually work – work that’s completely undocumented in any official Raspberry Pi / Raspbian documentation. (at best you’ll find the interwebs littered with accounts & instructions on installing a non-proprietary VNC server in replacement, which presumably also works to solve this problem)

Installing Homebridge

It wasn’t actually my purpose in buying the Raspberry Pi, but I decided that – before I go down the meat grinder that is presumably getting Swift to work on the Pi, since the Pi sadly lacks support for Swift out of the box – I figured I’d just real quickly install Homebridge, since I do have a couple of devices I’ve long wished would work with HomeKit.

Ugh.

What a fucking dumpster fire.

You can install Homebridge raw, but since it’s written in Node.js, I didn’t want it going into my real, bare system – infecting it with npn and JavaScript and all that horror.

This would be a perfect opportunity for Docker, and as one might expect there are many guides on how to install Homebridge via Docker.

Installing Homebridge Docker

Sadly – and frankly bizarrely – running Homebridge through Docker isn’t officially supported.

This 3rd party guide appeared to be the best, based on this third party project to support Homebridge in Docker. Step zero, of course, is to install Docker itself. Surely that’s trivial. It’s Docker. What doesn’t run Docker these days? Hell, macOS runs Docker and it doesn’t even support containers. I was baffled that Raspbian didn’t include Docker pre-installed.

Many, many hours later, it still wasn’t working. One would think that Docker, of all things, would be a seamless thing to sudo apt install, but far from it. For example, the official Docker apt repo for Raspbian tries to install some ‘aufs-kdms’ as a dependency, even though – turns out – it’s not a real dependency and doesn’t even compile on Raspbian. WTF?!

So that wasted hours, in figuring that out – predominately consumed in trawling the interwebs for a solution. After many hours and reading through dozens if not hundreds of StackOverflow, blog, and similar sources quoting similar issues and offering bogus remedies, I finally found a thread that’s actually helpful.

The worst was yet to come.

At some point in this process something also screwed with my Pi’s boot settings to force the root directory to be mounted – at boot – as an overlay with writes going to tmpfs (i.e. nowhere). That wasted yet more hours as I painstakingly root-caused why my Raspberry Pi suddenly had alzheimers (and lost a lot of progress otherwise on installing Homebridge, too).

The web utterly failed in this case, as I couldn’t even find how to disable overlayfs. All I got, mockingly, was endless articles explaining how to enable it and voluntarily ruin your day.

Even just figuring out that it was overlayfs that was screwing me took quite some time, since the first failure symptom was a baffling error message when trying to start dockerd:

failed to start daemon: rename /var/lib/docker/runtimes /var/lib/docker/runtimes-old: invalid cross-device link

With love and fuck you, dockerd

Ultimately I found a fix, in part thanks to this forum post which had enough transparency on enabling this bullshit situation that I could deduce how to disable it – long story short you need to mount the SD card on another, working computer and remove ‘boot=overlay‘ from /boot/cmdline.txt and ‘initramfs initrd.img-4.19.75-v7l+-overlay‘ from /boot/config.txt.

How that ever got enabled I have no idea. Absolutely no commands I ran had anything to do with that at all. Evidently something buried inside Docker installation and/or execution performs this system lobotomy. Even then, I’ve since reviewed every single command I ran, and nothing seems even remotely like it could nor should have caused that.

Despite ultimately defeating all this failure, I was greeted by merely another fatal failure, just as inscrutable as the last:

$ docker-compose up -d
ERROR: Couldn't connect to Docker daemon at http+docker://localhost - is it running?

If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.
$ ps auxww | grep docker
root       427  0.6  1.4 966720 58856 ?        Ssl  15:42   0:01 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
pi        1942  0.0  0.0   7348   472 pts/0    S+   15:46   0:00 grep --color=auto docker

Turns out this was because my user (‘pi’, the default) wasn’t a member of the ‘docker’ group. I’d added it previously, but it must have been under the tyrannical overlayfs regime, and all memory of that event purged. Adding it again (then logging out & back in) fixed it (sudo usermod -aG docker pi btw).

This is why Linux can’t have nice things

In summary, Linux in general, and certainly Raspbian specifically, continues to be the same giant clusterfuck it’s always been. I’m no Linux novice – I’ve been writing software for Linux for over a decade as my day job. I’ve just had the luxury of teams of dozens if not hundreds of other engineers to insulate me from the bare wiring that is installing, configuring, & maintaining a Linux installation.

At this point I’m two days in and have only just gotten Docker working. For all the time I’ve wasted I’ve completely blown the price savings between a Raspberry Pi and even a brand new, $800 Mac Mini.

And I still haven’t even started installing Swift, let-alone actually running my Swift app on the Raspberry Pi, which – contrary to where all my time has gone on this project – is the actual purpose of this whole sad enterprise.

The recent kerfuffle with Microsoft Edge vs YouTube was particularly interesting since while I have no specific knowledge of that instance, I certainly do have some cultural insight from nearly eight years working inside the so-called Chocolate Factory – though not on any web stuff, to be clear, so my experience is in the broadest internal sense.

Like everyone else last week, I was trying to determine how much intent or malice was behind Google’s actions, but with a marginally more informed perspective – or at least a relatively unusual one.

Permit me to first provide some larger context, though.

When I worked at Apple, back in ~2006-2010, I insisted on using Camino, because it was superior to Safari at the time (which, among other flaws, was particularly crashtastic in its early years – an attribute which thankfully is long gone, but has burned itself permanently into my emotional memory).

That choice to not use Safari caused periodic issues because some Apple-operated websites wouldn’t work properly with anything but Safari. When I reported those issues internally, the typical response was “we only support Safari”. From the perspective of Apple, once they had their own web browser, that was all that mattered. Thankfully it wasn’t a huge issue since the web wasn’t that important to day-to-day work at Apple, as they used native applications for most things (sidenote: I still miss Radar… I didn’t miss Xcode for a long time, until I was forced more recently to use IntelliJ). And certainly the world outside Apple didn’t care about this cute little ‘Safari’ thing, at the time.

My experience at Google was essentially the same.

The vast majority of Google’s internal websites do not work properly in any browser except Chrome. This is a very real problem since it’s practically impossible to perform any job function at Google without using their internal websites heavily, since Google is so dogmatically opposed to native applications. Google has worked extremely hard to [try to] make it possible to do almost anything through Chrome (often to the point of absurdity).

Ironically even Microsoft – whom I currently work for, via LinkedIn – are on the Chrome bandwagon, as some of their websites – that I am required to use for work – require Chrome.

Most interestingly – and distinct from Apple’s behaviour, where dysfunction in browsers outside their own was predominately based on actual functional differences between them – this ‘requirement’ to use Chrome is often not because of any actual, functional dependency on Chrome, but because Google’s (and Microsoft’s) web developers will specifically require a Chrome user-agent, and explicitly block any other browser. While this is easily worked around by spoofing the user-agent field – and is how I know that the purported Chrome requirement is usually a fallacy – it emphases the mentality at Google:

There is no web, there is only Chrome.

This is, I believe, the crux of the matter in not just this Edge vs YouTube issue, but with web development broadly in space & time. The vast majority of web developers don’t create content for The Web, they create content for a browser. One browser, usually.

I saw it unashamedly unfiltered inside Google, but it inevitably leaks out in time – through things like carelessly & needlessly crippling other browsers’ performance on YouTube.

While today that browser happens to be Chrome, before Chrome existed there was still always that browser – e.g. the 90s and much of the 00s was defined by Microsoft Internet Explorer’s dominance and the refusal by the majority of so-called web developers to create content for The Web rather than just Internet Explorer. (Of course, back then The Web really was almost synonymous with Internet Explorer, with ≥90% marketshare for many years, so at least it was more pragmatic back then than Chrome obsession is now.)

So, I’m actually sceptical that the YouTube team explicitly sabotaged Edge – rather, I think it’s just one of endless cases of web developers not really caring about The Web – ignorance & indifference, in other words, rather than [outright] malice. But just as caustic & dangerous.

What particularly concerns me today is that it’s not quite the same as the terrible 90s and 00s. Then, when Internet Explorer dominated, the vast majority of important websites were not operated by Microsoft.

Today, Google’s web properties are dominant in mindshare if not marketshare to the point of essentially being monopolies (certainly in the case of YouTube, at least) – way more than anything Microsoft ever did was.

More to the point, 90s web developers chose to develop for Internet Explorer exclusively – they were not coerced into it for the most part, nor firmly bound to that choice, because their corporate masters did not have a horse in the browser race and were pragmatically & unemotionally going for audience reach. A meritocracy was possible, and existed to a degree, and was essential to the rise of Firefox, my nostalgic Camino, and yes, even Chrome.

Google very much does have a horse in that race, and I know – from many years of experience inside Google seeing their unfiltered opinions – that they absolutely do want Chrome to become the only horse in that race. Not because of some comically-evil secret council scheming at the heart of Mountain View, but because they culturally & corporately just don’t care about anyone else. Modern Google is just as paranoid, fearful, power-hungry, and ruthless as 90s-era Microsoft ever appeared to be – Google want control, and the browser today is as fundamental to control as operating systems ever were.

Given all that, my fear is that there’s no longer a practical way for another browser to compete on merit with Chrome – anymore so than a third party app store can compete on iOS, for example.

Chrome is open source in the literal sense, but not in the more important governance & existential senses. The only way to give The Web a chance is to remove any corporate browser bias from the minds of the top websites’ developers – Google’s web developers. (Or, technically, to just supplant Google’s numerous dominant web properties. Good luck with that.)

This assuredly won’t happen anytime soon by way of government intervention, given the current U.S. political circumstance, but it is conceivable that Google themselves would perform this surgical separation voluntarily, for the good of The Web.

Sadly, I fear that’s unlikely in an era post-“Do no evil”.