Undocumented – Wade Tregaskis

Recording audio to an iPhone via a Tascam Portacapture X8

Thu, 22 Jan 2026 06:04:04 +0000

It is possible to use the Tascam Portacapture X8 as an ADC, input converter, and mixing board for an iPhone, but there’s a few things you’ll need to know.

Thankfully, any USB-C or USB-C-to-Lightning cable will do

You just connect one end of the USB-C cable to the Portacapture X8 (to its built-in USB port) and the other to your iPhone (USB-C or Lightning work, as suits your particular model of iPhone).

The Portacapture X8 can both record audio via USB and output it. It will automatically output it to any connected device that accepts it. As far as I can tell it uses the master track mix (there doesn’t seem to be any way to send multiple tracks simultaneously – I assume the USB audio protocol only allows a plain stereo transmission).

You must use 48kHz sampling

If you don’t, you’ll quickly get an error dialog on the Portacapture X8 saying “USB FS Mismatch”. That’s its daft way of trying to say that the receiving USB device won’t accept the sampling frequency it’s outputting. Why it can’t have a coherent error message, we’ll probably never know.

You can set the sampling rate in the “General Settings” app, under “Rec Settings” (along with the file format and bit depth / rate, though it doesn’t seem to matter what those are set to w.r.t. iPhone compatibility – they affect only recording to the microSD card).

You don’t have to use the iPhone to provide power

By default the Portacapture X8 will try to use the iPhone to provide power, rather than its batteries. But it doesn’t trust USB power sources – it will ask, via a dialog, “Is the AC adapter 1.5A or more?”. On USB-C iPhones you can power the Portacapture X8 over USB – by selecting “Yes” – although it will drain your iPhone’s battery. But whichever option you choose, the Portacapture X8 will then refuse to provide phantom power to your mics. If you’re not using phantom power then no worries, but if you are you must change the Portacapture’s settings – in “General Settings” app, under “Power/Display”, you must set “Power Source Select” to “Battery” instead of “Auto”. That will essentially turn off the Portacapture’s desire for power from USB, leaving USB as an audio channel only. With it powering itself from its batteries, the phantom outputs will work like normal.

You don’t have to record on the Portacapture X8

It passes audio through to the iPhone automatically, even if you’re not actively recording.

Note: I vaguely recall having some issues with this not happening if you change some settings… it’s possible that things like “Pre Rec”, “Auto Rec”, or “Rec Pause” affect this. I have all those off and it works as I’ve described.

Backblaze seemingly does not support files greater than 1 TB

Thu, 02 Jan 2025 23:27:59 +0000

For nearly a month now, Backblaze has been fixated on a particular file of mine, that happens to be over 1 TB in size. Backblaze seemingly uploads it completely, but then on the next backup it uploads it again, even though it has not changed (in eight years!). Ad infinitum.

Using their Explainfile tool to dig into the log files, the clue seems to be:

  - line 288 - 2024-12-16 16:16:17 0000000646 - ERROR: UpdateBzDoneRegardingFlsToBeExp - Z_B_TOO_MANY_CHUNKS bz_done_ line chunk related, numBytesInLargeFile=1099512156951, totNumChunks=104858, bz_done_line_is: 5	! …
  - line 522 - 2024-12-16 16:18:46 0000000646 - ERROR - bz_done_ INCONSISTENCY_FOUND - 20241216161846 - BadBadBadChunkRecord hexAsciiVal=0x78 - AfterBzdoneLargeFileAnalysis: chunkSeq=100001, highestChunkSeqSeen=104857, fileIdOfLargeFile=00000000002c53cd, dateTimeOfLargeFile=20231217091843, XYXBXXX_FILE_NAME: …

Admittedly I’m guessing somewhat, since that’s a rather reader-hostile log message, but the combination of the Z_B_TOO_MANY_CHUNKS error mnemonic and chunkSeq=100001 (because of its proximity to the arbitrary round number 100,000) strongly suggests that Backblaze is imposing a 100,000 chunk limit. Since chunks are 10 MB each, that’s exactly 1 TB.

This is unequivocally at odds with what they claim repeatedly on their website, on pages like What Backblaze Backs Up and File Sizes.

It’s not clear to me why this is suddenly a problem; is this a newly-imposed limit? It’s possible that a month ago I removed some exclusion on the file, but I don’t remembering doing that and I can see no reason why I would have excluded it to begin with. If it is newly imposed, that would imply it’s also retroactive – that Backblaze actually deleted the existing backup of the file from their servers, thus causing the client app to try uploading it again.

I reached out to their technical support, of course, but thus far have only received mindless responses – restart your computer, reinstall Backblaze, etc.

Update

I received no further response from Backblaze’s technical support. They asked me to send them the log files, which I did on January 2nd, 2025, and they never responded again.

As of this update (February 4th 2025) their website still falsely advertises support for files of any size.

Addendum

I was surprised to see that many folks on HackerNews were surprised by the idea of a 1 TiB file. I certainly agree that’s large, but it doesn’t seem unusual or inexplicable to me. In my case, this particular “problem” file is an encrypted, compressed disk image of the boot drive of a prior computer, that I saved when I upgraded to my current computer.

It’s true that I could probably throw it out at this point – it was just a precaution in case I forgot to migrate something over, so now (eight years later) it seems that either I made no such mistake or whatever I forgot to migrate doesn’t matter anyway. For now I’ve just manually excluded it from the backup, to work around Backblaze’s bugs.

There are other cases in which I’ve had files over 1 TiB, though – e.g. video files:

With some cameras and recording modes (e.g. documentarian, interviews) it’s in principle easy to exceed 1 TiB per file. e.g. the Nikon Z9 & Z8 record around 700 MB/s for 8k60 N-RAW, which is about 24 minutes per TiB.

Note that I don’t recall if I personally have ever actually exceeded 1 TiB this way. I mention it mainly for illustration. It’s also possible that the Z9 & Z8 shard large recordings into multiple files (I don’t recall seeing this in years – not since the 4 GiB per-file limit of cameras a decade ago – but perhaps I’ve just not had a single recording large enough).
Usually (for me) it’s output files that are largest, since they can combine many clips. I use Final Cut Pro and its video compression capabilities aren’t great, so I export essentially lossless ProRes and then use ffmpeg or Handbrake for the real compression. ProRes 422 HQ is nearly a gigabyte per second for 8k60, so it takes less than twenty minutes of video to exceed 1 TiB. Fortunately these large intermediaries only have to live as long as the final compression takes (though that can be days, especially with the latest formats like AV1).

presentedWindowStyle is not windowStyle

Mon, 23 Sep 2024 23:12:29 +0000

This post is mostly to herald a pretty good Apple bug report response, which as we know is a too-rare event. But it might also help others with this confusing SwiftUI API.

What’s the difference between presentedWindowStyle(_:) and windowStyle(_:)?

Well, one does something, the other doesn’t, basically.

I tried using the former, and observed that it never has any effect. I filed FB14892608 about it, a month ago. While the long delay isn’t great, the response I got today was actually pretty helpful:

We’re sorry you ran into trouble with that API.

To adjust the style of a window group’s windows, you have a couple of options. If all of the windows in the group should have the same style, then using the windowStyle() scene modifier is what you want:

struct MyApp: App {
    var body: some Scene {
        WindowGroup {
            ContentView()
        }
        .windowStyle(.hiddenTitleBar)
    }
}

If you need to adjust the style on an individual basis using state for that window, there are also some related view modifiers:
struct MyApp: App {
    var body: some Scene {
        WindowGroup {
            ContentView()
                .toolbar(removing: .title)
                .toolbarBackgroundVisibility(.hidden, for: .windowToolbar)
        }
    }
}

A simple apology followed by clear and specific instructions on how to actually achieve what I wanted. I genuinely applaud Apple – or perhaps I should more specifically thank the individual(s) within DTS that actually provided this response. Either way, it was a very pleasant surprise.

Now, it still leaves unanswered the question of what presentedWindowStyle(_:) is supposed to be for, and in what situations it actually does anything. So not a perfect reply, per se, but really the proper solution for that is either:

Fix the API to not have such confusingly similar modifiers.
Improve the documentation to:
- Explicitly point out the other, similarly-named modifier.
- Distinguish them.

Currently the documentation is just:

Sets the style for windows created by this scene.
windowStyle(_:) documentation

Versus:

Sets the style for windows created by interacting with this view.
presentedWindowStyle(_:) documentation

The latter might technically convey something important here, in the created by interacting with this view part, but so much in SwiftUI is modifiers stuck haphazardly in weird and arbitrary places, that I’m been conditioned to ignore the fact that I’m often applying modifiers to views that don’t actually affect those views. And for all I know in SwiftUI parlance views do “create” and/or “interact” with their parent windows (a lot about SwiftUI is backwards, part of its nature as a declarative API).

Especially if that’s the first candidate API you stumble across, when searching for a way to style a window, it’s very easy to presume it’s the right API. Why would there be multiple APIs for styling the window; why would you continue searching after finding one?

Similarly, the presentedWindowStyle(_:) variant is the only one that appears in Xcode’s auto-complete if you try to add the modifier to your main view, which is both where you sometimes have to add window-level modifiers and also just a really easy mistake to make (instead of adding the modifier to the WindowGroup, one indentation level up).

Lastly, it doesn’t help that I can’t find any situation in which presentedWindowStyle(_:) has any effect, even knowing it’s not intended to style the parent window. One might assume it’s somehow related to sheets or somesuch, but apparently not? Presumably I’m overlooking some use-case – or maybe it doesn’t actually do anything on macOS, only iDevices? I welcome clues or tips.

Matching prefixes in Swift strings

Thu, 02 May 2024 01:46:02 +0000

How do you determine if one string starts with another, in Swift?

Surely that’s exactly what the hasPrefix(_:) method is for:

let greeting = "Hello"
let sentence = "hello, this is dog."


if sentence.hasPrefix(greeting) {
    print("Hi!  Nice to meet you!")
} else {
    print("No can haz etiquette?")
}

No can haz etiquette?

Wot?

The problem is that hasPrefix is not meant for general use with human text; it’s barely better than a byte-wise comparison. It only guarantees that it won’t be fooled by mere differences in Unicode encoding, which is a good start, but not remotely sufficient for general use.

Let’s step back a bit, and first consider the slightly simpler case of just comparing two whole strings. We can worry about the prefix-matching aspect later.

NSString (from Foundation) provides Swift Strings with a variety of more powerful comparison methods, such as caseInsensitiveCompare(_:) (which is really just an alias for compare(_:options:range:locale:) with the caseInsensitive option).

let a = "hello"
let b = "Hello"

if a.caseInsensitiveCompare(b) {
    print("Hello indeed.")
} else {
    print("Hmph… rude.")
}

Hello indeed.

So that works. For case sensitivity. But what about other situations?

let plain = "cafe"
let fancy = "café"

if plain.caseInsensitiveCompare(fancy) {
    print("Right, either way it's a shop that sells coffee.")
} else {
    print("So… no coffee, then?")
}

So… no coffee, then?

Well, shit.

It may vary in other languages, but in English “café” is just an alternative spelling of “cafe”, and you almost always want to consider them equal. In fact, in English it’s basically never really required that you observe accents on letters – some are technically required, such as blasé, but English speakers are very blase about such things. Unlike e.g. Spanish, with n vs ñ, accented letters are not considered distinct letters in English.

But, letter accents may creep into English text anyway (just like spoken accents). Some people prefer them, for any of numerous reasons, like:

In proper nouns out of respect for the so-named.
To honour words’ roots in other languages.
For technical correctness of pronunciation.
Just aesthetically.

So you do need to support them, which means accepting and preserving them, but (usually) otherwise ignoring them.

Ah, but wait, the documentation for caseInsensitiveCompare(_:) has a footnote which surely addresses exactly this problem, albeit obliquely:

Important

When working with text that’s presented to the user, use the localizedCaseInsensitiveCompare(_:) method instead.

No worries – we’ll just use that instead:

let plain = "cafe"
let fancy = "café"

if plain.localizedCaseInsensitiveCompare(fancy) {
    print("Finally!")
} else {
    print("Oh come on!")
}

Oh come on!

It turns out this mistake is made by most of the String / NSString methods of similar ilk. And the discrepancies are inscrutable – e.g. localizedStandardCompare(_:) doesn’t handle accents correctly but localizedStandardRange(of:) does.

Long story short, you need to base most (if not all) your string comparison on compare(_:options:range:locale:) or its sibling range(of:options:range:locale:), because the other string methods don’t work properly.

So, with compare(…) you can do e.g.:

let plain = "cafe"
let fancy = "Café"

if .orderedSame == plain.compare(fancy,
                                 options: [.caseInsensitive,
                                           .diacriticInsensitive],
                                 locale: .current) {
    print("Finally!")
} else {
    print("😤")
}

Finally.

But there’s two other options you should almost always use, which are easy to overlook:

widthInsensitive.

In all the Latin-alphabet languages of which I’m aware, there is no notion of “width” and therefore no issues with width [in]sensitivity. It seems it most-often comes up in Japanese, where for historical reasons there were multiple versions of the same character that merely different in their visual dimensions. e.g. “カ” and “ｶ”. They are semantically the exact same character, even moreso than “a” is to “A”.

Even if the locale uses a Latin alphabet, there may still be mixed character sets and languages in the text your app processes – e.g. someone writing mostly in English but including Japanese names.
numeric.

There are more numeric systems than just the modern Arabic numerals as used in English. e.g. “٤٢” is 42 in Eastern Arabic. What matters is usually their meaning (i.e. numeric value), not their representation, just like the other factors we’ve already covered.

So, incorporating all that, the magic incantation required to correctly compare two human pieces of English text, in Swift, is:

let plain = "cafe カ 42"
let fancy = "Café ｶ ٤٢"

if .orderedSame == plain.compare(fancy,
                                 options: [.caseInsensitive,
                                           .diacriticInsensitive,
                                           .numeric,
                                           .widthInsensitive],
                                 locale: .current) {
    print("Actually equivalent.")
} else {
    print("Not equivalent")
}

Actually equivalent.

Note that the explicit locale argument may be important for some use-cases, for two reasons:

Generally it seems to turn on some – but not all – locale-appropriate options, in addition to any you specify explicitly.

While that may be redundant when you’re explicitly turning them on anyway, it’s possible it will have additional effects that aren’t expressible with the options parameter. You’ll probably want those too, as if they exist they’ll be things like special handling of unusual cases and exceptions to the rules.
Sometimes it turns hidden options off, such as whether to consider superscripts & subscripts equivalent. This might be a reason to not use it sometimes, if you don’t like the end result.

It’s less clear, even just considering English, what the correct default behaviour is regarding superscripts and subscripts, or “baseline sensitivity”. It’s quite conceivable that a user might intend to match a superscript or subscript even though they entered a plain digit, because most people don’t know how to actually type superscripts and subscripts (it’s not easy on most computers, at least not without 3rd party utilities like Rocket, and practically impossible on mobile devices).

And plenty of programs – particularly those not written in Swift, that might not handle Unicode correctly even at the most basic levels – erroneously devolve superscripts and subscripts to plain digits, which ideally wouldn’t prevent subsequent tools from still working with them (e.g. still finding “but1” when looking for “but¹”).

Yet in some contexts the differences very much do matter – e.g. in mathematical notation, x² is very different to x₂.

For reference, here’s a breakdown of the behaviour of some key String / NSString methods (as tested in the en_AU locale), to help you decide what specific incantation you need in a given situation:

Method	Case insensitive (“Hello” vs “hello”)	Diacritic insensitive (“cafe” vs “café”)	Width insensitive (“カ” vs “ｶ”)	Numerals insensitive (“42” vs “٤٢”)	Baseline insensitive (“but1” vs “but¹”)
`==`	❌	❌	❌	❌	❌
`hasPrefix`	❌	❌	❌	❌	❌
`commonPrefix(…, options: .caseInsensitive)`	✅	❌	❌	❌	❌
`commonPrefix(…, options: .diacriticInsensitive)`	❌	✅	❌	❌	❌
`commonPrefix(…, options: .widthInsensitive)`	❌	❌	✅	❌	❌
`commonPrefix(…, options: .numeric)`	❌	❌	❌	❌¹	✅
`localizedCompare`	❌	❌	✅	✅	❌
`localizedCaseInsensitiveCompare`	✅	❌	✅	✅	❌
`localizedStandardCompare`	❌	❌	❌	❌	❌
`localizedStandardRange(of:)`	✅	✅	❌	❌	❌
`compare(…, options: .caseInsensitive)`	✅	❌	❌	❌	❌
`compare(…, options: .caseInsensitive, locale: .current)`	✅	❌	✅	✅	✅
`compare(…, options: .diacriticInsensitive)`	❌	✅	❌	❌	❌
`compare(…, options: .diacriticInsensitive, locale: .current)`	❌	✅	✅	✅	✅
`compare(…, options: .widthInsensitive)`	❌	❌	✅	❌	❌
`compare(…, options: .widthInsensitive, locale: .current)`	❌	❌	✅	✅	✅
`compare(…, options: .numeric)`	❌	❌	❌	✅	✅
`compare(…, options: .numeric, locale: .current)`	❌	❌	✅	✅	❌²
`compare(…, options: [.caseInsensitive, .diacriticInsensitive, .numeric, .widthInsensitive])`	✅	✅	✅	✅	✅
`compare(…, options: [.caseInsensitive, .diacriticInsensitive, .numeric, .widthInsensitive], locale: .current)`	✅	✅	✅	✅	❌

Now, the real challenge is making code that works across all locales. In a nutshell, that’s practically impossible with Swift’s standard libraries today – they just don’t support it. To do it right, you’d have to determine what the appropriate comparison options are for every possible locale, manually, and bundle that database with your app.

But, given it’s usually better anyway to err on the side of matching rather than not matching, you can get pretty far by just assuming insensitivity to the above five factors.

Even in cases where this does cause mistakes – e.g. conflating “Maßen” (in moderation) with “Massen” (en masse) in German – it’s potentially unavoidable without deeper, context-specific knowledge anyway (since “ß” is normally equivalent to “ss” in German, just not regarding those specific two words – you can read more about this unpleasant situation on e.g. Wikipedia).

So, back to prefixes…

Fortunately, compare(_:options:range:locale:) and range(of:options:range:locale:) have a couple of additional options which make them easier to apply to situations other than just comparing whole strings.

Checking for a specific prefix

There is an anchored option which is perfect for this – it restricts the match to the start of the receiving string. e.g.:

let reaction = "😁👍"

if nil != reaction.range(of: "😁",
                         options: [.anchored,
                                   .caseInsensitive,
                                   .diacriticInsensitive,
                                   .numeric,
                                   .widthInsensitive],
                         locale: .current)) {
    print("Happy!")
} else {
    print("Sad¡")
}

Happy!

Note that you must use the range(of:…) variant, not compare(…), because the latter essentially requires that the two strings fully match, not merely that one is a prefix of the other (more on that later, in case you’re not convinced).

Finding common prefixes

Fortunately, there’s a convenience method for exactly this, commonPrefix(with:options:):

let happy = "😁👍"
let party = "😁🎉"

print("Similarities:", happy.commonPrefix(with: party,
                                          options: [.caseInsensitive,
                                                    .diacriticInsensitive,
                                                    .numeric,
                                                    .widthInsensitive]))

Similarities: 😁

Do not use this if you merely want to see if they share a specific prefix, because:

It’s more efficient to just check that directly on each string separately (rather than allocating and returning an intermediary string).
You still have to use compare(…) with the full set of options to check the result.

Note also that it does not have a locale parameter, so you cannot opt in to any system-default options defined for the current locale; you must explicitly specify every option you need.

⚠️ Beware: it doesn’t honour the numeric option.

Working with suffixes

You can of course reverse both strings and then compare what are now their prefixes, but this is expensive and awkward, since the result of String‘s reversed() method is a ReversedCollection, not a String or even a Substring, and it does not have the necessary comparison methods, so you have to convert it to a real String first.

Far easier and more efficient is to make use the backwards option to range(of:…). e.g.:

let word = "doing"

if nil != word.range(of: "ing",
                     options: [.anchored,
                               .backwards,
                               .caseInsensitive,
                               .diacriticInsensitive,
                               .numeric,
                               .widthInsensitive],
                     locale: .current)) {
    print("It's an 'ing' word.")
} else {
    print("Nyet.")
}

It's an 'ing' word.

Note how – conveniently – it does not require reversal of the argument string (“ing” in the above example).

Beware the `range` parameter

The compare(…) and range(of:…) methods also have a range parameter. This seems like a great idea – you can specify which specific subset of a string you care about, without having to actually break it out into a whole new String instance.

However, the range parameter is both a little unintuitive in its behaviour and fundamentally hard to use correctly.

On the first aspect, it’s critical to realise that it specifies the range within only the receiver (“happy” in the example below). It has no effect on the argument string (“party” in the example below). So you might innocently write the following:

let happy = "😁👍"
let party = "😁🎉"

if .orderedSame == happy.compare(party,
                                 options: [.caseInsensitive,
                                           .diacriticInsensitive,
                                           .numeric,
                                           .widthInsensitive],
                                 range: happy.startIndex ..< happy.index(after: happy.startIndex),
                                 locale: .current)) {
    print("Grins all round.")
} else {
    print("…not happy?")
}

…not happy?

If you want to compare subsets of both strings, you need to explicitly slice the second string, e.g.:

let happy = "😁👍"
let party = "😁🎉"

if .orderedSame == happy.compare(party.prefix(1),
                                 options: [.caseInsensitive,
                                           .diacriticInsensitive,
                                           .numeric,
                                           .widthInsensitive],
                                 range: happy.startIndex ..< happy.index(after: happy.startIndex),
                                 locale: .current)) {
    print("Grins all round.")
} else {
    print("…not happy?")
}

Grins all round.

Or, more simply:

let happy = "😁👍"
let party = "😁🎉"

if .orderedSame == happy.prefix(1).compare(party.prefix(1),
                                           options: [.caseInsensitive,
                                                     .diacriticInsensitive,
                                                     .numeric,
                                                     .widthInsensitive],
                                           locale: .current)) {
    print("Grins all round.")
} else {
    print("…not happy?")
}

Grins all round.

But, you should rarely if ever actually do the above, because of the second aspect: slicing strings is actually really hard. Not technically, obviously, but if you want to do it correctly. The crux of the challenge is that two strings can have different lengths but still be equivalent (e.g. “ß” and “ss” in German), so slicing them independently is error-prone, unless you somehow account for the specific differences in their encoding. If you naively assume things like a specific length for a target string (e.g. the single character of “ß”) and apply that length to the input string, you might get incorrect results. e.g.:

let input = "Füssen"
let target = "Füß"

if .orderedSame == input.prefix(target.count).compare(target,
                                                      options: [.caseInsensitive,
                                                                .diacriticInsensitive,
                                                                .numeric,
                                                                .widthInsensitive],
                                                      locale: .current)) {
    print("Something about feet.")
} else {
    print("Nothing afoot.")
}

Nothing afoot.

(in case you don’t speak German, that’s the wrong result logically – Füß is a prefix of Füssen)

Yes, really. I have no idea why commonPrefix(with:options:) doesn’t work correctly with the numeric option, given it presumably uses compare(_:options:range:locale:) or range(of:options:range:locale:) under the hood. Possibly some bad interaction with locale-specific settings, given it doesn’t let you specify the locale and it doesn’t document what it hard-codes it to. ↩︎
Yes, really. I don’t know why using the current locale (en_AU in this case) turns off baseline insensitivity when the numeric option is used, whereas it turns it on otherwise. Seems like a bug in Apple’s framework. ↩︎

Including Services in contextual menus in SwiftUI

Tue, 19 Mar 2024 01:35:39 +0000

SwiftUI provides a way to provide a contextual menu for a view, with contextMenu(menuItems:) and friends, but it requires you to manually specify the entire contents of the contextual menu. That means it does not include the standard Services submenu.

A brief history of Contextual Menus

Contextual menus were introduced [to the Mac] in 1997 with Mac OS 8, with the new Contextual Menu Manager system extension and associated Contextual Menu Modules. See also the Mac OS 8 edition of the HIG (page 93).

Support varied a lot in the early days, though, and some user-interface conventions didn’t solidify until a few years later. e.g. SimpleText didn’t support contextual menus at all in Mac OS 8.0, and while 3rd party apps like BBEdit eventually did, it took longer still for now-standard items to become commonplace, like Cut / Copy / Paste.

Broadly-speaking, contextual menus have changed very little over the decades. At some point the app-specific commands were separated from the system-wide commands, the latter becoming relegated to the current “Services” submenu. And the specific menu items have of course evolved over time, but the basic idea has persisted: to provide app-specific or built-in commands followed by commands proffered by system-wide extensions.

It’s chuckle-worthy to remember that when Mac OS 8, with Contextual Menus, was introduced, Macs still had one-button mice. I don’t recall but assume it was possible to connect a mouse with multiple buttons, to a Mac, but remember that no Macs had USB at this time (that didn’t arrive until the iMac in August 1998, nearly a year later). Prior to USB, Macs used different I/O ports than PCs (ADB vs PS/2, for the most part), so there were a lot fewer mouse (and keyboard) options for Macs. Thus, contextual menus were originally – and to this day still also – invoked by control-clicking. You might have encountered the term “secondary-click[ing]”, which arose in that era to abstract over whether it was a physically distinct mouse button or just a modified left click¹.

It apparently wasn’t until the Mighty Mouse in 2005 that Apple actually shipped a multi-button mouse. (I say apparently because I don’t personally recall, and I had switched to 3rd party mice long before then anyway so I probably didn’t even notice at the time)

🤔 I don’t know why SwiftUI gets the terminology wrong, calling it the “context menu” rather than contextual menu. It’s not just SwiftUI – Apple’s latest Human Interface Guidelines have the same spelling error.

The correct term (in both English and from precedence) is contextual. The menu provides contextual commands (or context-sensitive, if you prefer). It is not a “context” menu. That doesn’t even make sense. It doesn’t provide context, it is contextual.

Are Services still relevant?

I assume that the Services submenu – whether through contextual menus or the app menu – is not used by most Mac users today, if only because the Mac user-base has expanded massively and most people barely leave their web browsers. I’ve heard Services derided or overlooked as a “power-user” or “niche” feature. Which is sad, because they can be very handy.

More importantly, some of your app’s users might rely heavily on Services as part of their personal workflow and choice, and it’s really not our place – as native Mac application developers – to tell them they shouldn’t use a standard system feature.

It’s frustrating for end-users to encounter applications which don’t support standard Mac features, like Services, and makes such applications stand out as non-native or otherwise broken.

It is sad that SwiftUI is in the general company of Electron and its ilk, here.

So how’s it done?

There’s probably at least two ways to do this, one of them being to manually insert the Services menu item into an otherwise vanilla SwiftUI menu. But I quickly ran into non-trivial challenges in pursuing that avenue, as you can’t just ask AppKit for the Services menu item, or its contents. Alas.

Ultimately I found it easier – and more in keeping with the grain – to instead just not use SwiftUI for contextual menus at all. But fortunately that doesn’t mean abandoning SwiftUI entirely, merely intermingling some AppKit into it.

The basic design is the standard AppKit sandwich²: an NSViewRepresentable containing an NSHostingView.

The first part is particularly easy, and just the usual annoying boilerplate:

struct ContextualMenuView<Content: View>: NSViewRepresentable {
    let viewContent: () -> Content
    let textProvider: @MainActor () -> String

    init(@ViewBuilder viewContent: @escaping () -> Content,
         text: @autoclosure @escaping @MainActor () -> String) {
        self.viewContent = viewContent
        self.textProvider = text
    }

    func updateNSView(_ nsView: ContextualMenuViewImplementation,
                      context: NSViewRepresentableContext) {
        nsView.rootView = self.viewContent()
        nsView.textProvider = self.textProvider
    }

    func makeNSView(context: Context) -> ContextualMenuViewImplementation {
        ContextualMenuViewImplementation(rootView: viewContent(),
                                         textProvider: textProvider)
    }

    func sizeThatFits(_ proposal: ProposedViewSize,
                      nsView: ContextualMenuViewImplementation,
                      context: Context) -> CGSize? {
        return nsView.fittingSize
    }
}

I’ve hard-coded it for text in this example, for simplicity, but you can adjust that to suite your needs.

⚠️ The sizeThatFits(…) implementation is a bit arbitrary. SwiftUI’s view sizing methodology is mildly infuriating, in the sense that it’s both very limited in its capabilities and very confusing for what little it does. It took me a lot of trial-and-error and reverse engineering to figure out what value I needed to return. But I suspect it’s context-sensitive, based on what the subview does. So feel free to adjust it as necessary for your own use.

For convenience it’s nice to add a view modifier for this too:

extension View {
    func contextualMenu(for textProvider: @autoclosure @MainActor () -> String) -> ContextualMenuView<Self> {
        ContextualMenuView(viewContent: { self },
                           text: textProvider)
    }
}

And now on to the real guts of all this, the custom NSView subclass that will define the contextual menu. Fortunately, NSView has very straightforward built-in support for contextual menus, so you don’t need to worry about mouse-event handling – you just provide it a non-nil NSMenu and it does the rest.

class ContextualMenuViewImplementation<Content: View>: NSHostingView,
                                                       NSServicesMenuRequestor {
    @MainActor fileprivate var textProvider: @MainActor () -> String

    @MainActor required init(rootView: Content,
                             text: @autoclosure @MainActor () -> String) {
        self.textProvider = text
        super.init(rootView: rootView)
    }
    
    // Mandated by NSHostingView, but not actually necessary for our purposes here.  But feel free to give this a real implementation, if that makes sense for your use and needs.
    @MainActor @objc required dynamic init?(coder aDecoder: NSCoder) {
        fatalError("init(coder:) has not been implemented for ContextualMenuView")
    }

    // As above.
    @MainActor required init(rootView: Content) {
        fatalError("init(rootView:) has not been implemented for ContextualMenuView")
    }

    @objc override func validRequestor(forSendType sendType: NSPasteboard.PasteboardType?,
                                       returnType: NSPasteboard.PasteboardType?) -> Any? {
        guard sendType == .string || sendType == .init("NSStringPboardType") else {
            return super.validRequestor(forSendType: sendType, returnType: returnType)
        }

        return self
    }

    @objc nonisolated func writeSelection(to pboard: NSPasteboard,
                                          types: [NSPasteboard.PasteboardType]) -> Bool {
        guard types.contains(.string) || types.contains(.init("NSStringPboardType")) else {
            return false
        }
        
        let text = if Thread.isMainThread {
            self.textProvider()
        } else {
            DispatchQueue.main.sync {
                self.textProvider()
            }
        }

        pboard.setString(text, forType: .string)
        return true
    }

    @objc override func menu(for event: NSEvent) -> NSMenu? {
        NSApplication.shared.registerServicesMenuSendTypes([.string, .init("NSStringPboardType")],
                                                           returnTypes: [])

        let menu = NSMenu()
        menu.allowsContextMenuPlugIns = true

        // Insert other menu items here.

        return menu
    }
}

☝️ registerServicesMenuSendTypes(_:returnTypes:) is normally called in +initialize, but Swift doesn’t provide any way to do that (it explicitly bans declaring the method on your NSObject subclasses, for no apparent reason – perhaps a limitation of Swift’s Objective-C interoperability).

So, in the example above I’ve called it (every time!) in menu(for:). That works, but it is inefficient – you only need to call it once per app session. If your application has a natural, better place to put it, e.g. during app launch, move it there.

The only real complexity is in the NSServicesMenuRequestor delegate method writeSelection(to:types:). Because it’s declared nonisolated by the NSServicesMenuRequestor protocol, you’re forced to assume it can be called in any isolation context; from any thread. Unfortunately, at runtime it’s sometimes – but not always! – called from the main thread. Swift doesn’t have an elegant way to say “run this synchronously on the main thread / actor” – if you naively call DispatchQueue.main.sync(…) and you’re already on the main queue, it crashes your application! So you must manually check for being on the main thread, and handle that specially. 😒

The other thing you might want to consider, not shown in the simple example above, is whether you want to support Services sending data back to your view. e.g. they might transform the text and return the new text to you, or generate / source text from somewhere else entirely for you. If you wish to support that, you need to populate the returnTypes argument to registerServicesMenuSendTypes(_:returnTypes:) and implement the readSelection(from:) delegate method.

How’s it done better?

The above works – and is the canonical way to do it. It results in the same plain Services submenu as you’ll see throughout Mac apps.

But that menu’s not great. It just dumps everything it a big amorphous list, with no delineation and merely alphabetical ordering.

Whereas if you look the Services submenu in the application menu:

…it is superior in many ways:

There are more items.
There’s a link to System Preferences / Settings to adjust which services are shown.
There’s dividers in appropriate places, with subheadings, to help visually organise everything.
App icons are shown to better visually distinguish each service.
The grouping is somewhat alphabetical but with all services from a given app shown contiguously.

Some of the items shown aren’t actually context-specific – or at least, to no more detail than merely which application is targeted – but they’re intentionally relegated to the bottom, and could be handy anyway. e.g. if you can see any right-clickable area of any window of an application, you can quickly start profiling that application in Instruments.

It’s not hard, per se, to get the better menu. But it’s not documented, not officially supported, and has some broken edge cases. It is how many applications insert the Services menu into their menus (in fact it was the Electron source which helped me figure out how to do this in the first place), so it’s highly unlikely Apple will break it in the foreseeable future. Nonetheless, be warned.

The main edge case / bug that I’ve encountered is that this better Services submenu only works if the window is key. And there’s no way to force the window to be key (all of the AppKit APIs which seem specifically for that purpose flat-out do not work, such as makeKeyWindow). So you have to fall back to the lesser version of the Services menu in those cases. Which is basically whenever the view’s window is not the active window when the right-click occurs (or, whenever that macOS bug hits whereby the window is shown as if it’s the key window but actually isn’t 😤).

Anyway, with all that in mind and apparently not having dissuaded you, here’s the code to go in menu(for:):

menu.allowsContextMenuPlugIns = !(self.window?.isKeyWindow ?? false)

if !menu.allowsContextMenuPlugIns && !(self.window?.makeFirstResponder(self) ?? false) {
    print("Unable to make self the first responder - reverting to built-in Services submenu.")
    menu.allowsContextMenuPlugIns = true
}

// Add all your other items here.

if !menu.allowsContextMenuPlugIns {
    menu.addItem(NSMenuItem.separator())

    let services = NSMenuItem(title: "Services", action: nil, keyEquivalent: "")
    let serviceSubmenu = NSMenu()
    services.submenu = serviceSubmenu
    services.representedObject = textItem.provider
    NSApplication.shared.servicesMenu = serviceSubmenu
    menu.addItem(services)
}

Fortunately – and thanks to the deep elegance of AppKit’s design – that’s it! It otherwise uses all the same machinery as before (like writeSelection(to:returnType:)).

☝️ Making the view into the first responder is necessary to ensure it’s the one that gets called [first] about what data is available to the Services etc. And it’s also in principle the correct thing to do – any view that responds to your interactions should generally be [made] first responder as a result. And it seems to work perfectly in my use-cases.

But, be aware that it could cause issues in some applications, if it interacts poorly with whatever other view(s) lose first responder status as a result. I can only leave that as a warning and potential exercise for each reader, to figure out how to adapt the above to their circumstances as necessary.

(this isn’t a concern unique to this code, by any means, more the general warning for whenever you manually change the first responder)

⚠️ macOS 14 Sonoma has a rendering bug when first opening the Services submenu within a given application session, as shown below. It’s not a big deal insofar as most of the items still work, and dismissing the menu and re-opening it fixes the rendering.

Bonuses

Adding a Copy menu item

Since the contextual menu is empty except for the default Services menu, you’ll probably want to add in some of the other options that are typically found in the contextual menu – Copy being perhaps the most prominent and often-used.

Fortunately, it’s trivial:

@MainActor @objc func doCopy(_ menuItem: NSMenuItem) {
    let pb = NSPasteboard.general
    pb.prepareForNewContents(with: nil)
    pb.setString(self.textProvider(), forType: .string)
}

// Then, in your `menu(for:)` method:
do {
    let copyMenuItem = NSMenuItem(title: "Copy", action: #selector(Self.doCopy(_:)), keyEquivalent: "")
    copyMenuItem.target = self
    copyMenuItem.isEnabled = true
    menu.addItem(copyMenuItem)
}

One obvious missing piece is localisation of the “Copy” label; left as an exercise for the reader.

Adding a Share menu item

It’s similarly simple to add a standard Share menu item:

@MainActor @objc func showSharePopup(_ menuItem: NSMenuItem) {
    let picker = NSSharingServicePicker(items: [self.textProvider()])
    picker.show(relativeTo: self.bounds, of: self, preferredEdge: .maxY)
}

// Then, in your `menu(for:)` method:
do {
    let picker = NSSharingServicePicker(items: ["🐞"])
    let pickerMenuItem = picker.standardShareMenuItem
    pickerMenuItem.target = self
    pickerMenuItem.action = #selector(Self.showSharePopup(_:))
    pickerMenuItem.isEnabled = true
    menu.addItem(pickerMenuItem)
}

This one benefits from using the standard, AppKit-provided menu item, so you don’t need to handle localising its label.

The 🐞 is there for two reasons:

You have to provide NSSharingServicePicker the nominal item(s) to share up front, in order to initialise it (and so it can tailor its display to the content). standardShareMenuItem should actually be a class member variable, not an instance member variable – a design flaw in AppKit.

You don’t want to invoke the textProvider closure unless you really need to (it could be time-consuming to run, so you don’t want to run it unnecessarily nor while the user is trying to navigate the contextual menu lest it cause GUI hiccups). So an equivalent placeholder value – any other string, in this case – is better, and suffices.
I use the ladybug emoji so that it stands out if the value ever somehow gets shown to the user (it’s a bug, get it? 😄).

Adding a Look Up menu item

This one’s a little bit more iffy. If you think you genuinely have need of a Look Up item, consider whether you should be instead making the text selectable and simply utilising the built-in contextual menu support that selectable text views have in SwiftUI.

@MainActor @objc func lookUp(_ menuItem: NSMenuItem) {
    self.showDefinition(for: NSAttributedString(string: self.textProvider()),
                        at: NSPoint(x: CGFloat.infinity, y: CGFloat.infinity))
}

// Then, in your `menu(for:)` method:
do {
    let lookupItem = NSMenuItem(title: submenu == menu ? "Look Up “\(self.textProvider())”" : "Look Up",
                                action: #selector(Self.lookUp(_:)),
                                keyEquivalent: "")
    lookupItem.target = self
    lookupItem.isEnabled = true
    menu.addItem(lookupItem)
}

This doesn’t display perfectly. The use of infinities for the at argument causes it to draw a small yellow box at the top left of the view, layered underneath the contextual menu but usually still visible. That’s a hack for cases where you can’t determine where the text is, or the text being looked up doesn’t exactly match what’s shown in the view.

If you know the actual location of the selected text, you can specify that instead to get the yellow text drawn in that location. But beware: the text from textProvider must exactly match what’s rendered in the view, otherwise the yellow overlaid box – intended to look like a highlight effect – will look weird, because it [re]renders the text based on what textProvider returned. It also might not correctly match the font, in any case.

Thus why I caution against using the reinvention of this particular wheel.

Possibly borrowed from earlier computers that had multi-mouse-button support, such as some Unixes, but I suspect just coincidence. I vaguely recall that they were the philosophical antithesis of Apple w.r.t. mouse buttons, with some *nix GUIs requiring at least three mouse buttons for their basic function. I seem to recall some of them labelling the buttons primary, secondary, and tertiary. ↩︎
I doubt I came up with this metaphor – although it’s pretty obvious in any case – but it’s worth considering if it’s more than just a cute superficial analogy. AppKit forms the meaningful contents of the sandwich, providing its flavour, substance, and value, while SwiftUI serves only as the bread, there mainly just to convey the contents. 🤔 ↩︎

no platform load command found in ‘libxyz.a’, assuming: macOS

Thu, 14 Mar 2024 23:36:16 +0000

This is a linker warning I see frequently since Xcode 15.0. It appears it’s a result of Apple’s new linker, “ld_prime”, which replaced “ld64” that was in use [by Apple] since around 2005 (per Quinn the Eskimo).

☝️ “ld_prime” might be an internal code name, or perhaps is just Quinn’s personal nomenclature. The actual binary and project name is simply ld (not to be confused with the original ld, the old old linker, that Apple used until ld64 replaced it in 2005).

The old linker – “ld64” – was retroactively renamed ld-classic. They emit almost identical version strings, making them easy to confuse at a glance, but for subtly different program and project names:

👾 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld -v
@(#)PROGRAM:ld PROJECT:ld-1053.12
BUILD 15:44:24 Feb  3 2024
configured to support archs: armv6 armv7 armv7s arm64 arm64e arm64_32 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em
will use ld-classic for: armv6 armv7 armv7s arm64_32 i386 armv6m armv7k armv7m armv7em
LTO support using: LLVM version 15.0.0 (static support for 29, runtime is 29)
TAPI support using: Apple TAPI version 15.0.0 (tapi-1500.3.2.2)

👾 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld-classic -v
@(#)PROGRAM:ld-classic  PROJECT:ld64-951.9
BUILD 15:44:42 Feb  3 2024
configured to support archs: armv6 armv7 armv7s arm64 arm64e arm64_32 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em
LTO support using: LLVM version 15.0.0 (static support for 29, runtime is 29)
TAPI support using: Apple TAPI version 15.0.0 (tapi-1500.3.2.2)

Tangentially, I believe the similarity of their version output is entirely intentional, as a lot of programs (particularly within build systems like CMake) make assumptions about the output, for better or worse.

Apple’s new linker appears to be much more pedantic than the old one – it warns about a lot of things that the old one didn’t care about. One of these is missing platform load commands:

/Users/SadPanda/Documents/vmaf/libvmaf/ld:1:1: no platform load command found in 'src/libvmaf.a[62](cpuid.obj)', assuming: macOS

This doesn’t technically break anything – assuming it guessed the platform correctly, which I suspect it just takes as being the host’s platform – but it’s super annoying because it’s emitted for every afflicted object file the linker sees (that’s individual files, even if they’re buried in archive files – e.g. libfoo.a). You can have hundreds or even thousands of these warnings for a single library. Worse, they’re emitted when you link against the library, not just when you build it. And with nested static libraries they can propagate up a build chain endlessly.

If we look at the offending file with otool, we see that indeed it has no platform load command:

👾 otool -vl cpuid.obj
Load command 0
      cmd LC_SEGMENT_64
  cmdsize 152
  segname 
   vmaddr 0x0000000000000000
   vmsize 0x000000000000002d
  fileoff 208
 filesize 45
  maxprot rwx
 initprot rwx
   nsects 1
    flags (none)
Section
  sectname __text
   segname __TEXT
      addr 0x0000000000000000
      size 0x000000000000002d
    offset 208
     align 2^4 (16)
    reloff 0
    nreloc 0
      type S_REGULAR
attributes PURE_INSTRUCTIONS SOME_INSTRUCTIONS
 reserved1 0
 reserved2 0
Load command 1
     cmd LC_SYMTAB
 cmdsize 24
  symoff 256
   nsyms 4
  stroff 320
 strsize 56

For reference, here’s an example of the load command it’s looking for:

Load command 1
      cmd LC_BUILD_VERSION
  cmdsize 24
 platform MACOS
    minos 14.2
      sdk 14.4
   ntools 0

Why is it missing?

There’s likely many reasons for this, but they all come down to the same general reason: something is wrong with one [or more] of the tools in the toolchain that created the object file. All object files should contain this load command, on Apple platforms. Though the linker currently treats it as optional, the warning is likely a hint from Apple that it may become required in future. More importantly, without it the linker cannot know which SDK the file is compatible with, and therefore cannot validate that it’s linking the right things.

In the example case above, the problem tool is nasm. It’s an open-source, non-Apple x86-family assembler. It’s been around for nearly thirty years – its first platform was DOS. Though it supports Apple’s platforms, it is predominately focused on Linux & Windows. So it’s not entirely surprising that it’s causing issues.

It also hasn’t released even a minor patch update since 2022 (with version 2.16.01), so nominally it predates and doesn’t support Xcode 15 (announced & released mid-2023).

However, the absence of the LC_BUILD_VERSION command was in fact a problem even before the new Apple linker in Xcode 15, because of cross-compiling (e.g. targeting the iOS simulator on a Mac). There was even a patch to address the problem, submitted by Byoungchan Lee way back in 2021, which was never accepted (it seems to have been completely ignored – not even a response on the mailing list).

Workarounds?

No good ones, that I can find. Stoically ignoring the warning for now is probably the best option.

It is still possible (in Xcode 15, at least) to switch to using the old linker instead, using the -Wl,-ld_classic flags, although that’s not a long-term solution and may have downsides (e.g. some folks report significantly smaller object files with the new linker).

It’s probably possible to manually insert the missing load command – if your build process is amenable to that – although it’s not apparent to me what tool or command invocation you would use to do that. Possibly ld itself, though its documentation provides no insight into this.

For some projects, you might be able to switch to a different assembler (e.g. Apple’s native as from Clang, or Yasm, etc). Apple’s toolchains don’t have this problem (so far as I’ve seen) but are of course not available on other platforms (though, being based on Clang, you can probably get essentially the same thing direct from Clang). I haven’t tested whether Yasm has the issue.

Copying whole folders in an Xcode Copy Files Build Phase

Mon, 26 Feb 2024 00:00:26 +0000

If you try to copy a regular folder (what Xcode calls a “Group”) into a file list for any Build Phase, Xcode refuses. But it does work if you use a folder reference .

I have been unable to deduce any reason for this.

Module verification must be enabled in order for Swift to use the module

Mon, 05 Feb 2024 00:48:07 +0000

Ugh. This was annoying to figure out.

If you have a framework target in Xcode with a modulemap – e.g. because you’re wrapping a C or C++ library for use in Swift – you must keep the module verifier enabled (the ENABLE_MODULE_VERIFIER build setting) for that framework, otherwise any Swift targets using that framework won’t see the module (attempts to import the module will fail with the compiler claiming the module doesn’t exist).

Clearly the “verifier” does more than just verify the module. Or, perhaps Xcode is being obnoxious and silently ignoring the module if it doesn’t detect an explicit pass from the verifier.

SwiftUI drag & drop does not support file promises

Sun, 04 Feb 2024 19:04:31 +0000

SwiftUI doesn’t offer anything equivalent to NSFilePromiseProvider, i.e. to write data to the drop destination. You have to ditch SwiftUI and use AppKit’s drag & drop APIs instead.

FB13583826.

Is that it?

I know that’s not a very helpful in some sense, but I wasted days trying to figure out how to implement this very basic drag and drop functionality, in SwiftUI. From what I found online, I’m not the only person who’s struggled with this. So hopefully this post saves others from long and frustrating searches.

A big part of the difficulty in answering this simple question – does SwiftUI support this or not – is that a lot of documentation (first & third party) around SwiftUI’s APIs uses the word “promise” but not in the same way. They merely mean that the pasteboard data is populated on first access, not that it actually follows the file promise protocol.

Best alternative (other than ditching SwiftUI)

The closest you can get is to write data into some arbitrary location that you have to choose blindly (not having any idea where the actual destination is), and then rely on the receiving app to move or copy the file from there. It doesn’t matter, in this regard, whether you use the Transferable-based APIs or the NSItemProvider-based APIs.

view.onDrag {
    let provider = NSItemProvider()
    
    provider.registerDataRepresentation(for: UTType.fileURL) {
        do {
            let tmpDir = try FileManager.default.url(for: .itemReplacementDirectory,
                                                     in: .userDomainMask,
                                                     appropriateFor: URL.temporaryDirectory,
                                                     create: true)

            // You'll need to provide the `suggestedFileName` based on the particulars of your use-case.
            let file = tmpDir.appending(component: suggestedFileName)

            try bytes.write(to: file, options: .withoutOverwriting)

            $0(file.dataRepresentation, nil)
        } catch {
            $0(nil, error)
        }

        return nil
    }
    
    return provider
}

This makes it impossible to avoid unnecessary file copies, and requires you to keep the file around permanently, since you have no idea when the receiver is done with it.

e.g. if the Finder is the drop destination, you have no idea what volume the promised file was dropped on, so you don’t know which volume to create it on. If you guess wrong, the Finder is forced to copy the file. Not only does this confuse the user (by showing a copy badge on the mouse pointer) but it wastes time (in performing the copy) and the original file is left in place, requiring manual clean-up (which is impossible to do correctly because you have no idea when the receiving application is done with the temporary file – and the receiver might hold a reference to it permanently).

Understanding the SwiftUI drag & drop APIs

Tangentially, I found most documentation on SwiftUI’s drag & drop APIs – especially Apple’s – to be very poorly written, which is particularly frustrating in this case because the APIs are not intuitive in the slightest.

And most code / usage examples focus exclusively on trivial, disinteresting cases (e.g. dragging images around within a single window).

Only after I’d spend days reverse-engineering the APIs to figure out how they actually work and how they’re supposed to be used, did I stumble upon Dave Rahardja‘s All about Item Providers. It’s by far the best documentation I’ve found on drag & drop in SwiftUI. Note though that Dave also uses the word “promise” a lot even though he’s never actually talking about actual file promises.

Also, I found that the newer and ostensibly better Transferable-based API was harder to understand and harder to use than the NSItemProvider-based APIs. I recommend going straight to, and sticking with, the latter. Your code will be simpler and more reliable.

Bad API example: FileManager’s url(for:in:appropriateFor:create:)

Wed, 31 Jan 2024 20:40:07 +0000

I find FileManager‘s url(for:in:appropriateFor:create:) to be very unintuitive. It seems to have multiple, largely-orthogonal functions. It can provide paths to common folders (albeit badly). It can create temporary folders. It can locate volume-specific bins (Trash folders).

It is an example of bad API design. Specifically, regarding cohesion: the principle that an API should have one purpose. A litmus test for this is whether all the method parameters are always applicable¹.

It wasn’t until I wrote a test driver which explores its entire parameter space, that I was finally able to grok what the hell it’s doing and delineate its multiple modes of operation.

I’ve contrasted it with the results from its sibling urls(for:in:), to better understand what it’s doing (and expose some more of its flaws).

Test driver

Creating files safely in Mac apps

Wed, 31 Jan 2024 02:48:27 +0000

Creating a file is a pretty basic and conceptually simple task, that many applications do (whether they realise it or not – library code often does this too, at least for temporary files such as caches or for communicating between programs).

So you’d think it’d be trivial to do correctly. Alas, it is not.

☝️ This is a challenging topic, and I’ve done my best to research thoroughly and check everything experimentally. Still, it’s certainly possible I’ve made a mistake or overlooked something. Please let me know of any errors, in the comments at the bottom.

Also, it’s a dense topic, so I’ve tried to highlight (in bold) the most important points. In case of TL;DR. 🙂

What is the danger?

There are two key things to watch out for when creating files:

Security flaws due to incorrect use of, or badly designed, file system APIs. This is especially a concern for privileged applications (e.g. setuid) or those that ever run with elevated privileges (e.g. via sudo or by admin users). Unintentional reuse of existing files can be (and has been) the cause of major security vulnerabilities. See the appendix for more details.

Another security concern is leaking sensitive information to other programs (or users, on a shared computer). This can easily happen if files are created in places other programs or users can access, such as shared folders. The files may be inadvertently created with inappropriately broad permissions (e.g. world-readable) or the parent folder’s permissions might permit others to change the permissions of the file after the fact even if they don’t own it (e.g. a world-writable folder without the sticky bit set).
Data loss risks due to inadvertent overwrites or modifications of existing files. If you’re not certain you know what file is already at a given path on disk, and that you should be allowed to overwrite it, then you should not. Usually, the user has to give explicit permission (e.g. Save dialogs that explicitly ask the user if they intend to overwrite an existing file).

This risk is greatest for persistent files, e.g. files in your Documents folder. Those are usually where the user stores their most important data.

For temporary files the level of danger is generally lower but not zero. If you’re using a system-designated temporaries folder, then in principle anything in there is unimportant anyway. However, randomly mucking with temporary files can still cause data corruption or loss, depending on how those files are used by applications (including your own). e.g. they might store autosaves of the current document in temporary files, and directly copy / move those files when the user formally saves. Thus, modifying the temporary file might end up modifying the user’s actual save file.

How are these dangers mitigated?

App Sandboxing helps

As annoying & limiting as App Sandboxing can be in other regards, it is one of the best single steps an application author can take to improve file security. For the most part, your application is the only (non-system & non-privileged¹) application that can write within its sandbox, and you’ll usually be creating files only within your own sandbox.

Avoid /private/tmp (a.k.a. /tmp)

/tmp is merely a symlink to /private/tmp.

/private/tmp is world-readable: every program on the computer can access its contents. Thankfully, it’s not as bad it first appears – /private/tmp is special in that it has the “sticky bit” set, which imposes some key restrictions on what users may do to each other’s files (most importantly, that they can’t delete them even though /tmp is world-writable).

Still, it’s better to use the tmp folder designated via FileManager.default.temporaryDirectory or URL.temporaryDirectory. Though the location and security properties of that folder varies:

If App Sandboxing is in use, it’s an application-specific folder like /Users/SadPanda/Library/Containers/com.sadpanda.MyApp/Data/tmp/. No other (unprivileged) application can access that folder.
If App Sandboxing is not in use, it’s a user-specific folder like /var/folders/v3/8anbad56df64adf3_35gj346jg13v19a/T/ (the exact path varies between user accounts and computers, by design for security). That folder is still insecure – it is accessible to all programs of the same user – but at least you don’t have to worry about other [unprivileged] users.

If used correctly, /private/tmp has essentially the same properties as the user-specific temporary folders. Using it correctly starts with ensuring files are created with no group or ‘other’ privileges, but the full complexities are beyond the scope of this post.

☝️ /private/tmp is not accessible when App Sandboxing is enabled.

Consider setting a restrictive umask

When you create a file on any Linux or Unix system, such as macOS, its permissions are set based on a combination of the specific API used and the process-wide umask. Good APIs will require you to specify the initial privileges of the file, but some do not (e.g. fopen) and instead use some arbitrary default, such as creating files as readable and writable by anyone (0666). That is bad – usually you don’t want your files readable by any other users.

While you should generally avoid APIs that don’t provide proper control over file permissions, you realistically may not even know if you’re using such APIs because they might be employed by library code you don’t control (including Apple’s).

While it’s possible that the parent folder(s) will protect a given file (by preventing access to their contents by other users), it’s safest to not assume that.

The umask can help mitigate the dangers of bad APIs by specifying which privileges are not to be granted by default².

The umask defaults to denying write access to groups and other users (0022), which is a start but still not good – read access to private user data is still a concern.

However – beyond the overly permissive default setting – there at two problems with umask:

It is a process-wide global. Modifications to it apply to all threads in your process, which makes it dangerous to modify. e.g. you might be creating a particularly sensitive file and need the umask to be 0177, so you set it to that, but before you actually get to execute the file creation another thread sets the umask to 0000 because it wants to create an otherwise unrelated file that’s world-writable.

So unfortunately the only safe way to use umask is to set it very early in process launch before any additional threads are created³.

You can try to enforce your own mutual exclusion around umask, e.g. with a global lock, but beware of 3rd party code (including Apple’s) that might modify umask without following your mutual exclusion protocol. Generally umask isn’t modified often, so this arguably is possible to achieve in practice, but proving that there are no missed calls to umask can be practically impossible.
Lots of existing code, that you might be unwittingly using via libraries or frameworks, assumes the umask remains at its default. Thus, making it more restrictive might break things in ways & places that are difficult to foresee.

In general the more complicated your program, that more of a concern this is. e.g. most command-line programs can modify umask without causing issues, but GUI programs pull in a lot of framework code and functionality, some of which might implicitly rely on certain umask bits. Unfortunately the only way to find out is experimentally.

So umask is not a panacea. Still, setting a restrictive umask improves security if you can get away with it.

Correctly use the right file creation APIs

In short, creation of files needs to (generally) not extend nor replace existing files, and ensure correct initial file permissions.

There are quite a lot of APIs for creating a file in macOS. I’m going to enumerate only the most common ones provided by the system libraries & frameworks.

`⚠️` `open`

Nominally this is the low-level API for opening (existing or new) files, although as you’ll see later it’s not actually the only one.

It has a flags parameter, which is how you tell it what to actually do, between opening existing files, creating new ones, etc. Two of the most important flags are O_CREAT and O_EXCL. O_CREAT tells open to create the file. If O_EXCL is specified, open will fail if a file already exists at the target location. If O_EXCL is not specified, O_CREAT is interperted as “create the file if necessary” – meaning, it will actually open the existing file if it exists, and not create a new file.

You should almost never use O_CREAT without O_EXCL. If you really do intend to overwrite the existing file, then it’s safer to remove the existing file first (e.g. with unlink), and then create your new file (with O_EXCL to ensure no other file appears at the target location in the interim). That way you ensure the new file has your expected location (not a symlink) and attributes (e.g. file permissions).

Note that O_EXCL will fail if the target is a symlink, so you don’t need to specify O_NOFOLLOW (although it doesn’t hurt).

open requires you to specify the new file’s permissions if you use the O_CREAT option (and respects the umask), which is good as it makes you think about what the permissions should be, and lets you set them to something suitable for each use case. These permissions are only applied to new files, so in a nutshell you cannot rely on them if you don’t use O_EXCL.

For that reason also, you typically should not use O_TRUNC. If you don’t need the contents of the existing file, delete the file first. “Reusing it” via O_TRUNC also reuses its permissions and other attributes, which might not be set correctly for your intentions (e.g. a malicious program might have pre-created the file as world-readable, even though you intend it to be readable only by the current user and have otherwise done the right things such as set the umask to ensure that).

One reason you might validly use O_TRUNC is if you anticipate there being multiple hard links to the file, and you want to modify the file as seen by the other links too. This is very rare. Be sure that’s necessary before you use O_TRUNC, and consider putting validations in place to ensure the file you’ve just opened for reuse has the expected permissions and attributes (e.g. via fstat64 and similar APIs that operate on the file descriptor – do not use lstat or any other path-based APIs).

⚠️ `fopen`

This is essentially just a wrapper atop open, with the “w” and “a” flags mapping to O_CREAT (essentially) and the “x” flag mapping to O_EXCL. The same rules apply, so you should generally never use “w” without the “x” flag as well, nor “a” without “+”.

fopen does not let you specify the permissions of the created file, instead defaulting to 0666 (readable & writable by everyone) which is a terrible default. It does respect umask, so by default it will create files as 0644 which is marginally better. But, as discussed previously, it is difficult to guarantee what the umask actually is at any particular point in time. So in general you should prefer open instead of fopen.

❌ `OutputStream(toFileAtPath:append:)` & friends

This ultimately (when you call the open method) calls open with the flags O_WRONLY | O_CREAT (plus O_TRUNC if the append argument is false). As such it will always modify an existing file if present.

It also does not let you specify the permissions of the new file, instead defaulting arbitrarily to 0666 (but respecting umask, at least).

It is an unsafe API and should not be used.

⚠️ `NSData.write(toFile:options:)` & friends

This family of methods all ultimately call open_dprotected_np (implementation), a variant of open specific to Apple platforms which adds Apple-specific functionality regarding file encryption and isolation (see e.g. the protection-related flags within NSData.WritingOptions). It takes the same flags as the regular open, and write(toFile:options:) by default uses O_CREAT | O_TRUNC. If you use the withoutOverwriting option, it adds O_EXCL. So you should usually use withoutOverwriting.

If you use the atomic flag, it creates the file using a private function _NSCreateTemporaryFile_Protected which obtains a file path using mktemp ⚠️ and calls open_dprotected_np with the flags O_CREAT | O_EXCL | O_RDWR⁴. Once it has created & written to that temporary file, it uses rename to move it into place. rename just silently deletes any existing file at the destination path. So it’s safe against race attacks, but susceptible to data loss bugs from unintentionally overwriting existing files. As such, use atomic only with caution.

⚠️ If you add withoutOverwriting on top of atomic, the call crashes your program! It throws an Objective-C exception – NSInvalidArgumentException. Swift does not support Objective-C exceptions (it’s fundamentally unsafe to pass Objective-C exceptions up to Swift functions) so you have to use an Objective-C helper function as an intermediary.

This is a particularly unfortunate limitation – even aside from the crashiness – because using both options together is highly desirable and it should in principle work – the implementation can simply use renamex_np instead with the flag RENAME_EXCL.

FB13568491.

It also does not let you specify the permissions of the new file, instead defaulting arbitrarily to 0666 (but respecting umask, at least). For files containing sensitive data (such as private user data), this API should generally not be used.

❌ `NSString.write(toFile:atomically:encoding:)` & friends

These are essentially just wrappers over NSData.write(toFile:options:) & friends, where the atomically argument maps to the atomic option. They provide no way to use the withoutOverwriting option, so they should not be used in most cases.

Use randomised names for transient files

Whenever the file name doesn’t actually matter – i.e. it’s not chosen by the user and isn’t pre-defined by some system requirement – it’s best to use a randomised name. This serves two purposes:

If your code has any bugs that allow it to erroneously overwrite existing files, using random names at least makes it a lot less likely that you’ll trigger those bugs, by greatly reducing the odds of using the same name twice.
It makes it harder (if not impossible) for attackers to predict the file names, and therefore to attack them.

There are many ways to obtain a randomised name, but it’s wise to use one of the canonical methods detailed below.

☝️ If you do care about the file name, but not its location, you can use these APIs to create a randomly-named temporary folder, and then create your file within there.

This can be handy for e.g. preparing a file URL to be dragged from your app, where you want the file to have a proper name but don’t care (per se) where it lives.

⚠️ `mktemp`

mktemp is infamously a source of security vulnerabilities, because it doesn’t actually create the file (or folder) but merely returns a path. The caller is responsible for securely creating the file (or folder). This can be done safely – by following the guidance earlier in this post, in particular around the O_EXCL open flag – but it’s easy to screw up.

Generally it’s preferable to use mkstemp / mkdtemp & friends. Unfortunately, if you want to use higher-level file APIs on Apple’s platforms that’s a problem because most of those APIs don’t support initialisation from a file descriptor, only a path (or equivalently, URL). So you may find that you still need to use mktemp. If so, be very careful about how you actually create the files & folders.

✅ `mkstemp` / `mkdtemp` & friends

These replacements for mktemp actually create the file / folder (respectively), in a safe way.

mkstemp creates the file and returns the corresponding open file descriptor, instead of merely returning a path and leaving it to the caller to get the creation step right. It will never overwrite (nor open) an existing file. It also sets the file’s initial permissions to 0600 (i.e. read-writable but only by the current user), which is a pretty safe default.

mkdtemp still returns a path (not a file descriptor), like mktemp, but it ensures the folder was actually created (and not previously existent) with the permissions set to 0700 (i.e. usable only by the current user).

Neither make any guarantees regarding security – or lack thereof – due to parent folder permissions. The caller still needs to ensure necessary security protections for those (whether by choosing a suitable system-provided folder, or manually checking permissions and symlinks in the path). URL.temporaryDirectory is a good starting point.

Using these from Swift is a little awkward because they mutate their primary argument (the path template), but here’s an example. Once you have the file descriptor (in the mkstemp case) you can wrap it in e.g. a FileHandle and work with it at a slightly higher level.

`⚠️` `FileManager.default.url(for:in:appropriateFor:create:)`

In its special mode where ‘for’ is .itemReplacementDirectory and ‘in’ is .userDomainMask, this behaves like mkdtemp; it creates a randomly-named folder and returns the URL to it (note that it completely ignores the ‘create’ argument in this case – there is no way to have it not create the temporary folder).

This API seems to presume the use of App Sandboxing to mitigate its problems – and when App Sandboxing is enabled, it’s the best way to obtain a temporary file or folder path, though only if you use a suitable value for the ‘appropriateFor’ parameter, such as URL.temporaryDirectory⁵. When App Sandboxing is enabled, that will result in a path inside the sandbox, which is the most secure place an unprivileged application can use.

However, if instead a URL is provided which points to a different volume, it returns a path to a user-specific temporary folder on that volume, e.g. /Volumes/Example/.TemporaryItems/folders.501/TemporaryItems/. While that does exclude (unprivileged) other users, it’s still a big step down from a location inside the app’s sandbox.

Appendix: File system races in more detail

Generally-speaking, the file system is a shared resource. Multiple programs can access it simultaneously with no coordination required⁶ between them. That opens the door for races – where the state of the world changes in-between file system operations that a program might mistakenly assume are atomic.

In general, any single call to a low-level file system API – e.g. open – is atomic. Most such APIs correspond to a single syscall into the kernel, and the operation inside the kernel is wrapped inside a lock (conceptually if not also literally).

Conversely, any operation that takes multiple calls to a file system API is never atomic.

A textbook security vulnerability arises when you do something like:

Make up some random file name (e.g. with mktemp).
Check that it doesn’t exist (one syscall), and see that it doesn’t.
Write to that file (a separate syscall).

A malicious program could inject its own file in-between steps two and three – or even more dangerously, a symlink – and cause your program to overwrite something (many file APIs will automatically follow symlinks and open existing files, if not used correctly as detailed in this post).

The classic concern in this regard is with privileged programs that have the ability to overwrite sensitive files, e.g. /etc/passwd. Tricking them into doing so can cause major damage to the system (e.g. nobody can login anymore!) in the best case, and in the worst case – where the attacker can also influence the contents of the file, or those contents are conveniently just what the attacker wants – they might be able to implement a more subtle attack that doesn’t merely break the system but instead e.g. changes the root password, giving them superuser access to the computer.

Even for unprivileged applications, it can still be a concern. e.g. they might be tricked into writing a bunch of private user data into a shared location from where the attacker can exfiltrate it.

Appendix: File writing APIs that cannot create new files

These APIs are nominally irrelevant since they can’t be used to create new files, but it can be useful to know that fact, for use in converse scenarios where you do not want to create a file.

`FileHandle(forWritingAtPath:)`

…ultimately calls [[NSConcreteFileHandle alloc] initWithPath:… flags:0x1 createMode:0 error:nil], which turns the path string into a URL using -[NSURL fileURLWithPath:] and calls -[NSConcreteFileHandle initWithURL:flags:createMode:error:], which calls _NSOpenFileDescriptor to do the actual file system calls. That calls open with only the flag O_WRONLY; it does not pass O_CREAT nor O_EXCL. So FileHandle cannot create new files (which I find a bit unintuitive, as nothing in the name really suggests that limitation).

`NSData(contentsOfFile:options:)` & friends

…ultimately call open with no flags (meaning they can only read existing files, not even modify them). So again, cannot create new files. Which is perhaps implied and obvious from the name, but it’s good to be certain.

In principle system programs & privileged (e.g. root & admin) programs should be defended against too, but it’s often impractically difficult to do so, and beyond the scope of this post to try to explain how.

Root is of course the most impractical to defend against – even though the root user is not a traditional “God” user on macOS, Apple’s nerfing of root is designed to protect Apple’s programs, not yours. Root (and admin users) can ultimately still access any files your application(s) create and there’s nothing you can do about it (other than potentially through orthogonal protections, such as encryption). ↩︎
It is of course possible for libraries to override the umask, but that would be particularly foul of them and none that I’ve surveyed do that, thankfully. ↩︎
This can be hard to guarantee, in a non-trivial program. You can use an assertion or precondition on the return value of pthread_is_threaded_np to help ensure you’re modifying umask before additional threads are created. ↩︎
It’s not apparent to me why it needs read access to the file as well – that appears to be a bug. ↩︎
I’m not sure what happens if the App Sandbox container is not on the boot volume. ↩︎
There are various mechanism for voluntary coordination, such as NSFileCoordinator and flock, but programs are not required to use them (and malicious programs happily won’t). ↩︎

NSImage is dangerous

Tue, 23 Jan 2024 21:53:57 +0000

NSImage is formally documented as largely not thread-safe:

The following classes and functions are generally not thread-safe. In most cases, you can use these classes from any thread as long as you use them from only one thread at a time. Check the class documentation for additional details.
Apple’s Threading Programming Guide > Appendix A: Thread Safety Summary, subsection Application Kit Framework Thread Safety

What “in most cases” means is left to the reader’s imagination. Apple adds a little addendum for NSImage specifically:

One thread can create an NSImage object, draw to the image buffer, and pass it off to the main thread for drawing. The underlying image cache is shared among all threads. For more information about images and how caching works, see Cocoa Drawing Guide.
NSImage Restrictions

For a start, it’s talking only about creating an NSImage from scratch, not loading it from serialised form (e.g. a file, a pasteboard, etc). It doesn’t even deign to mention those other, much more common cases.

And even for that one mentioned use case, what does it mean, exactly? What “image cache” is it referring to?

I don’t have authoritative answers. The documentation is so infuriatingly vague that Apple could do basically anything to the implementation, between macOS updates, and claim to have broken no promises.

What I do have is some empirical data and the results of some reverse engineering (shout out to Hopper), from macOS 14.2 Sonoma.

NSImage 101

NSImages can represent a wide range of imagery. Most uses of them are probably for bitmap data (i.e. what you find in common image formats like WebP & AVIF), but NSImage also supports ‘raw’ images (e.g. Nikon NEF) as well as vector data (e.g. SVG & PDF). It also has a plug-in mechanism of sorts, so the supported image formats can be extended dynamically at runtime.

You can fetch the full list of supported types from NSImage.imageTypes – although it doesn’t distinguish between those it can read vs those it can write. Fortunately, sips --formats in Terminal gives you the same list with additional metadata. On my machine that list happens to be:

Supported Formats:
-------------------------------------------
com.adobe.pdf                pdf   Writable
com.adobe.photoshop-image    psd   Writable
com.adobe.raw-image          dng   
com.apple.atx                --    Writable
com.apple.icns               icns  Writable
com.apple.pict               pict  
com.canon.cr2-raw-image      cr2   
com.canon.cr3-raw-image      cr3   
com.canon.crw-raw-image      crw   
com.canon.tif-raw-image      tif   
com.compuserve.gif           gif   Writable
com.dxo.raw-image            dxo   
com.epson.raw-image          erf   
com.fuji.raw-image           raf   
com.hasselblad.3fr-raw-image 3fr   
com.hasselblad.fff-raw-image fff   
com.ilm.openexr-image        exr   Writable
com.kodak.raw-image          dcr   
com.konicaminolta.raw-image  mrw   
com.leafamerica.raw-image    mos   
com.leica.raw-image          raw   
com.leica.rwl-raw-image      rwl   
com.microsoft.bmp            bmp   Writable
com.microsoft.cur            --    
com.microsoft.dds            dds   Writable
com.microsoft.ico            ico   Writable
com.nikon.nrw-raw-image      nrw   
com.nikon.raw-image          nef   
com.olympus.or-raw-image     orf   
com.olympus.raw-image        orf   
com.olympus.sr-raw-image     orf   
com.panasonic.raw-image      raw   
com.panasonic.rw2-raw-image  rw2   
com.pentax.raw-image         pef   
com.phaseone.raw-image       iiq   
com.samsung.raw-image        srw   
com.sgi.sgi-image            sgi   
com.sony.arw-raw-image       arw   
com.sony.raw-image           srf   
com.sony.sr2-raw-image       sr2   
com.truevision.tga-image     tga   Writable
org.khronos.astc             astc  Writable
org.khronos.ktx              ktx   Writable
org.khronos.ktx2             --    Writable
org.webmproject.webp         webp  
public.avci                  avci  
public.avif                  avif  
public.avis                  --    
public.heic                  heic  Writable
public.heics                 heics Writable
public.heif                  heif  
public.jpeg                  jpeg  Writable
public.jpeg-2000             jp2   Writable
public.jpeg-xl               jxl   
public.mpo-image             mpo   
public.pbm                   pbm   Writable
public.png                   png   Writable
public.pvr                   pvr   Writable
public.radiance              pic   
public.svg-image             svg   
public.tiff                  tiff  Writable

Sidenote: notice how it supports modern formats like WebP, AVIF, and JPEG-XL, but only for reading, not writing. 😕

It used to be that NSImage was pretty much a one-stop-shop for typical app image I/O needs, but Apple have for some reason crippled it over the years. I feel like this is part of a larger trend of Apple providing increasingly less functionality, more complexity, and less coherence in their frameworks.

An NSImage can have multiple ‘representations‘. These are essentially just different versions of the image. One representation might be the original vector form (e.g. an NSPDFImageRep). Another might be a rasterisation of that at a certain resolution. Yet another might be a rasterisation at a different resolution.

It gets more complicated, however, because some representations are merely proxies for other frameworks’ representations, like CGImage, CIImage or somesuch.

Nonetheless, for the simple case of a static bitmap image read from a file, generally NSImage produces just one representation, which is the full bitmap (as an NSBitmapImageRep). That’s the only case I’m going to discuss here, for simplicity’s sake (though likely the lessons apply to the other cases too).

Accessing / using an NSImage

Generally to use an NSImage you need a bitmap representation. e.g. to actually draw it to the screen.

Unless you manually create an NSImage from an in-memory bitmap, the bitmap representation is not loaded initially, but rather only when first needed.

☝️ You can usually obtain the NSBitmapImageRep (via e.g. bestRepresentation(for: .infinity, context: nil, hints: nil)) even before it’s actually been loaded – initially it’s largely just a shell, that knows its metadata (e.g. dimensions and colour space) but not yet its actual imagery.

Ultimately the load is triggered when something asks for the raw bytes of the bitmap (either you, directly in your code, or indirectly when you e.g. ask AppKit to draw the image).

Fetching the raw bytes of a bitmap from NSBitmapImageRep is not a trivial exercise. I’m going to gloss over the complexities (like planar data formats) and just talk about the common case of single-plane (“interleaved channels”) bitmaps.

For those, you can access the raw bytes via the bitmapData property.

Nominally, bitmapData just returns a uint8_t*.

In fact, bitmapData calls a lot of internal methods in a complicated fashion, with many possible code paths. What exactly it does depends on the underlying source of data, but in a nutshell it checks if the desired data has already been created / loaded (it’s cached in an instance variable) and if not it loads it, e.g. by calling CGImageSourceCreateImageAtIndex.

That’s what makes NSImage dangerous to use from multiple threads simultaneously.

There is no mutual exclusion protecting any of these code paths.

Reading the cached bitmap data is essentially read-only (just some retain/autorelease traffic) so it’s safe to call from multiple threads concurrently (as long as something ensures the NSBitmapImageRep is kept alive the whole time and not mutated)

But loading or modifying it is not. As such, if you call bitmapData from multiple threads concurrently, and you don’t know for sure that it’s already fully loaded, you get a data race (also known as a “WTF does my app crash randomly?!” condition).

The consequences of that race vary. Maybe you “win” the race – one thread happens to run virtually to completion of bitmapData first, storing the fresh backing data into the caching instance member, and then all the other threads run and just return that same value – the ideal situation as everything works as intended.

Maybe you “lose” the race: every concurrent thread checks simultaneously and sees there’s no cached value, so they all – in parallel and redundant to each other – load the bitmap data and store it into the cache. They each return the one they created, even though ultimately only one thread wins – the last one to write into the cache – and the duplicate bitmap data that all the earlier threads created is deallocated. Even though pointers to them have been returned to you. And you might be in the middle of using them. Causing you to crash with a memory protection fault (or worse, read from some random other memory allocation that happened to be placed at the same address afterwards, reading essentially garbage).

Hypocritically, this is because Apple don’t follow their own rules on escaping pointers:

The pointer you pass to the function is only guaranteed to be valid for the duration of the function call. Do not persist the pointer and access it after the function has returned.
Apple’s Swift Standard Library > Manual Memory Management > Calling Functions With Pointer Parameters, subsection Pass a Constant Pointer as a Parameter.

The source code for the relevant NSBitmapImageRep methods is essentially:

- (uint8_t*)bitmapData {
    uint8_t **result = nil;
    [self getBitmapDataPlanes:&result];
    return *result;
}

- (void)getBitmapDataPlanes:(uint8_t***)output {
    [self _performBlockUsingBackingMutableData:^void (uint8_t* dataPlanes[5]) {
        *output = dataPlanes;
    }];
}

🤦‍♂️

The bitmapData property and getBitmapDataPlanes methods are fundamentally unsafe.

Unfortunately, while _performBlockUsingBackingMutableData and its many similar siblings can be made safe, they are (a) all private and (b) not currently enforcing the necessary mutual exclusion. This could be corrected by Apple in future (although I wouldn’t hold your breath).

Nominally the colorAt(x:y:) method could also be safe, but it currently lacks the same underlying mutual exclusion – and even if it didn’t, the performance when using it is atrocious due to the Objective-C method call overhead. Even utilising IMP caching, the performance is still terrible compared to directly accessing the bitmap byte buffer.

Why does it matter that NSImage is not thread-safe?

The crux of the problem is that you often have no good choice about it, because NSImage is a currency type used widely throughout Apple’s own frameworks, and you often have no control over what thread it’s created or used on.

For example, even using the very latest Apple APIs such as SwiftUI’s dropDestination(for:action:isTargeted:), you cannot control which thread the NSImages are created on (it appears to be the main thread, although that API provides no guarantees).

Similarly you have no control over where those images are used if you pass them to 3rd party code – including Apple’s. e.g. Image(nsImage:); possibly that only uses them on the main thread, but it might be pre-rendered the image a separate thread for better performance (to avoid blocking the main thread and causing the app to hang). In fact it should, in principle.

Loading an image – actually reading it from a file or URL and decompressing it into a raw bitmap suitable for drawing to the screen – should never be done on the main thread, because it can take a long time. A 1 GiB TIFF takes nearly 30 seconds on my iMac Pro, for example (and TIFF uses very lightweight compression, so it’s a relatively fast-to-read format). Anything involving the network could take an unbounded amount of time. Even small files – like a 20 MiB NEF – can take seconds to render because they are non-trivial to decode and/or decompress.

So you’re screwed on multiple levels: not only can you generally not guarantee what thread NSImage is born on nor used on, you can’t use it exclusively on the main thread because that will cause a terrible user experience.

What can you do?

In simple terms, the best you can realistically do is try ensure that all modifications to the NSImage (including implicit ones, such as loading a bitmap representation upon first use) happen exclusively in one thread [at a time]. For example, in my experiments, putting a lock around otherwise concurrent calls to bitmapData is enough to prevent any data races. Although I am mystified as to how I can still let the main thread draw the image concurrently without any apparent problems. 🤔

If you want to play it really safe, you have to create & pre-load each NSImage on a single thread (not the main thread), before ever sending it to another isolation domain. That means manually reimplementing things like drag and drop of images (because you can no longer work directly with NSImage with any drag & drop APIs, you have to instead use only – and all – the things that could potentially be images, like URLs or data blobs, and then translate those to & from NSImages manually).

Complicating all this is that NSImage is unclear about how its caching works. For starters, does this caching apply to the representations or is it a hidden, orthogonal system? Is it thread-safe? Etc.

You can supposedly modify the caching behaviour via the cacheMode property, but in my experience there is no apparent effect no matter what it is set it to (not on the image’s representations nor on render performance in the GUI).

It’s a shame that NSImage has been so neglected, and has so many glaring problems. Over the years Apple have seemingly tried to replace it, introducing new image types over and over again, but all that’s done is made everything more complicated.

Reminder: macOS system frameworks binaries are hidden (since Big Sur)

Tue, 23 Jan 2024 18:59:12 +0000

Every now and again I’ll go to do something really innocuous with an Apple framework, like disassemble it in Hopper or check the link headers. And every. single. time. I forget that Apple did some really weird shit in Big Sur, and removed the binaries.

$ ls -lh /System/Library/Frameworks/AppKit.framework/Versions/Current/AppKit
ls: /System/Library/Frameworks/AppKit.framework/Versions/Current/AppKit: No such file or directory

WTF, mate?

Invariably I spend half an hour websearching around to try to figure out how the hell my system got so broken, and how it’s possible to even boot macOS in such a corrupt state, until finally I chance upon Michael Tsai’s excellent summary of how Apple broke their frameworks starting in Big Sur.

The good news for Hopper is that it has since been updated to work around this – you can access the Apple framework binaries through File > Read File from DYLD Cache… There’s also tools like dyld-shared-cache-extractor which can resurrect the binaries from the cache.

Note also that in Sonoma, at least, the cache lives at /System/Volumes/Preboot/Cryptexes/OS/System/Library/dyld/ (in previous macOS releases it was apparently in /System/Library/dyld/).

SwiftData pitfalls

Wed, 15 Nov 2023 21:04:36 +0000

I’ve been exploring SwiftData lately, and I’ve been unpleasantly surprised by how many sharp edges it has. I’m going to try to describe some of them here, so that hopefully others can avoid them (or perhaps be dissuaded from using SwiftData to begin with).

I’m using Xcode 15.0.1 (Swift 5.9) on macOS 14.1 (Sonoma).

Background

I’ve dabbled in Core Data over the years – mostly pre-Swift – and have a kind of begrudging, distant respect. I understand what it’s trying to do and I can appreciate some of the ways in which it makes that easier, but I’m also all too familiar with its limitations and caveats. So mostly I’ve ignored it, preferring to use other approaches – whether that be JSON or plists for simple cases, or just directly using a SQL database when that feature-set genuinely is warranted.

When SwiftData was introduced, I was intrigued. On the surface it seemed like it greatly reduced the boilerplate and some of the complexity of using Core Data. It at least seemed like an appealing option for light use, with relatively simple models and especially with SwiftUI. It seems pretty obviously targeted at the typical, pretty trivial patterns that most iOS apps have – i.e. not much more than lists of objects.

But it’s only recently that I actually tried using SwiftData for more than just some trivial toy cases. I had a textbook case of a hierarchical type tree, with fairly simple 1-to-1 or many-to-1 relationships, the need for cascade deletes in some cases but not others, and a dataset a little larger than what I could get away with just stuffing into User Defaults.

Alas, what I’ve found is that SwiftData has so many glaring bugs and limitations, that even for this textbook example case, it just doesn’t make any sense to use it.

Example use case

My model is fairly simple – a root object called House which can contain multiple Floors each of which has multiple Rooms. Each of those has a few other attributes – simple strings and numbers – and the Rooms can reference a few other model types, like Weapons and Monsters, as 1-to-1 relationships.

The order of floors and rooms matters – they correspond to top to bottom (attic to basement) and left to right as will be rendered in the GUI.

Design flaws

Auto-save doesn’t work

By default, model containers are supposed to auto-save. That’s what the documentation says, both in the API reference and the tutorials:

In both instances, the returned context periodically checks whether it contains unsaved changes, and if so, implicitly saves those changes on your behalf. For contexts you create manually, set the autosaveEnabled property to true to get the same behavior [sic].
Apple’s Save models for later use (subsection of Preserving your app’s model data across launches)

I suppose it hinges on the word “periodically”. In my testing, that period must be tens of seconds, or worse. It’s trivial to see that it doesn’t work – just make some changes and close the window, or quit your app. Virtually all the time – even if you wait several seconds – those changes are silently lost!

It’s clear that SwiftData has no hooks into deinitialisation, view / window closure, app exit, etc. You would think it has to, in order to save things reliably even in the happy-path cases (as opposed to e.g. app crashes). But it simply does not, and thus it does not.

Workarounds

If you want full resilience to unexpected app termination (e.g. crashes) then any time you make a change to any SwiftData-managed object, you have to immediately call context.save() manually. Which is especially annoying since that method throws, so you have to figure out some sane way to handle that failure possibility in every situation in which you make any changes to your models.

For non-trivial applications, this leads to your code being riddled with save-management code. Think about every SwiftUI field which reflects an element of your model and enables mutation of it. Every button that adds or removes or reorders anything. Every editable list view. Etc.

It’s easy to miss a spot where you make a modification, leading to silent data loss.

If you’re willing to lose data when your app crashes (which you really shouldn’t be willing to do so easily do), you can limit your saves to certain key paths – e.g. window closure, app exit, etc. But you have to find and hook into all those things manually.

This is nominally no different to some SwiftData alternatives (e.g. NSCoding), although below par compared to User Defaults and most other SQL database frameworks.

Arrays of `@Model` objects are randomly shuffled

Array in Swift is very explicitly an ordered collection, in the sense that the elements within have a specific albeit arbitrary order, based on how they’re inserted. This is the same as arrays / vectors / lists in practically every programming language.

SwiftData violates this by randomly reordering elements at various times, such as when [re]loading model objects from persist storage.

The reason is pretty simple – it fails to record the order of elements in its underlying SQLite database, instead using a randomly-assigned, arbitrary integer as a uniquing key, and nothing else.

This is a pretty startling design flaw. I’m not sure how SwiftData got out the door with this.

Workarounds

I haven’t found any good workarounds. The advice you’ll generally see online (e.g. on StackOverflow) is to manually load the objects in the array, specifying a sort order as part of the query. But this basically has you manually reimplementing the relational aspects of SwiftData:

You have to add a member variable to the target class in question just to store its index in the array. You need one of these variables for every many-to relationship it can be a part of.
You have to then somehow populate those variables and keep them in sync with any changes to the arrays.
You can’t actually use the array member variables – since the objects are in random order – and instead have to manually query the database separately. Or, you can sort the array member variables at some point, but it’s difficult to do that efficiently (you can’t really be sure when SwiftData is going to mangle the sort order, so you basically have to do the sorting on the fly every single time you access the array).

Rather than going through all that hassle, it’s easier to just not use SwiftData – e.g. you can simply use a normal object graph that’s persisted to disk as JSON, or in property list form, or using NSCoding, etc.

`let` cannot be used for bidirectional relationships

All member variables (of @Model classes) that refer bidirectionally to other model classes must be variables (var), not constants (let). Irrespective of their actual intended semantics.

This is seemingly just an inherent part of SwiftData (and Core Data underneath) – the presumption that all relationships are optional and can change arbitrarily. It worked okay in Objective-C because that was closer to the reality of Objective-C objects, but it flies in the face of Swift’s much stronger controls and expectations on mutability and optionality.

It would be a relatively minor annoyance – losing compile-time protection of immutability is frustrating but not technically a show-stopper – except that there are no warnings or indications otherwise, of any kind, that you’re “holding it wrong”. If you declare the member with let, everything will compile without so much as a warning, but you’ll run into obtuse runtime errors and crashes, e.g.:

💣 Could not cast value of type 'Swift.KeyPath>' (0x12e00cc98) to 'Swift.ReferenceWritableKeyPath>' (0x13e0b4198).

What that error message is trying but failing to convey is that you cannot (outside of initialisation) write to a let, of course, so its attempt to modify the property failed. So it crashed your app.

Workarounds

None, really. You just have to remember, every time, to avoid let. Possibly you could obtain & configure some linter to check this for you.

Relationships are secretly implicitly unwrapped by default

@Model
final class Room {
    let monster: Monster

    …
}

That’s fine, right – every room has a Monster. Simple.

Except it’s not, if Monster is a @Model. In that case, despite the fact you’ve declared it as non-optional, it actually is optional. @Model essentially makes it implicitly unwrapped without telling you (and without the important ! character after the type to warn you).

The technical reason it’s implicitly and unavoidably optional is because Room and Monster are stored in separate SQLite tables, with a nullable foreign key relationship between them. SwiftData does not support non-nullable foreign keys. So it’s totally valid, as far as the actual database layer and persistent storage are concerned, for the relevant Monster to disappear entirely. But that means your model instance is basically corrupt when you load it, which can have a variety of ill-defined effects such as crashing your app.

You might think this is largely hypothetical, or academic – SwiftData’s not going to erroneously delete your Monster out from under the Room (you assume), and you’re not going to do it yourself, so it’ll never happen, right? Well, maybe. Consider what happens as your app evolves, undergoes refactors, etc. It’s quite possible you unwittingly create a situation in which what was previously valid data is no longer valid to your app.

Or, you might run into a SwiftData design flaw or bug which causes the data to be corrupted on save, such as is detailed in the next section. Even if the root cause is that your data was corrupted at save time, it’s frustrating to have that manifest only through obtuse crashes at some ill-defined time in the future after your app is relaunched.

Workarounds

None, really.

At the very least, you can explicitly make all relationships optional. That way you can code defensively around them, by putting in appropriate unwrapping code and guards. If nothing else you can at the very least crash with a clear failure message (there is no log message at all when SwiftData crashes your app via implicit unwrapping).

You cannot directly initialise bidirectional relationship properties

@Model
final class House {
    @Relationship(deleteRule: .cascade,
                  inverse: \Floor.house)
    let floors: [Floor]

    init(floors: [Floor]) {
        self.floors = floors
    }
}

If you do something like the above – making use of Xcode to generate your initialiser for you, like you normally would for classes – you’ll get the wonderful behaviour that everything works fine until you quit your app and reload it. Then you’ll discover that floors is empty.

There are no error messages, neither at compile time nor runtime. No warnings. Nothing.

Looking at the underlying SQLite database, its apparent that something is seriously broken because while it does save the Floor instances, their foreign key back to House is NULL (even if logically that should be impossible – refer back to the prior point on how SwiftData forces all relationships to be optional).

What’s happening is that the initialisation of floors in init is in some sense bypassing SwiftData’s custom hooks; it’s “directly” setting the value in memory without SwiftData updating its corresponding Core Data model underneath (this is another place SwiftData’s abstraction is leaky – your @Model classes aren’t the real model classes, they’re merely a layer atop the actual Core Data models).

Workarounds

Thankfully this does have a simple workaround (iff you know it and can remember to always use it): never assign the relationship in init. You can append to it, and you can assign it outside of init. So e.g.:

@Model
final class House {
    @Relationship(deleteRule: .cascade,
                  inverse: \Floor.house)
    let floors: [Floor] = []

    init(floors: [Floor]) {
        self.floors.append(contentsOf: floors)
    }
}

Relationship constraints are only enforced at save time

@Relationship lets you specify the arity of a relationship – the minimum and maximum number of related objects that are permitted. However, this is not actually enforced at any time except save time (when you explicitly call context.save()).

Arguably this is better than nothing – vanilla Array member variables provide no way to constrain the array’s contents – but it can lead to mistakes if you unwittingly rely on more rigorous enforcement.

Workarounds

None, really, that I’ve been able to come up with.

It sort of helps to think of these ‘constraints’ as actually just ‘guidelines’; to not actually rely on them being enforced. That may mean you have to do manual validation, that’s in principle redundant, in order to ensure the rules are enforced.

That said, since you need to manually save aggressively anyway (since auto-save doesn’t work), you might be able to conjoin the two workarounds.

Singletons are unsupported

In some use-cases your root object is a singleton – you don’t ever actually want to have multiple instances of it, it’s merely the top-level of your object graph, the object which organises all other objects.

Unfortunately SwiftData doesn’t support this. Every @Model class can have multiple instances.

This can make things pretty awkward when dealing with your singleton, as you cannot do intuitive and simple things like:

struct Map: View {
    @Environment(\.modelContext) private var context
    @Query var house: House

    var body: some View {
        VStack {
            ForEach(house.floors) {
                …
            }
        }
    }
}

Workarounds

Basically you have three choices:

Remove all your singletons, redesigning your model and app to accomodate. This might make some contrived sense if you rationalise it as a way to have multiple game sessions concurrently, or to facilitate concurrent unit testing, or somesuch. But you’ll know in your heart that it’s not real.
Use force-unwrapping (or equivalent), making your app crash or otherwise fail if your singleton expectations are violated by SwiftData.
Silently ignore all but an arbitrary instance of the model class in question.

These are all pretty flawed. And no matter which approach you choose, they introduce some significant boilerplate into your code. e.g.:

struct Map: View {
    @Environment(\.modelContext) private var context
    @Query var houses: [House]

    var body: some View {
        VStack {
            ForEach(house[0].floors) { // Silently ignore other houses.
                …
            }
        }
    }
}

In the above example, which house is chosen is arbitrary and may change between view updates. You can do extra work to pick a house deterministically – e.g. sort by some unique attribute and pick the first – and you probably should if you’re going to use this hack, but you’ll still be operating in a messed-up state.

You cannot create model objects outside of SwiftData contexts

This limitation is explicable and reasonable on the face of it, but does limit your app architecture in ways that can be annoying. e.g. you cannot initialise your views with test models like you normally would:

#Preview {
    MapView(house: House.default())
}

That will compile without any complaints, but when you try to actually view the preview in Xcode you’ll get the dreaded “Cannot preview in this file” error, with absolutely no indication why.

If you happen to try a similar thing in your app initialisation, e.g.:

@main
struct Game: App {
    var body: some Scene {
        WindowGroup {
            Overview(house: House.default())
                .modelContainer(for: [House.self])
        }
    }
}

…then you can at least run your app and will see an error message before it crashes, like:

💣 SwiftData/ModelContainer.swift:144: Fatal error: failed to find a currently active container for Room
Failed to find any currently loaded container for Room) [sic]

So far as Apple error messages go, this is practically perfect – it at least does mention almost the pertinent object (the [SwiftData] container is similar to the SwiftData context) and provide some explanation of the genuine problem.

Workarounds

I haven’t explored this in great detail, so there might be other options, but the best I could promptly come up with is to move initialisation inside your SwiftUI views. Whether that be through explicit user interaction (e.g. a “New Game” GUI in my use case, when there’s no existing House in the database), or in a suitable onAppear handler, etc.

That’s reasonable for actual app execution, but doesn’t immediately help you for SwiftUI previews. However, if you attach a suitable model container to your preview instance:

#Preview {
    MapView()
        .modelContainer(for: [House.self])
}

…then your preview will actually use the real app data. This can actually be handy sometimes, as you can manipulate the state – either in the preview or your actual app execution – which can be handy for debugging view problems (e.g. you’re running your app, playing around, and you notice a rendering error – you can quit your app and its saved state will be used in your Xcode previews, automatically showing you the view in its broken state).

Alternatively, you can go through a more laborious process of [more-]manually creating the SwiftData context, in order to allow you to use a distinct database and to create test objects, e.g.:

#Preview {
    let container = try! ModelContainer(for: House.self,
                                         configurations: ModelConfiguration(isStoredInMemoryOnly: true))

    MapView(house: House.forPreviewing())
        .modelContainer(container)
}

I believe using an in-memory-only store will prevent it touching your actual app’s saved state, but I haven’t poked at it much. I also believe you can use the more complicated configuration initialiser to explicitly specify a URL, if you want a persistent test dataset that’s nonetheless separate from your app’s real dataset, but I haven’t played with it myself.

There are various tutorials online that go into more detail, e.g. Paul Hudson‘s How to use SwiftData in SwiftUI previews.

Apple’s documentation is wrong

This one barely rates a mention since Apple’s documentation is pretty infamously unreliable (what little of it exists at all). Still, it’s important to point out a couple of particularly glaring errors that will hit most SwiftData newbies:

@Model
class Trip {
    var name: String
    var destination: String
    var startDate: Date
    var endDate: Date
    var accommodation: Accommodation?
}

Apple’s Preserving your app’s model data across launches

That is Apple’s very first code example in introducing SwiftData, and it’s wrong. It has no initialiser, therefore you cannot actually use this Trip class.

Now, you might say that’s pedantic, but it’s actually quite pertinent – Apple never show a complete example of a @Model class, so they never show you how you’re supposed to initialise one, and thus they never even try to prevent people falling into pits such as the aforementioned issues with arrays.

Another poignant example:

@Relationship(.cascade) var accommodation: Accommodation?
Apple’s Preserving your app’s model data across launches

That’s not valid; it doesn’t even compile. The syntax actually is:

@Relationship(deleteRule: .cascade) var accommodation: Accommodation?

Workarounds

There are quite a few tutorials available online which provide an alternative onboarding guide, to Apple’s official documentation. I did find a broad perusal of them useful in gleaning bits and pieces that Apple themselves don’t cover (or are wrong about). Unfortunately many are essentially just copy-pastes or paraphrases of Apple’s documentation, and even those that aren’t tend to make the same mistakes.

But, since you’ve now read this post, you’re better equipped than most to try diving into SwiftData.

Non-pitfalls

Just as a bit of a bonus section, I want to call out one aspect of SwiftData which actually works as you might expect.

You only need to register the root(s) of your model graph

When setting up your model container, you don’t have to manually enumerate every model object you use, merely some root(s). SwiftData automatically walks your class(es) member variables to find relationships to other @Model classes, and implicitly registers those too. So e.g.:

@main
struct Game: App {
    var body: some Scene {
        WindowGroup {
            Map()
                .modelContainer(for: [House.self])
        }
    }
}

You don’t need to list Floor.self, Room.self, etc. You can – it doesn’t hurt anything, and might make your app a little more robust against future refactors – but it’s nice that you don’t have to.

Bonus: TablePlus recommendation

It’s unfortunately critical, when working with SwiftData (or Core Data), to be able to view & edit the underlying SQLite database. You can do that out-of-the-box on your Mac using just the built-in sqlite3 CLI, but that’s pretty awkward.

There are many GUI apps for working with SQLite databases. Quite a few are free. Most are functional but clumsy.

Years ago – when I was mostly working with MySQL – I came across TablePlus. It’s a bit pricey at $90 up front – plus $50 per subsequent year if you want to keep getting software updates – but it’s by far the best database client I’ve found, on any platform. It’s easily the most native-feeling on a Mac, even though it’s cross-platform (macOS, iOS / iPadOS, Linux, and Windows!), and one of the fastest. It doesn’t quite have all the features – I do sometimes fire up MySQL Workbench to access its query performance visualisation – but it covers 99% of what I need.

My only notable complaint with it is that it has historically been a tad buggy. Not dramatically, but consistently. That said, I only just got back into it as a result of this SwiftData work – and it’s worked flawlessly so far – so it may well have upped its game since I last properly used it a couple of years ago. 🤞

getifaddrs never specifies broadcast addresses

Thu, 27 Apr 2023 19:25:36 +0000

Apple “Feedback” #12149764.

According to man 3 getifaddrs:

The ifa_dstaddr field references the destination address on a P2P interface, if one exists, otherwise it contains the broadcast address.

In my testing the ifa_dstaddr field is never non-null. I’m not sure I have any suitably configured P2P interfaces, but I definitely have interfaces with broadcast capabilities and getifaddrs reports them as having no broadcast address (even though it correctly notes they support broadcasting, in ifa_flags).

This is on macOS Ventura 13.3.1. I have no idea if this applies to other versions of macOS, or other OS’s.

It appears there’s a workaround, though – the broadcast address is deterministic based on the IP address & netmask, according to Wikipedia, so you don’t technically need getifaddrs to spell it out for you. That’s for AF_INET (IPv4) networks. IPv6 doesn’t really have the same notion of a broadcast address anyway, and for other network types I have no idea if the concept applies or if there’s workarounds for this getifaddr bug.

getifaddrs returns truncated sockaddr_in’s for AF_INET ifa_netmasks

Thu, 27 Apr 2023 18:57:02 +0000

Apple “Feedback” #12149675.

Some netmasks returned by getifaddrs have family of AF_INET yet a length less than sizeof(sockaddr_in), e.g. 5, 6, 7, or 8. On macOS Ventura 13.3.1, at least.

It looks like it’s actually allocating only eight bytes for the ifa_netmask (not the 16 that is the size of sockaddr_in per MacOSX13.3.sdk/usr/include/netinet/in.h), as it clearly has a different sockaddr_in immediately after it, e.g.:

sockaddr
  - sa_len : 5
  - sa_family : 2
  ▿ sa_data : 14 elements
    - .0 : 0
    - .1 : 0
    - .2 : 255
    - .3 : 0
    - .4 : 0
    - .5 : 0
    - .6 : 16 // This is obviously the start of the next sockaddr_in; a correct entry with length of 16.
    - .7 : 2
    - .8 : 0
    - .9 : 0
    - .10 : 127
    - .11 : 0
    - .12 : 0
    - .13 : 1

It’s thus clear that it’s both wrong about the actual length in memory and it is indeed truncating the sockaddr_in structure, although it looks like it’s actually including the full address (the lo0 netmask in the above example is indeed 255.0.0.0).

Here’s another example, where the netmask is 255.255.255.0:

▿ sockaddr
  - sa_len : 7
  - sa_family : 2
  ▿ sa_data : 14 elements
    - .0 : 0
    - .1 : 0
    - .2 : 255
    - .3 : 255
    - .4 : 255
    - .5 : 0
    - .6 : 16 // Again, clearly the start of the next sockaddr_in.
    - .7 : 2
    - .8 : 0
    - .9 : 0
    - .10 : 192
    - .11 : 168
    - .12 : 0
    - .13 : 24

So you can see it’s:

Only allocating up to the address of sockaddr_in, not counting the padding (so sizeof(sockaddr_in), MemoryLayout.size, etc disagree with what getifaddrs is doing), and
Weirdly truncating the stated length (sin_len) to the end of the non-zero bytes in the netmask.

This is undocumented (well, all these data structures are completely undocumented 😔) and is all very messy. getifaddrs should just provide the full sockaddr_in without any of this weirdness, so that users which blindly cast to sockaddr_in (without checking sin_len) don’t risk segmentation faults or reading corrupt data.

I have no idea why it does this only for ifa_netmask, not ifa_addr or ifa_dstaddr. That discrepancy just makes it even more troublesome.

As far as I can tell there are no similar issues with AF_INET6 netmasks, although without knowing under what criteria it’s truncating, I can’t be sure I have all relevant examples covered on the happenstance networks of my laptop.

For these systems, there’s seemingly not much you can do… it seems you have to hard-code the assumption that ifa_netmask contains exactly & only the first eight bytes of the sockaddr_in. You have to completely ignore sin_len (at least if it’s less than eight).

Nikon Z7 very first impressions

Wed, 03 Oct 2018 16:32:33 +0000

This is in the context of coming from a D500 (and a number of DX DSLRs prior to that), and is based only on the first hour or so of using it.

No XQD card included in the U.S.A. This is disappointing, since it appears that every other country on the planet is getting XQD cards included in theirs, to a value of ~$150USD, so it feels a little mean that the U.S.A. is getting screwed. Especially since by all accounts U.S. shipments of the Z7 were delayed by nearly a week compared to most of the rest of the world. It also seems like simply a bad idea on Nikon’s behalf – very few people will have an XQD card already (luckily I have one and only one, from my D500), so Nikon’s running a real risk that a lot of people will open their new shiny only to realise that there’s no memory card they can use in it, and acquiring one is going to be hard (local retailers don’t seem to stock them consistently) and very expensive (XQD cards are currently selling at all-time high prices, despite there being more brands selling them than ever, and more demand than ever, and commodity NAND flash being at its lowest price in a long time… grrr).
Autofocus really struggles in “low light” (e.g. a well-lit restaurant at night), where the D500 would have no problems at all, using the kit 24-70/4 lens. In fact at first I thought the camera was faulty, because I could not for the life of me get it to take a photo, of anything. Eventually I realised it was defaulting, out of the factory, to Focus-priority, and once I switched to Release-priority it started working. But focus was missed most of the time, usually significantly (e.g. headshots had no part of the head in focus most of the time; at best the ears). This was true irrespective of focus mode. In fairness, the D500 is over-confident in its autofocusing abilities – in similar conditions it would also miss focus in many shots, despite claiming it had quickly acquired focus. Note also that “Low Light AF” makes no apparent difference, neither in autofocus speed, ability, nor accuracy.
It’s a very small camera. It has some density to it, so it doesn’t necessarily feel cheap or plastic, but ergonomically it’s not great. The D500 is a much better camera ergonomically (as is the D850, being a very similar design). The Z7 in principle has an interesting advantage which is the ability to do everything through the viewfinder, but the camera is so small and squished that having your face up against it, to look through the viewfinder, makes it very difficult to use any of the buttons or the D-pad. It’s doable, but it’s awkward and I won’t be making a habit of it. The D500 / D850 / etc are actually much more usable when your eye is at the viewfinder, control-wise.
Button placement is a bit weird. The D500 / D850 / etc have a superior layout – and more buttons. My thumb rests over the ‘Disp’ button by default, not the AF-ON where it should, because the camera is so squished that the ‘Disp’ button – relative to the hand grip & other buttons – is basically where AF-ON is on the D500 / D850 / etc. I hope I’ll get used to it, but it is definitely more awkward to hold the Z7 with your hand on its AF-ON button, because your entire hand and fingers are all relatively far to the right edge of the camera, putting a lot more torque on your grip in order to hold the camera flat.
The function buttons on the lens are actually an improvement over the equivalents on Nikon’s DSLRs – they naturally rest under two of my fingers, more or less, making them easier to use.
The mount diameter is way bigger than the old F-mount. Not that it’s intellectually a surprise, but upon first seeing it in person I was irrationally gleeful.
Image quality vs the D500 in low light appears mixed… even by the most optimistic objective measures the D850 (and by extension Z7) are only about 2/3rds of a stop better than the D500 at ISOs 100 and above (the ISO 64 base does push the advantage to one full stop in principle, vs the D500 at ISO 100). However, given the recent, disappointing revelations from DPReview on the nasty banding exhibited by the Z7, my fear is that the D500 will actually turn out to have better image quality in many situations (i.e. anything with significant dynamic range). This is obviously very disappointing for a very expensive, top-of-the-line, brand new camera with an FX vs DX sensor size advantage.
Contrary to some reporting, and some of Nikon’s own misleading product material, 100fps & 120fps 1080p video is only available from a ~DX crop region.
Focus peaking is very difficult to actually get to work. It took me nearly an hour to figure out how – it only appears if (a) you have AF-ON held down, (b) you move the manual focus ring on the lens a significant distance in order to engage MF override, and (c) you have a lot of light and contrast in the scene. In low light, or scenes with low contrast, it simply doesn’t show any peaking, even on the most sensitive setting, and provides no indication why. This is all very unfortunate, as competing focus peaking systems in every other mirrorless camera I’ve ever used all perform much more reliably, easily, and consistently than the Z7’s system does. e.g. the Sony a7R II’s focus peaking was excellent in practice for ensuring correct focus, whereas my tests so far with the Z7, when it bothers to work at all, have shown that it’s not accurate nor clean enough for me to actually get correct focus most of the time. It’s much faster & more reliable to just engage image zoom and focus without peaking. Also, peaking doesn’t work when zoomed in.
The focus ring on the 24-70/4 is awkwardly placed – it’s way too close to the camera body, which is very thin to begin with, so it feels like you’re picking your nose when you operate it. Even with a light lens like the 24-70/4, holding the lens by the focus ring makes the entire thing very front-heavy. The focus ring is also very thin, making it a bit difficult to find and get a good hold on.
Being able to zoom in, in the viewfinder, is awesome. I’ve used this previously on other mirrorless systems and know from that experience that it’ll be immensely valuable in getting focus correct. It also works pretty intuitively – e.g. it zooms in on the selected focus point, naturally – and can be assigned to most (but bizarrely not all) the configurable buttons for easy toggling.
I miss the Nikon rubber eye-cup add-on I applied to my D500. The Z7’s naked viewfinder, while slightly rubbery, is very hard in comparison, and – being – rectangular & flat – doesn’t fit any human face I’ve ever encountered. No different from most cameras, of course – I just hope Nikon release an equivalent eye-cup for the Z7 soon (though I worry, from looking at the viewfinder assembly, that there’s no apparent way to pull it apart, attach anything to it, etc).
On first use the battery jammed in the battery slot, requiring some shaking and application of fingernails to force it out. Very weird – I’ve never encountered this in many years & many Nikon cameras. It hasn’t done it since… yet.
The box it comes in is surprisingly large given it’s a small camera & lens. Much bigger than the equivalent box for the D500, or any of Nikon’s consumer DSLRs.
The fully electronic (“silent”) shutter is very nice. The D500 is a 5 AM garbage truck in comparison – it has always bothered me using the D500 in any even remotely quiet environment.
Viewfinder blackout is so-so. While I’d seen videos on YouTube demonstrating it in various modes etc, in practice I find it’s much more difficult than I expected to track moving subjects when shooting at anything approaching the maximum frame rate (8 FPS). The D500, despite having significant black-out itself vs the D5, is notably superior than the Z7.
SnapBridge is stupidly hard to get to work – mainly in the initial pairing. It took me multiple tries and about an hour overall to get it to finally pair to my iPhone. It requires an extremely precise, pedantic, and rather long sequence of steps in order to get it to pair, and some of those steps are not documented by Nikon. I vaguely recall it being similarly bad with the D500 when I first got it – thankfully it’s a process that only needs doing once per camera body, in principle.

iOS Family Sharing users cannot mix authentication schemes

Sat, 15 Apr 2017 19:51:04 +0000

Apple supports two styles of two-factor authentication, that they call (and distinguish as) “two-step” vs “two-factor”. “Two-step” is their older method, though functionally they’re basically equivalent.

If you have multiple accounts on a Family Sharing arrangement, and some use “two-factor” while others use “two-step”, you’re in for a bag of hurt.

For example, any time you change the password on any of the non-master accounts, you’ll have to reauthorise all devices on that account with the master purchaser. You’ll be prompted, when trying to download apps or purchase anything etc, with a dialog saying “Your Family Organizer, [foo], must enter the security code for their payment method”, asking for some kind of input. There is literally nothing you can enter there that will make it work. Not the password for any of the relevant Apple IDs, not any security codes for any credit cards, nada.

The problem is that it’s asking for a verification code that you can only create on a device which has “two-factor” authentication enabled. Compare for example what you see with “two-factor” authentication enabled on your iDevice:

Versus what you see with “two-step”:

That “Get Verification Code” “button” is what you’re looking for. As you can see, it simply doesn’t exist with “two-step” authentication enabled.

The only solution – to allow your family members to download apps, purchase music / videos / books / etc, or pretty much do anything else on their iDevices – is to force the master account over to “two-factor” authentication.

To do this, you have to go to https://appleid.apple.com/ and turn off “two-step” authentication (which will require you to complete some stupid ‘security’ questions). You cannot turn off “two-step” authentication from any of your actual iDevices’ Settings apps.

Then, stupidly, you can’t actually enable “two-factor” authentication from that same website. That can only be done in the Settings app on one of your iDevices – by (in iOS 10.3 or later) going into Settings ➜ ➜ Password & Security.

There’s no way to enable “two-step” authentication anymore. And not having any form of two-factor authentication enabled is a very bad idea. So if any of your family’s accounts have “two-factor” authentication enabled, you basically have to switch to “two-factor” on all of them.

Which would be broadly fine, if Apple hadn’t made it so needlessly complicated, and the two systems so incompatible that their own software can’t figure out what’s going on.

Rotated Windows

Sun, 26 Feb 2017 08:08:13 +0000

I’d forgotten about this until I stumbled across a reference to it again recently.

This was a little hack I worked on back in 2004, with Mac OS X Tiger (10.4). Yes, kids, macOS was called Mac OS X back in ye Olden Times.

Wow, Slashdot looked even uglier than I remember, back then. Though amusingly my daily reading list hasn’t changed substantially – it still features Slashdot and MacSurfer’s Headline News.

Also… 1024 x 768. That’s just over 5% of the resolution of my current display (27″ Retina iMac). It’s nearly as big as my iPhone 6s’s screen.

Man, do I not miss those shitty old monitors.

I don’t recall what the exact impetus was for the project. I do recall that I was spurred on by Claus Atzenbeck, who was doing some kind of academic work into graphical user interfaces and, IIRC, wanted a way to explore window rotation and general manipulation in a real OS.

Claus’s personal website still exists, all these years later, though alas the link to his relevant research is now broken.

What reminded me of this was finding an attribution to me in a header file that was associated with the project – CoreGraphicsServices.h. This was something I generated (presumably with the help of class-dump or similar) from the CoreGraphicsServices framework, and then partially reverse-engineered (in the sense of figuring out parameter types, function prerequisites, etc). It’s what was necessary to find & use the private APIs for doing window geometry manipulation.

And the only reason my name is on it is because I splatted a 3-clause BSD license into the header file I made, which in hindsight seems highly dubious since the APIs themselves are owned by Apple (insofar as one can ‘own’ APIs, I guess…).

A quick web search reveals a few more mentions:

The aforementioned header is apparently used by Growl.
“BOMGAR”, some kind of remote computer support software, apparently uses the header too.
As does something on marsthemes.com, though at time of writing that website has been largely destroyed for some reason.
This one particularly amuses me – a brief thread on cocoa-dev@ about the header, in which John C. Randolph categorically takes no particular position on the hack. :)

The source & other paraphernalia were originally posted on my La Trobe University student web hosting account, though of course that’s long gone. Here’s the original StuffIt archive, if you’re interested. I don’t actually know if it’s the very latest version – I do still have the project in full – but it’s the latest version I ever published, AFAIR.

I leave it as an exercise to the reader on how to decompress StuffIt files in this day and age. :)

Undocumented Swift conditional compilation macros

Sat, 14 Jan 2017 21:45:55 +0000

swift/lib/Basic/LangOptions.cpp has most of the conditional compilation macros (called “Language Options” in the compiler internally). Notably the swift() version macro is absent, and doesn’t seem to be defined anywhere…

At time of writing the two undocumented additions, to the os(), arch(), and swift() set, are _endian() and _runtime().

I have no idea if they’re useful or not – does anyone have to care about CPU endianness these days, really? – but there they are.

#if DEBUG in Swift

Sat, 14 Jan 2017 21:15:33 +0000

Sigh.

The Swift team give an impeccable impression of a group of people who’ve never actually tried to use Swift.

An incredibly basic compiler task is to provide code a way to distinguish between debug & release builds, in order that it can behave accordingly (e.g. change the default logging verbosity, change asserts from fatal to non-fatal, etc).

Long story short there is no way to do this that works correctly with swift build.

You can make it work with Xcode only by way of a simple workaround – you manually define a custom Swift flag in your target’s settings (here’s one of a bajillion explanations of how do this). You end up with basically identical code to other C-family languages, e.g.:

#if DEBUG
    let logVerbosity = 1
#else
    let logVerbosity = 0
#endif

But there is no way, when using swift build, to specify custom Swift flags in your package config.

You can specify them manually with every single swift build invocation, e.g.:

swift build -c debug -Xswiftc '-DDEBUG'

But now you have extra work and the possibility of screwing it up (e.g. omitting the flag, or mismatching it to your actual build style).

The closest you can get is to use some undocumented, hidden Swift internal library functions:

func _isDebugAssertConfiguration() -> Bool
func _isFastAssertConfiguration() -> Bool

These are defined in swift/stdlib/public/core/AssertCommon.swift.

Only the first is likely to be useful. The second applies in the case where you’re building not just for release but unchecked (-Ounchecked). If you just want to conditionalise on release builds generally, you have to do !_isDebugAssertConfiguration().

The additional problem with this approach is that these are then, in your code, runtime checks. And the compiler then thinks it’s being helpful by pointing out that some of your code that uses them is unreachable.

And of course Swift has absolutely no way to silence compiler warnings, or otherwise tell the compiler not to trust its reachability checks.

Sigh.

Update (February 2018)

While the above method – using the _isDebugAssertConfiguration() method – does still work at time of writing, in Swift 4.1, there are some marginally better ways now available. There’s still not a proper solution, infuriatingly, but with some creativity, as you’ll see momentarily, you can get pretty close to expected functionality.

First alternative

This will only suit some cases, but its relative cleanliness & simplicity makes it appealing when it is an option.

You can use #if targetEnvironment(simulator). This of course only distinguishes between the simulator and a real iDevice environment, though that can often be useful too, perhaps orthogonally to DEBUG vs RELEASE concepts.

Second alternative

Wrap all uses of _isDebugAssertConfiguration() inside a minimal set of functions. All this gets you is a reduced (and fixed) number of unreachable code warnings, though.

For example:

func inDebugBuilds(_ code: () -> Void) {
    if _isDebugAssertConfiguration() {
        code()
    }
}

func inReleaseBuilds(_ code: () -> Void) {
    if !_isDebugAssertConfiguration() {
        code()
    }
}

You can then use these in a fairly streamlined way:

inDebugBuilds {
    print("Hello, I only greet in debug builds!")
}

inReleaseBuilds {
    print("While I only greet in release builds - aloha!")
}

Third alternative

It’s possible to do one better than the above, and eliminate those unreachable code warnings entirely. Plus, doing so actually removes any use of unofficial APIs… but it requires abusing the official API a bit.

The built-in assert() method is implemented atop _isDebugAssertConfiguration() just the same as inDebugBuilds() is, above. However, because it’s part of the Swift standard library, you don’t have to be concerned about its implementation, its use of private / undocumented language, compiler, or library features – that’s up to the Swift compiler team. Most importantly, use of it in your code doesn’t emit an unreachable code warning.

So we can do something like:

func inDebugBuilds(_ code: () -> Void) {
    assert({ code(); return true }())
}

Ugly and obtuse, but functional and reasonably expected to work in all future versions of Swift.

You can similarly create a variant for release-build-only code, though it’s even hackier:

func inReleaseBuilds(_ code: () -> Void) {
    var skip: Bool = false
    assert({ skip = true; return true }())

    if !skip {
        code()
    }
}

Icky, but it works. On the upside, you only have to define this ugliness in one place in your module, and then you can try to forget about its implementation. 😝

One caveat is that I don’t know if the above approach will properly strip out the unreachable code from your built binaries.

Another caveat with these is that since you’re invoking a function, you can’t do a natural if … else … pattern – instead you need explicit, distinct inDebugBuilds & inReleaseBuilds blocks. So it can’t be completely like the vanilla #if DEBUG … #else … that C-family languages have had since before the dinosaurs. You could create versions that take two closures as parameters – one for the affirmative case, one for the other… the trade-off is that your invocations are then a little more verbose and obviously function-cally, e.g.:

func forBuildStyle(debug debugCode: () -> Void,
                    release releaseCode: () -> Void) {
    var debugBuild: Bool = false
    assert({ debugBuild = true; return true }())

    if debugBuild {
        debugCode()
    } else {
        releaseCode()
    }
}

forBuildStyle(
    debug: {
        print("I'm built for debuggin'!")
    },
    release: {
        print("I'm built for the wild!")
    }
)

Up to your personal preferences as to which API you adopt, and which implementation you choose (streamlined but private-Swift-bits-dependent with bonus unnecessary compiler warnings, or official-APIs-only but hacky).

Swift’s String.write(toFile:…) can’t handle tildes

Sat, 10 Dec 2016 06:57:13 +0000

let path = "~/Desktop/sigh.txt"
try "WTF?".write(toFile: path, atomically: true, encoding: .utf8)

Result? Explode:

Error Domain=NSCocoaErrorDomain Code=4 "The folder “sigh.txt” doesn’t exist."
UserInfo={NSFilePath=~/Desktop/sigh.txt,
          NSUserStringVariant=Folder,
          NSUnderlyingError=0x1018110b0 {Error Domain=NSPOSIXErrorDomain Code=2 "No such file or directory"}}

And since there’s no documentation on that write() method, and this is obviously a perfectly reasonable request that can’t sanely yield that bizarre error message, you could be forgiven for having no idea how to fix this.

Long story short, in macOS Sierra at least that write() method happens to be implemented as NSString‘s writeToFile:atomically:encoding:error: method. If you look at the documentation for that method, it states:

path: The file to which to write the receiver. If path contains a tilde (~) character, you must expand it with stringByExpandingTildeInPath before invoking this method.

WTF it can’t actually say that in the Swift method’s documentation (or indeed why it has none at all), or why the exception thrown can’t at least give the real reason, or of course WTF it can’t just do the right thing and handle tildes in the path… I have no idea. This smells like a combination of laziness, snafu, and poor judgement in carrying over bad behaviours and design flaws from Objective-C.

So instead you have to do:

try "WTF?".write(toFile: (path as NSString).expandingTildeInPath, atomically: true, encoding: .utf8)

Sigh.

Building John The Ripper Jumbo for macOS Sierra

Sat, 12 Nov 2016 02:42:40 +0000

It’s quickly apparent that John The Ripper Jumbo doesn’t build out of the box on macOS, and probably hasn’t for a long time, due to its complaint about missing OpenSSL headers.

This guide was almost helpful, except it’s out of date – e.g. the Makefile.in patch it provides no longer applies cleanly – and simply doesn’t work – once you get the John The Ripper configure script to see the OpenSSL devel headers, it then just complains that it can’t find a valid libssl anyway.

Even the Pro version of John The Ripper, which isn’t cheap, doesn’t look like a good option since its web page has the hallmarks of something that hasn’t been updated in many, many years. e.g. talk about support for Mac OS X 10.7 Lion being planned.

And although it appears to support using CommonCrypto instead of the now deprecated OpenSSL, that doesn’t work – even when configured that way it still compiles in code that requires OpenSSL, for SHA1. Sigh.

Trying to get it to use a fresh build of OpenSSL (1.1.0) also seems an intractable failure – OpenSSL 1.1.0 out of the box produces libraries which don’t contain SSL_library_init, which is of course necessary for any OpenSSL user and foils any attempt to use the built libraries by way of missing symbol errors in the link phase.

And OpenSSL 0.9.8zh’s build system is just screwy. By default it builds only 32-bit and only static libraries (no matter how hard you tell it to build shared ones). You have to bypass its first layer of configery and do it ‘manually’, like so:

./configure --prefix= darwin64-x86_64-cc -no-shared enable-camellia

You can then configure JohnTheRipper to point to that version of OpenSSL, like so:

./configure CPPFLAGS='-I /include' LDFLAGS='-L /lib' OPENSSL_LIBS="-lcrypto"

Now it’ll finally get past the OpenSSL issues, and build successfully.

Note: don’t use the Makefile.in patch provided in the aforelinked guide. That actually breaks the build now, even if you properly apply it manually.

Do Nikon teleconverters work on the Sigma 105 macro?

Wed, 29 Oct 2014 12:50:20 +0000

In a word: no. And not just “they’re unsupported”, or that they have optical issues – they physically will not connect. They are deliberately keyed to be incompatible. Nikon teleconverters have a protrusion on their lens mount, which prevents any ‘standard’ Nikon-mount lens from attaching, unless that lens is missing a particular obstruction on the rear mount.

This is not unique to the Sigma 105, by any means. Nikon’s own 70-300, for example, has the exact same obstruction and also will not mount to any Nikon teleconverter (or at least, not any of the current models, of the 1.4x, 1.7x or 2x).

This is very disappointing, and an odd decision by Sigma – it was they, after all, who deliberately included the obstruction on the rear mount. The teleconverters would mount just fine otherwise – there is just enough clearance to the rear element.

In contrast, Nikon’s competing 105 macro doesn’t have the obstruction on the mount, apparently, as it’s listed by Nikon as officially supporting use with Nikon teleconverters. [I have not personally verified this]

I have heard, though haven’t verified first-hand, that the Sigma teleconverters do not have the same protrusion, and so will happily mount to any lens on which they fit (i.e. where their front element doesn’t collide with the rear element of the lens).

iCloud Photo Streams have limits

Wed, 25 Sep 2013 02:01:14 +0000

Officially, Apple claims that you can store your last 30 days of photos in iCloud via Photo Streams. This is true only if you don’t have very many. If you actually take a lot of photos, iCloud boots your arse out and doesn’t let you use Photo Streams anymore. At all.

24/09/13 6:54:08.457 PM PhotoStreamAgent[305]: Transaction forbidden. Code: 403.
24/09/13 6:54:08.457 PM PhotoStreamAgent[305]: MSPublishStreamsProtocol - XXXXX Put connection has failed. Error: __NSCFError:
Domain : streamsProtocolCoreErrorDomain
Code : 3
Desc : The Photo Stream server has rejected the user's request.
UserInfo: {
 NSLocalizedDescription = "The Photo Stream server has rejected the user's request.";
}
24/09/13 6:54:08.457 PM PhotoStreamAgent[305]: MSPublisher - XXXXX Received a quota error.
24/09/13 6:54:08.457 PM PhotoStreamAgent[305]: MSPublisher - XXXXX Quota error code: 4033

Thanks iCloud. Screw you too.

Undocumented – Wade Tregaskis

Recording audio to an iPhone via a Tascam Portacapture X8

Thankfully, any USB-C or USB-C-to-Lightning cable will do

You must use 48kHz sampling

You don’t have to use the iPhone to provide power

You don’t have to record on the Portacapture X8

Backblaze seemingly does not support files greater than 1 TB

Update

Addendum

presentedWindowStyle is not windowStyle

Matching prefixes in Swift strings

So, back to prefixes…

Checking for a specific prefix

Finding common prefixes

Working with suffixes

Beware the range parameter

Including Services in contextual menus in SwiftUI

A brief history of Contextual Menus

Are Services still relevant?

So how’s it done?

How’s it done better?

Bonuses

Adding a Copy menu item

Adding a Share menu item

Adding a Look Up menu item

no platform load command found in ‘libxyz.a’, assuming: macOS

Why is it missing?

Workarounds?

Copying whole folders in an Xcode Copy Files Build Phase

Module verification must be enabled in order for Swift to use the module

SwiftUI drag & drop does not support file promises

Is that it?

Best alternative (other than ditching SwiftUI)

Understanding the SwiftUI drag & drop APIs

Bad API example: FileManager’s url(for:in:appropriateFor:create:)

Creating files safely in Mac apps

What is the danger?

How are these dangers mitigated?

App Sandboxing helps

Avoid /private/tmp (a.k.a. /tmp)

Consider setting a restrictive umask

Correctly use the right file creation APIs

⚠️ open

⚠️ fopen

❌ OutputStream(toFileAtPath:append:) & friends

⚠️ NSData.write(toFile:options:) & friends

❌ NSString.write(toFile:atomically:encoding:) & friends

Use randomised names for transient files

⚠️ mktemp

✅ mkstemp / mkdtemp & friends

⚠️ FileManager.default.url(for:in:appropriateFor:create:)

Appendix: File system races in more detail

Appendix: File writing APIs that cannot create new files

FileHandle(forWritingAtPath:)

NSData(contentsOfFile:options:) & friends

NSImage is dangerous

NSImage 101

Accessing / using an NSImage

Why does it matter that NSImage is not thread-safe?

What can you do?

Reminder: macOS system frameworks binaries are hidden (since Big Sur)

SwiftData pitfalls

Background

Example use case

Design flaws

Auto-save doesn’t work

Workarounds

Arrays of @Model objects are randomly shuffled

Workarounds

let cannot be used for bidirectional relationships

Workarounds

Relationships are secretly implicitly unwrapped by default

Workarounds

You cannot directly initialise bidirectional relationship properties

Workarounds

Relationship constraints are only enforced at save time

Workarounds

Singletons are unsupported

Workarounds

You cannot create model objects outside of SwiftData contexts

Beware the `range` parameter

`⚠️` `open`

⚠️ `fopen`

❌ `OutputStream(toFileAtPath:append:)` & friends

⚠️ `NSData.write(toFile:options:)` & friends

❌ `NSString.write(toFile:atomically:encoding:)` & friends

⚠️ `mktemp`

✅ `mkstemp` / `mkdtemp` & friends

`⚠️` `FileManager.default.url(for:in:appropriateFor:create:)`

`FileHandle(forWritingAtPath:)`

`NSData(contentsOfFile:options:)` & friends

Arrays of `@Model` objects are randomly shuffled

`let` cannot be used for bidirectional relationships