Copy-on-write on APFS

Thu, 16 May 2024 21:14:10 +0000

APFS (like many modern file systems but unlike its predecessor HFS+) supports copy-on-write. This means you can logically copy a file – it looks and behaves like a distinct file – but it doesn’t immediately copy the file’s contents on disk – it merely shares them with the original. Only if and as you modify either version do they start to diverge on disk, with APFS dynamically allocating new storage for the modified parts¹.

This is kind of a sister function to hard links, which similarly avoid copying the file’s contents but where modifications apply to all copies. See also this article on the differences, including versus aliases and symlinks.

Copy-on-write is beneficial for several reasons:

Copies don’t take up any significant space (just whatever tiny amount is necessary for their metadata).
The initial copy operation is practically instantaneous (just a few small metadata writes & updates).
Deferring (if not entirely avoiding) the actual disk I/O reduces wear on the disk.
The copies can share common segments, saving disk space even when they’re not ultimately identical copies.

It does have some potential downsides:

You don’t get the increased data redundancy and error resilience that actual copies provide (although if you’re aiming for data redundancy or backup, you should be using separate physical disks anyway).
It can make subsequent modifications of the file slower, as even just modifying a single byte can trigger the actual copy to be performed.

And some basic limitations:

It’s only supported on APFS (and maybe additional file systems added by 3rd party extensions, but I haven’t tested nor can I find any accounts of this).
It only works within individual volumes (it doesn’t work even between two volumes in the same APFS container, or sharing the same physical disk).

Contrary to what I saw online in a few places, copy-on-write works on all APFS volumes, irrespective of whether they are backed by SSDs, HDDs, or some other type of storage.

Should I use it?

Yes!

For most purposes those downsides aren’t an issue, and the limitations merely mean that it’s wise to have a fallback option (of just copying the actual file contents) whenever copy-on-write isn’t available. And many of the tools & APIs fallback automatically (unless you explicitly require them not to, such as with COPYFILE_CLONE_FORCE to copyfile).

How do I use it?

Method	Uses copy-on-write (where possible)	Does actual copy if copy-on-write isn’t available
cp	❌	N/A
cp -c	✅	❌
ditto	❌	N/A
ditto –clone	❌	N/A
dd	❌	N/A
scp	❌	N/A
Finder Copy then Paste	✅	✅
Finder Duplicate	✅	✅
Finder ⌥-drag	✅	✅
`clonefile`²	✅	❌
`copyfile(…, …, …, COPYFILE_DATA)`	❌	N/A
`copyfile(…, …, …, COPYFILE_CLONE)`	✅	✅
`copyfile(…, …, …, COPYFILE_CLONE_FORCE)`	✅	❌
`FileManager.copyItem(at:to:)`	✅	✅
`FileManager.copyItem(atPath:toPath:)`	✅	✅

This is accurate for macOS 14.5 (23F79). The behaviour might vary across OS releases.

Testing method

Not that this is interesting, just for posterity and to show my work a bit, in case I made a mistake.

I created large-enough files on each of my test volumes (APFS on SSD, APFS on HDD, HFS+ on SSD) that an actual copy would take tens of seconds at least, using dd e.g.:

dd if=/dev/random of=/tmp/bigfile oflag=direct status=progress bs=1k count=104857600

I then ran the various command line tools on these files, attempting to clone the file to a different name in the same folder, and observed disk I/O activity with Activity Monitor and iStat Menus.

For actual copy-on-writes I would observe that the program successfully concluded practically instantly, and there’d be at most a small blip of disk writes (for metadata modifications).

For failed copy-on-writes I would observe that the program would not exit promptly and I’d see voluminous disk writes (hundreds of megabytes to gigabytes per second, depending on the disk, sustained for many seconds until I was satisfied with the results and killed the test).

For API tests the overall approach was the same, but the APIs were invoked from inside swift repl where possible, and from a throw-away Swift script otherwise³.

I haven’t tested it, but as far as I’ve heard APFS does not actually check if the modifications actually diverge the files. e.g. if you “modify” a byte of the file to the value it already has – a pointless but technically possible operation that leaves both copies still identical – APFS will still copy the modified block.

Furthermore, APFS & Apple’s operating systems appear to have no tools (nor APIs) to deduplicate files – e.g. to detect full or partial copies and deduplicate their actual storage on disk. Not even tools that you could invoke manually if you do the hard work of first determining that two files’ contents are identical.

APFS / Apple’s operating systems rely entirely on user applications using copy-on-write explicitly and upfront. ↩︎
Curiously, clonefile first shipped in macOS 10.12 (Sierra), according to its man page, which is the release preceding the introduction of APFS in macOS 10.13 (High Sierra). Yet, as far as I can tell HFS+ doesn’t and never did support cloning – nor do any of the other file systems supported by macOS 10.12 (e.g. FAT16 & FAT32, exFAT, NFS). It’s possible it was just convenient for Apple to include it in the prior release as they were probably testing APFS with it internally, during APFS’s development.

Update: Michael Tsai pointed out that Sierra included APFS support in beta form (e.g. you could create disk images with it, but not use it for a boot disk). Thus why clonefile (and other APFS-related tools) were included in Sierra. ↩︎
While the C APIs worked just fine inside swift repl irrespective of what volumes I was targeting, the Foundation APIs oddly refused to work whenever the files were not on the boot volume, throwing “Operation not permitted” errors (NSCocoaErrorDomain 513, with no user info). If it weren’t for the C APIs working just fine I’d think it’s a sandboxing issue, but clearly it’s not (and swift repl -disable-sandbox makes no difference). 🤔 ↩︎

Bad API example: FileManager’s url(for:in:appropriateFor:create:)

Wed, 31 Jan 2024 20:40:07 +0000

I find FileManager‘s url(for:in:appropriateFor:create:) to be very unintuitive. It seems to have multiple, largely-orthogonal functions. It can provide paths to common folders (albeit badly). It can create temporary folders. It can locate volume-specific bins (Trash folders).

It is an example of bad API design. Specifically, regarding cohesion: the principle that an API should have one purpose. A litmus test for this is whether all the method parameters are always applicable¹.

It wasn’t until I wrote a test driver which explores its entire parameter space, that I was finally able to grok what the hell it’s doing and delineate its multiple modes of operation.

I’ve contrasted it with the results from its sibling urls(for:in:), to better understand what it’s doing (and expose some more of its flaws).

Test driver

FileManager – Wade Tregaskis

Copy-on-write on APFS

Should I use it?

How do I use it?

Testing method

Bad API example: FileManager’s url(for:in:appropriateFor:create:)