Blocking access to Time Machine makes it very difficult to actually use Time Machine, since it’s then difficult to retrieve files from a backup (you have to then use the stupid ‘warp’ Time Machine interface, which is slow, ugly, and buggy).

Luckily, it turns out there is a fairly simple solution that isn’t disabling SIP entirely (which requires multiple reboots in order to do, so is typically quite disruptive & slow). It appears that any application granted Full Disk Access (System Preferences → Security & Privacy → Full Disk Access) can read Time Machine backups.

In case you’re unfamiliar, the symptoms of this problem include:

• Being unable to navigate into Time Machine backups in the Open / Save / etc dialogs.
• Being unable to see – through ls or similar tools – the contents of Time Machine backups via Terminal.
• Apps reporting errors like “The file “Foo” couldn’t be opened because you don’t have permission to view it” or bluntly “Operation not permitted” when trying to read something in a Time Machine backup.

There’s a strange & ironically very bad security quirk though – curiously, any tools run via Terminal inherit Terminal’s access (or lack thereof) to Full Disk Access. They don’t use whatever setting might be specified for them in the Security & Privacy preferences. This is pretty baffling, as it means to give Full Disk Access to anything you run via Terminal, you have to give it to everything you run via Terminal. Anything you specifically give Full Disk Access won’t actually receive it if it happens to be launched via the Terminal (which confused me for a while, since it’s so unintuitive).

I’m guessing whatever mechanism enforces all this so-called security is based in LaunchServices or somesuch – while the Finder and most things in general will launch apps via LaunchServices, as detached & independent process sessions, Terminal doesn’t – everything it runs, from the shells down, run under it in the process hierarchy, and seemingly share its security & privacy settings.

Sigh.

It turns out there’s actually a nominally supported way to address exactly this scenario – tmutil associatedisk (kudos to Simon Heimlicher for documenting this).  From the man page:

   associatedisk [-a] mount_point snapshot_volume
           Bind a snapshot volume directory to the specified local disk, thereby reconfigur-
           ing the backup history. Requires root privileges.


           In Mac OS X, HFS+ volumes have a persistent UUID that is assigned when the file
           system is created. Time Machine uses this identifier to make an association
           between a source volume and a snapshot volume. Erasing the source volume creates
           a new file system on the disk, and the previous UUID is not retained. The new
           UUID causes the source volume -> snapshot volume association to be broken. If one
           were just erasing the volume and starting over, it would likely be of no real
           consequence, and the new UUID would not be a concern; when erasing a volume in
           order to clone another volume to it, recreating the association may be desired.


           A concrete example of when and how you would use associatedisk:


           After having problems with a volume, you decide to erase it and manually restore
           its contents from a Time Machine backup or copy of another nature. (I.e., not via
           Time Machine System Restore or Migration Assistant.) On your next incremental
           backup, the data will be copied anew, as though none of it had been backed up
           before. Technically, it is true that the data has not been backed up, given the
           new UUID. However, this is probably not what you want Time Machine to do. You
           would then use associatedisk to reconfigure the backup so it appears that this
           volume has been backed up previously:


           thermopylae:~ thoth$ sudo tmutil associatedisk [-a] "/Volumes/MyNewStuffDisk"
           "/Volumes/Chronoton/Backups.backupdb/thermopylae/Latest/MyStuff"


           The result of the above command would associate the snapshot volume MyStuff in
           the specified snapshot with the source volume MyNewStuffDisk. The snapshot volume
           would also be renamed to match. The -a option tells associatedisk to find all
           snapshot volumes in the same machine directory that match the identity of
           MyStuff, and then perform the association on all of them.

Perfect – and I particularly like the subtext of the prose, which seems to be a subtle acknowledgment that this is a thing that happens frequently, and that macOS’s default behaviour is stupid… “recreating the association may be desired”.  No shit.

Unfortunately, that command doesn’t work in Mojave.  I’m apparently not the first person to notice.

It appears the tightened security, and in particular expansion of SIP to cover many more parts of the system including Time Machine backups, are to blame.  Even granting tmutil Full Disk Access etc in the system security settings is of no use (contrary to the stated purpose of Full Disk Access).

So you have to disable SIP first – which requires a reboot, obnoxiously – and only then does tmutil work again.  You’ll want to enable SIP again once you’re done, most likely, as the protections it provides are useful – it appears tmutil nve eeds to be updated to account for the new protections.

One thing I have very quickly discovered, however, is that Lightroom “Classic” deliberately chooses not to perform some functions if it is le tired.  Or it thinks your computer is le tired.  By which I mean, if there is pretty much anything else running and consuming CPU time (and/or RAM?), it refuses to even attempt some operations.  HDR merges is the first one I hit.  I was a bit flummoxed by it just happily queuing up a number of HDR merge operations, and them just sitting there in its queue, with no indication of error – just never executing.

Only after I quit or disabled a bunch of other processes – any and all that were using any measurable CPU time – did it finally, about ten seconds later, decide that it was now willing to consider my ‘requests’.

#%@!ing fussy little turd.

It’s worth noting that it’s not the only popular app, on macOS, that does this same bullshit.  Time Machine is another big one.  At least in Time Machine’s case I can see a more plausible line of reasoning behind it, even if it is misguided – the user’s probably not explicitly waiting for a Time Machine backup to complete.  As in, not all the time.  Sometimes they are. And they certainly expect backups to happen at all, which on a consistently busy machine simply doesn’t happen.  So Time Machine’s reluctance to function on a working machine is still stupid overall.  But Lightroom refusing to complete a user initiated, user-interactive, and user-blocking operation, is just patently stupid by its very notion.

Update:  Worse, now it doesn’t work at all.  And a quick web search shows many other people having the same problem, and Adobe as usual doing nothing about it.

Incidentally, I tried to log in to Adobe’s forums in order to ‘Me too’ those issues, only it won’t let me log in anymore, falsely claiming my password is invalid.  Good job, Adobe, good job.

I was quite disappointed to see that APFS isn’t even trying to provide data integrity.  Data integrity is kind of step 0 of any file system, and checksums or use of ECC is pretty much standard in modern & leading-edge file systems.  APFS doesn’t want to be one of those, it seems.

Case in point why this matters:

I have a bunch of old backup drives, because drives are cheap and until recently I could just buy a new one once the current one filled, instead of ever deleting a backup.  Periodically I go back through these old backup drives and do some basic integrity checks (S.M.A.R.T. bad block scans, file system checks, etc).

I also run a comparison of key data between those backups and the current versions on my computer, for files which generally shouldn’t change nor disappear – e.g. photos, videos, key documents, etc.

And today I found that at least half a dozen valuable personal videos (and a few photos) were corrupt, in the versions on my computer.  Luckily, the versions in the ancient backups were still good, so I could replace the corrupt ones.

This corruption was completely silent, until my ‘paranoid’ and time-consuming checks discovered it.

It’s far from the first time.  A failing drive years back corrupted a huge portion of my music library – silently, as far as the file system & OS were concerned.  Periodically I’ve discovered photos (of which I have huge numbers – the majority of my data) which have become corrupt at some indeterminate point.  And I’ve of course had file system [metadata] corruption occur many times, sometimes requiring complete erasure of the disk, and recovery or rebuilds from backup (a few times I’ve had to use data recovery software, where backups weren’t available).

Most, if not all, of these issues would have been discovered by even the most trivial file integrity protections, in the file system.

The notion that modern disks somehow magically protect against all silent data corruption is abject poppycock.  They’re more likely to suffer from it than older disks – a byproduct of higher densities and market demand for cheaper, crappier storage products.

And the implicit assertion that Apple’s file system driver, and kernel overall, are somehow completely free of bugs… is just batshit crazy.

Addendum

Since Apple aren’t interested in protecting anyone’s valuable personal data, I’m on the look-out for other options.  Manual use of shasum is one, for now, but a more streamlined and fool-proof system would be better.  Alas, none seems to exist[1. There is chkbit, but it relies on MD5… probably acceptable for this use case, but needless in the face of decades of better hash algorithms.  And it’s written in JavaScript.  Ew.].  Yet.

Luckily I’m far from the first person to need to create RAID and/or encrypted CoreStorage volumes, in El Capitan.  Florian Knapp has a concise summary of how to set up an encrypted RAID volume.  Tom Nelson (of About.com) has a slightly more detailed tutorial for managing the RAID part.

Now I just wish the hard drive industry would actually push capacities up, like they once did, so that I don’t have to resort to striped RAID sets just to make a disk big enough for Time Machine backups.  It feels like we’ve been effectively stuck at 6 TB for many years now, and affordable 8+ TB drives aren’t really on the horizon (Seagate & Western Digital have offerings, but historically have been bad brands for drive reliability, e.g. Backblaze’s data, plus my own personal experience with their drives).

An internal state error occurred
Operation failed…