I don’t really remember the timing – sometime between 2006 and 2010 – but presumably around 2006 as I recall it was when Core Data was still relatively new. For whatever reason, there was a huge push internal to Apple to use Core Data everywhere. People were running around all over the place asking “can it be made to use Core Data?”, for Apple’s frameworks and applications.

Keep in mind that Core Data at that time was similar to SwiftData now – very limited functionality, and chock full of bugs. But of course it’s the nature of ‘shiny’ new things that their proponents think it’s the second coming and the cure for all ills.

So, I recall sitting down with a couple of folks from the Core Data team, that were there to see if Shark could adopt Core Data. A little like letting the missionaries in, if only out of morbid curiosity.

Have you heard the good news? Core Data is here to save your very data. It’s effortless and divine and its unintuitive, thread-unsafe API will definitely not be the bane of all its users for the next fifteen years.

Jokes aside, they were in fact earnestly curious if Shark could use Core Data, instead of its own purpose-built binary formats, for storing & querying its profiling data. It was perhaps the classic case of naively underestimating the complexity of a foreign domain. By my recollection, they assumed our profiling data was just a small handful of homogenous, relatively trivial records. “At second N, the program ran the function named XYZ” or somesuch.

I think we (Shark engineers) tried to be open-minded and kind. We were sceptical, but you never know until you actually look. We could see some potential for a more general query capability, for example. But of course the first and most obvious hurdle was: how well does Core Data handle sizeable numbers of records? Oh yes, was the response, it’s great even with tens of thousands of records.

We asked how it did with tens of millions of records, and that was pretty much the end of the conversation.

It wasn’t just the volume of data, it was also the dramatic difference in representation efficiency. The in-memory representation in Shark was basically as efficient as it could be – basically just arrays of compact structs, sometimes with pointers to other arrays (which might share a malloc block to avoid the overhead of small allocations) which were usually just of uint32_t or uint64_t. The most important operations – indexing to an arbitrary point in the profile’s timeline, then scanning forward over the data – were about as fast as they can possibly be.

In contrast, Core Data would have required an entire object (NSManagedObject subclass) for at least every sample, if not every uintXX_t in the callstack (depending on how ‘pure’ you wanted the design to be). It would have increased memory usage by at least an order of magnitude – and Shark already struggled with big profiles on the hardware of the day, which typically had just a couple of GiB of RAM. Even the most trivial operations – like reading the data in from disk and iterating it sequentially would have been thousands of times slower.

In defence of the Core Data folks in the meeting – and I don’t remember who specifically it was – they never tried to misrepresent or exaggerate what Core Data could do. I seem to recall them being quite nice people. But as soon as we started explaining the type and volume of data that we worked with, they clearly gave up on any kind of pitch. Core Data was designed for developer convenience, not runtime efficiency or performance.

It’s never ceased to surprise and disappoint me how many folks try to arbitrarily apply generalised data storage systems – particularly SQLite and MySQL, or wrappers thereover. Usually for the same reasons – perceived convenience to them, right now, not necessarily efficiency (nor the convenience of their successors).

I guess by modern standards SQLite is considered efficient and fast, but – hah – back in my day SQLite was what you used when you didn’t have time to write your own, efficient and fast persistent data management system.

See also JSON and its older sister XML. 😔

The question was raised whether an official Swift backtracer should try to support code that doesn’t use frame pointers. Which immediately raised the question – in my mind – of if anyone is still using the “optimisation” of omitting frame pointers, anyway. And perhaps more importantly, whether they should still be omitting frame pointers.

What is a frame pointer?

A pointer to a stack frame, held in a well-known location. That location can be in the stack itself (forming a linked-list of the stack frames) or in registers (e.g. the x29 register on AArch64, or RBP register on x86-64).

The controversial part – insofar as there is any controversy – is in dedicating a CPU register to hold a frame pointer (to point to the start of the current stack frame). It’s super convenient for a lot of things, but particularly for debuggers and profilers as it gives them a reliable and very fast way to find the top of the current callstack. But it’s not technically required for the program to function.

No live CPU architectures, that I’m aware of, have a dedicated hardware register for frame pointers. So you nominally have to “give up” a GPR (general-purpose register) in order to have a frame pointer.

Franz Busch pointed out that some notable software still ships with frame pointers omitted, e.g. apparently some major Linux distros. I suspect it’s merely some inertia (or simply oversight) that’s delaying getting people off of that old crutch. I’m not remotely surprised that some big Linux distros are in this bucket – they tend to be absurdly conservative and slow to change¹. And it’s mind-boggling how much vitriol restoring frame pointers generates from the peanut gallery.

From watches to servers these days – and frankly most of the embedded space, since it’s mostly ARM – everything generally has an ISA with sufficiently many GPRs to negate any big benefit from omitting frame pointers. Giving up one of 31 GPRs (for e.g. AArch64, the dominant CPU architecture family today) is pretty insignificant for the vast majority of code, because almost nothing actually uses all 31 GPRs anyway. It only makes a significant difference² when the CPU design is register-starved to begin with, like i386. And those architectures are largely dead, in museums, or restricted to very tiny CPUs as used in some microcontrollers (“embedded” systems).

Even back when i386 et al were still a concern, the proponents of -fomit-frame-pointer often argued not on the potential merits of the trade-off, but rather that it was a “free” performance boost, so even if it was only by a percentage point or two, why not? They of course were either naively or deliberately overlooking the detrimental effects.

There may still be software for which omitting frame pointers is the right trade-off, even on modern CPUs. But I find it hard to believe there’s enough cases like that to warrant accomodation in standard tools.

A brief trip back to Apple circa 2007

Back in the brief window of time when i386 was a thing for the Mac (32-bit Intel, e.g. Core Duos³ as used in the first MacBooks), I was at Apple in the Performance Tools teams (Shark & Instruments), and it was a frustration of ours that -fomit-frame-pointer was a noticeable performance-booster on the register-starved i386⁴ architecture⁵, so it was hard to just bluntly tell people not to use it… yet, by breaking the ability to profile their code, people who used it often left even bigger performance gains on the table (or otherwise had to invest much more labour into identifying & resolving performance problems).

At one point there was even an Apple-internal debate about whether to abandon kernel-based profiling in favour of user-space profiling⁶ because implementing backtracing without frame pointers is possible but very expensive and requires masses of debug metadata (e.g. DWARF), making it highly unpalatable to put in the kernel. Thankfully there were too many obvious problems with user-space profiling, so that notion never really got its legs, and then x86-64 finally arrived⁷ and it was mooted.

1. e.g. Ubuntu still not officially supporting ImageMagick 7 even though it’s been out for nearly a decade. ↩︎
2. Aside from the question of register space, there is additional cost to implementing frame pointers, as additional instructions are required around function entry & exit in order to maintain the frame pointers – to push & pop them off the stack, etc. The cost of those is usually insignificant – especially in superscalar microarchitectures, as is the norm – so that aspect is not typically the focus of the controversy. ↩︎
3. Tangentially, I vaguely recall us Apple engineers kinda hating the Core Duo (Yonah), or more specifically Apple’s choice to use it. Apple used them only for a tiny window of time, from May 2006 to about November 2006 when the Core 2 Duo (Merom) finally replaced them across the line. I don’t recall all the reasons that the Core 2 Duo was superior, but they included that Core 2 Duo corrected the 32-bit regression (for Macs) and performed much better. Anytime Apple releases a Mac with a dud processor in it, like those Core Duos, a lot of Apple engineers die a little inside because they know they’re going to be stuck supporting the damn things for many years even after the last cursed one rolls off the assembly line.
   
   It’s still a mystery to me why Apple rushed the Intel transition in this regard. They only had to wait six more months and they could have had a clean start on Intel, with no 32-bit to burden on them for the next seven years. ↩︎
4. Why do I keep calling it “i386”? Isn’t it officially “IA-32”? Well, yes, but that’s (a) only retroactively and (b) only ever used by Intel. Though I guess “x86” is probably the more common name? Yet “i386” is in my mental muscle memory. Maybe that’s just how we used to refer to it, at Apple? Maybe just because that’s the name used in gcc / clang arch & target flags?
   
   Incidentally, clang -arch i386 -print-supported-cpus on my M2 MacBook Air still lists Yonah (those damn Core Duos) as supported. Gah! They won’t die! 😆 ↩︎
5. It’s funny how the Intel transition is now heralded as being amazing and how much better Intel Macs were than PPC Macs, but for a while there we lost a lot of things, like a 64-bit architecture, an excellent SIMD implementation, and the notion of more than [effectively] six GPRs.  ↩︎
6. There were at the time already some Apple developer tools that did user-space profiling, most notably Sampler (now a niche feature in Activity Monitor) and early versions of Instruments (in fact Instruments still has the Sampler plug-in which does this, although I can’t really fathom why anyone would ever intentionally use it over the Time Profiler plug-in). ↩︎
7. In the sense of all Macs adopting it, not just the Mac Pro. It was easy to ignore i386 at that point because it was then all but officially a dead architecture as far as Apple were concerned. ↩︎

It immediately reminded me of the Shark 5 re-implementation of the profiling interface between the kernel and Shark. Shark 5 of course didn’t actually survive to birth – it was snuffed out by politics and, admittedly, a bit of our own hubris in the Shark team – but that underlying infrastructure did, as the guts of Instruments‘ Time Profiler and System Trace features.

In Shark 4 and earlier, userspace would allocate a single big buffer of memory for the profiling data, hand it [back] to the kernel¹, the kernel – specifically the Shark kernel extension – would write into that buffer, then when the buffer was full profiling would end². The userspace driver – Shark – could technically start profiling again immediately, but of course you’d often have a gap in your profiling – potentially a big one, for the more expensive profiling modes (e.g. System Trace) or on heavily-loaded machines. Shark didn’t run with elevated priority, as far as I recall, so it could get crowded off the CPU(s) by other programs. All communication between user & kernel spaces was via Mach messages, which are ultimately syscalls, and was time-sensitive since kernel- & user-space had to be in lock-step all the time.

For Shark 5, we wanted to up the ante and remove limitations. We wanted to be able to record indefinitely, limited only by disk space. More importantly, we also wanted to be able show profiling results live.

So, we – and I use the term loosely, as I think it was mostly Rick Altherr with possibly the help of Ryan du Bois – came up with a mechanism that’s very similar to io_uring, but years earlier, in ~2008. Userspace would allocate multiple smaller buffers – also easier to acquire given the requirement to be physically contiguous – and hand those to the kernel somewhat as needed. Userspace merely needed to stay ahead of the kernel’s use, which didn’t necessarily mean pre-allocating all the buffers. When each of those individual buffers filled up with profiling data from the kernel, they’d be made available back to the userspace side³. Shark (in userspace) would optionally – if in ‘live’ mode or if running short on empty buffers – process those buffers while profiling was still actively occurring, freeing them up to go back into the ring for the kernel’s use. It did mean you slightly increased the probability of perturbing the program(s) under profiling, but that was mostly just a matter of ensuring the buffers were large enough to sufficiently amortise the cost of their processing & handling. I don’t recall the extent of the processing, but I think it was not necessarily much more than copying the data out into non-wired memory (maybe even a memory-mapped file?). That was cheaper than allocating and wiring new buffers.

It worked really well, and is possibly still the implementation used to date – while it appears a lot of the frameworks (e.g. CoreProfile) have disappeared from macOS, presumably rewritten somewhere else, I’d be surprised if the kernel-user interface itself has changed much. It was pretty much perfect.

The creation of this new, superior profiling system also spurred me to create rample, a CLI tool kind of like top which showed you what your CPUs were doing in real time⁴ as a heavy tree, much like you’d get in a Time Profile in Shark (and later Instruments). It was super quick to write – just a day or two, I believe – and yet I unexpectedly found that it was more useful than Shark itself. Being able to see – practically instantly by virtue of how lightweight it is to launch a simple CLI program – what’s chewing on the CPU, including inside the kernel, was incredibly useful. Instruments is the closest you can get today on a Mac, but it’s super slow and clunky in comparison. It’s one of my top regrets, of my time at Apple, that I didn’t get rample into Mac OS X, or at least into the Dev Tools.

1. I don’t recall precisely why, but it was apparently better (or only possible?) to allocate the buffer in userspace, even though that was ultimately done with syscalls serviced by the kernel. I think it had something to do with being easier to handle failures, like being out of memory, in particular contiguous memory – the memory was wired during use to ensure the kernel wouldn’t have to deal with the practical nor performance problems of page faults while recording profiling data. ↩︎
2. Which was a problem because generally profiling sessions were specified to run for a specific time duration, by the end-user. So the buffer had to be sized to avoid filling prematurely, which meant wasting memory. ↩︎
3. Technically they were always available – we didn’t bother changing access permissions – but the userspace app would coordinate with the kernel via an elegant lockfree state machine. That state machine was a superb design – it deserves its own detailed nostalgia trip. ↩︎
4. Well, updated every second or so, like top. ↩︎