As for an example of them being helpful - on Windows when you download a file from the Internet using a browser an extended attribute is used to mark that. Trying to execute the file from Explorer then explains that it was downloaded and asks if you really want to proceed.
On Linux selinux can store labels in the extended attributes.
Older ignorant tools aren't going to know about this, but don't substantially harm anything. Modern tools do know about them and do the right thing. (eg copying a downloaded file elsewhere on Windows will still give the warning). The Linux GNU cp command does require a --preserve xattr flag to copy extended attributes and does not do so by default. Dropbox does support them by default and cross platform.
Conflating extattrs with HFS style resource forks doesn't really make sense.
Extended attributes are just that... attributes. They are not the main "data" of the file. Each and every unix file system has some attributes to begin with (permissions, timestamps etc.) and they vary from filesystem to filesystem (e.g., setuid bit, immutable bit etc.). User supplied attributes just extends this concept in a natural way. All the standard tools expect to be able to call open() on the file and start reading a stream of bytes assuming there's sufficient access. That's what a "file" is.
Moreover, it's not just unix as an isolated systems. Those expectations are baked into the structure of the entire Internet. When you receiving a "file" over any medium like email, web etc., you're expecting to receive the aforementioned stream of bytes. The attributes (or extended attributes) are not expected to accompany the file data as a general case.
Resource forks on the other hand just completely work against reasonable user expectations. The example given in the OP's post is one such instance. A font file that shows up as having zero bytes to every tool that works with files including tools that expect to transfer "files" over the internet. It's just broken by design.
Just a datapoint: Macs have been sending files via the Internet with resource forks for a long time, basically since the invention of the Macintosh: https://en.wikipedia.org/wiki/BinHex
Resource forks are basically legacy from the original MacOS, and something that's being retained for compatibility, not something that's really a current design. The current replacement of resource forks is bundles, where a directory masquerades as a file in the GUI.
There are lots of reasons to hate on HFS+, but I wouldn't consider this the most important one.
It's not broken, it's just old. It's a legacy technology from the original Macintosh system released in 1984 - before it was even called "Mac OS". Your "reasonable user expectations" are conditioned by three decades of experience with systems that largely didn't exist when the Macintosh resource fork was designed.
Yes the filesystem is from the 80's and from a period where Unix filesystem model had perhaps not gained the dominance that it did in later years. The question is why is it still being used in OSX 15 years after a complete redesign of the system around a Unix kernel?
They had to support HFS+ in order to convince existing users to upgrade. The first five releases of OS X had something called "Classic mode" which was basically an instance of OS 9 running as a process, so that people could continue using their pre-OS X apps. It took six years before enough developers had upgraded that Apple could stop supporting Classic. And Classic, of course, was all about HFS+. So Apple could either backport this hypothetical unix-style filesystem to OS 9, do something kludgerous with single-machine file-sharing, require all users to partition their drives and keep everything they wanted to use in Classic on the HFS+ partition (a tough sell!), or just use HFS+ and get on with it. They wanted to stay afloat, so they stuck with HFS+, and here we are today.
BTW, one of the first very popular vulnerabilities of IIS involved making it serve a file's source code rather than executing it (a very convenient way to get database login credentials) through a ::$DATA appended to the URL.
Actually, resource forks go all the way back to the origins of classic MacOS circa 1984-85! And they were a lot more than just extended attributes -- they were a b-tree storage abstraction full of stuff like icons, chunks of 68K machine code, and anything else that simply didn't belong in a linear data file.
It's a hold-over from MacOS's very non-UNIX origins, and had the makings of an OO filesystem back in the day -- but then the adoption of NeXTStep -- er, sorry, OSX -- obsoleted it overnight. And now it's basically the veriform appendix of OSX, and leads newbies to jump to this rather odd conclusion (that it's somehow "crazy" to do something in a fundmentally non-UNIXy way).
MacOS also used to store things like the icon and position on the desktop in the resource forks. As commenters here are making clear, they have been repeatedly implemented across a variety of operating environments because it really is useful being able to store additional information alongside the single stream of bits conventionally thought to make up a file. Even on Unix :-)
Yeah. And if I recall correctly, in very old versions of the Mac OS. You actually installed things like fonts into your system by copying the resource fork data from the font file into resource fork of the (singular) "System file" which contained the entire OS (except for the shell).
That the system file contained the "entire OS" is a bit of a misconception: the bulk of the OS primitives are stored in ROM, and the system file mostly contained diffs between the OS version stored in each machine's ROM and the current OS. Eventually, the practice of storing all these diffs in the system file stopped with 7.1.2, and they began releasing the diffs as seperate files called "enablers" (before then, the introduction of any new model required a new version of the OS to be released to contain that model's diffs). It also contained a number of resources that are too big to go in ROM, such as sounds, fonts, and "desk accessories".
Let's talk desk accessories, because they're cool and they exemplify the most interesting properties of classic Mac OS. Originally, Mac OS was purely a single-tasking OS. But Apple knew that people would be very frustrated if they couldn't run small apps, like a calculator, over the active application. They got around this with an applet system they called "desk accessories", or DAs for short. DAs were technically coded as device drivers, and they were stored in the system file as resources of type DRVR, just like any other device driver. The system file populated the Apple Menu with a list of installed DAs, and when you selected one, it would patch the DA's code directly into the running application's memory, giving the illusion of multitasking.
Fonts, too, were stored in the system file as resources of type FONT. Apple honestly never expected third parties to write their own DAs or fonts, which is why they opted for such a hardcoded solution. But people did so anyway, and they wrote particularly ugly hacks to do it. So Apple caved and wrote a utility called the Font/DA Mover, which allowed you to install FONT and DRVR resources from other files into the system file. It also allowed you to do the opposite, and extract those resources out of the system file and into new files (IIRC, the system file had a limit of how many resources of each type it could have, so power users had to have some way to swap things out). Apple also introduced the word "suitcase" to refer to files that consisted entirely of FONT or DRVR resources and existed solely as holding pens for fonts and DAs that weren't installed into the system file.
System sounds, too, were stored in the system file as resources of type SND (actually "SND ", as names had to be four characters), but there was so little interest in custom system sounds at the time that Apple never made a utility for it (third parties provided them, though).
A lot of this changed in System 7. First off, System 7 came with built-in coöperative multitasking, so DAs were no longer necessary. DAs were evicted from the system file, and Apple implemented a clever hack so DA suitcases could be directly executed as if they were programs, by causing them to be patched into the Finder's memory space when double-clicked in the Finder or run from the Apple Menu. Accordingly, the Apple Menu was no longer generated by the system file but reflected the contents of a folder under the System Folder called "Apple Menu Items", which typically consisted of aliases (another new System 7 feature: aliases were kinda like symlinks, but smarter). Second, the Font/DA Mover was obsoleted, and custom sounds endorsed, by giving the Finder the ability to browse the FONT and SND resources of the system file as if the system file was a folder, allowing the user to drag and drop font suitcases and sounds files in and out of the system file. Apple also redefined the word "suitcase" to now mean "any file whose resource fork can be browsed in the Finder like it was a folder": the system file was now listed in the Finder as having a file type of "suitcase", and the contents of font suitcases could now be browsed as if they were folders (users could now drag and drop individual FONT resources between suitcases, and individual font resources within a suitcase could be double-clicked to show a preview of the font). DA files were no longer described as "suitcases", as they were simply displayed to the user as if they were executable programs. The DA API was also deprecated, and Apple began telling developers to just write regular programs instead.
System 7.1 changed this further, by pulling fonts out of the system file, leaving only system sounds. Instead, the System Folder now contained a directory named "Fonts", which was filled with font suitcases. IIRC, at some point, they also extended the "suitcase" functionality of the system file to allow keyboard layouts to be dragged and dropped in and out of the system file, but I don't remember too much about that.
The technical details of Classic Mac OS were really interesting.
What I find interesting is it you pop open Creation Kit and look at the database that powers Gamebryo/Creation Engine games like Morrowind, Oblivion, Skyrim, Fallout 3, Fallout 4, etc... take a look at their database structure.
It's virtually identical to a Mac Classic resource fork.
Great comment. Is there anywhere I can read about how it worked and what it was used for before it became a "veriform appendix"? It seems really interesting.
This article by John Siracusa was written right around the time Apple was ditching use of resource forks in OS X and contrasts some of their uses with how it's done today: https://news.ycombinator.com/item?id=10678261
To be fair, resource forks have been around since HFS first came out. I doubt Apple invented this concept, but they definitely predate MS on this front.
Not the same thing at all. On VMS, when you opened a file, made changes and saved them, you automatically got a new version. You could open any version using the colon syntax, but it's not the same thing at all as resource forks.
>As for an example of them being helpful - on Windows when you download a file from the Internet using a browser an extended attribute is used to mark that.
One maybe lesser-known use for ADS: FlylinkDC++ (and derivatives) have an option to store TTH hash data in the file's own ADS, instead of in a central hash store. It means that the hash data could be used by multiple applications, but, it's less I/O efficient to make thousands of these <4KB blocks everywhere.
On Mac Classic, if you wanted to ship a custom font with your application, you'd put it in the Resource fork in a resource of "FONT" type.
So why wouldn't a stand-alone font file consist of a Resource fork with a single resource of "FONT" type? Otherwise, the OS engineers have to develop two entirely different ways of reading in font data. Why duplicate the effort?
The system made perfect sense, both then and now. It bothers me that so few people know anything about Mac Classic, it really was an amazingly well-designed OS for its time.
It had a few interesting ideas. But no desktop OS based on cooperative multitasking can be called 'well-designed', almost anything could hard-lock the entire system at any time.
>On Mac Classic, if you wanted to ship a custom font with your application, you'd put it in the Resource fork in a resource of "FONT" type.
On Windows you can do the same thing with .rsrc in an .exe/.dll file.
It's possible you would ship a font as a PE .dll with a font resource, but for a single file the encapsulation is a little unnecessary (unless your application architecture already makes it easy to deal with PE encapsulation).
Except that in the windows example you gave the .rsrc section is part of the main data stream. It is counted in the file's size when doing a dir command. When the file is copied by all command line and shell tools or sent over the internet the data in the .rsrc section is copied/sent just like all the other data in the exe/dll.
None of this is true for the resource fork in the Mac case.
It does not strike me as a sane design. You already have file extensions, unique prefixes (like those used by the 'file' command to identify files). Why would you use fs forks to differentiate between file types?
There are various arbitrary restrictions in various filesystems and operating systems. The intention is to store smaller amounts of useful information with the file. Smaller means in the bytes/kilobytes range. For example you wouldn't store multi-gigabyte alternate language versions of a video file in its extended attributes. But you could store URLs of where to get them.
Or you could attach authorship, review, dates, keywords and similar metadata which would work for any file type, not just those whose format explicitly has that support.
If you want to get a handle on what filesystem design is like then I highly recommend Dominic Giampaolo's book on the design and implementation of the Be filesystem (he wrote BeFS). The book is freely available from his website as http://www.nobius.org/~dbg/practical-file-system-design.pdf and includes information about the design of other filesystems too. It isn't exhaustive, but does give a very good grounding in filesystems and does cover extended attributes.
Fonts, when part of a program, were stored in a "FONT" fork (conversely, icons went into "ICON"). It's natural that font files would have nothing in the data fork and the font definition itself would be under FONT.
Ah, good to know. My last Mac was m68k (Performa 630CD, one of the very last m68k models), so my memories include seeing plenty of CODE resources when playing around with ResEdit.
Oh they stuck around, of course, an entire other chapter of historical MacOS zaniness in which many programs and the OS itself contained and ran both 68k and PPC code at the same time, there were 'universal procedure pointers', 'accelerated' CODE resources (with PPC code in them) and on and on.
Actually, I believe OS X does this for small files in order to not burn a whole filesystem block on a sub 1k file.
"In Mac OS X Snow Leopard 10.6, HFS+ compression was added. In open source and some other areas this is referred to as AppleFSCompression. Compressed data may be stored in either an extended attribute or the resource fork.[13] When using non-Apple APIs, AppleFSCompression is not always completely transparent."
FS compression is transparent to any regular file-reading API; it only shows up in lower-level APIs, like if you want to copy the file and keep the compression.
I was responding to the comment "You wouldn't put the entire content of a file in an extended attribute, would you?" HFS+ does, although in this case, it does without leaking that detail to callers.
On Windows/NTFS they are called alternate data streams - https://en.wikipedia.org/wiki/Fork_(file_system) - to use specify a colon and name after the filename (eg example.txt:myads).
On Unix, Linux, OS/2 etc you can find extended attributes - https://en.wikipedia.org/wiki/Extended_file_attributes - which allow storing key value pairs on a file. Restrictions exist and vary.
As for an example of them being helpful - on Windows when you download a file from the Internet using a browser an extended attribute is used to mark that. Trying to execute the file from Explorer then explains that it was downloaded and asks if you really want to proceed.
On Linux selinux can store labels in the extended attributes.
Older ignorant tools aren't going to know about this, but don't substantially harm anything. Modern tools do know about them and do the right thing. (eg copying a downloaded file elsewhere on Windows will still give the warning). The Linux GNU cp command does require a --preserve xattr flag to copy extended attributes and does not do so by default. Dropbox does support them by default and cross platform.