MacOS Extended Encrypted vs APFS Encrypted for "cold storage"

[RLight]

I just reformatted a WD RED 18TB after swapping out the Seagate EXOs in the Orico external enclosure for both MacOS Extended Encrypted (aka HFS+ with Core Storage), benchmarked, and APFS encrypted, and benchmarked and APFS encrypted is significantly slower on writes than good ol MacOS Extended Encrypted... That's interesting.

Notably, MacOS Extended Encrypted is no longer available in the GUI for formatting in the drive utility in Ventura; had to use an old Mojave machine to format the drive with the GUI for MacOS Extended, Encrypted. Ventura only permits MacOS Extended these days. I bet the CLI option is still available but I don't prefer formatting drives with CLI whenever possible (risk of human error).

APFS is the successor file format, but is running a container on a spinner a good idea? Especially if the container is the entire drive?

Thought I'd get the opinions of others if this is a good idea, bad idea, perhaps why from someone smarter than I on Mac file formats.


MacOS Extended Journaled with Core Storage (aka HFS+ with Encryption)


APFS with Encryption

I re-ran the results after re-formatting back to HFS+ with encryption after APFS encrypted formatted to ensure it wasn't some kind of garbage collection getting in the way, nope, same results. Definitely a repeatable performance regression using APFS encryption on spinners anyways compared to HFS+ with Core Storage (Encryption). APFS encrypted and HFS+ Encrypted work on both my Mojave and Ventura machines, btw. Odd move for Apple to depreciated it in the GUI. Well perhaps not, they were the first to ship a PC without a CD-ROM drive... Sure, whatever.


Is this the droid we're looking for? Okay, Star Wars jokes aside, figured I'd put up a picture, it's DPR afterall

FYI...

Enclosure:

https://www.newegg.com/p/0VN-0003-00227?Item=9SIA1DSJA85697

Drive:

https://www.amazon.com/gp/product/B08K3TFM92/

 

[Nick on the Baltic]

There is a) more to disk speed than single threaded data transfer rates, and b) other factors that may be important to you.

A couple of fairly random links to provide some info on HFS+/APFS:

https://www.techrepublic.com/article/apfs-vs-hfs-which-apple-filesystem-is-better/

https://www.amacsite.com/apfs-vs-hfs/

Looks like Apple is deprecating HFS+ encrypted, at least on Apple Silicon. and sooner or later they will probably stop offering vanilla HFS+ as an option, though with the installed base they'll have to support it for years to come.

 

[Jacques Cornell]

APFS can be significantly slower than HFS+ on spinners, but Bombich Software, developers of Carbon Copy Cloner, nonetheless recommend it for backup spinners, arguing that APFS' snapshot and other features offer useful benefits. For "cold storage", I don't imagine speed is going to be a high priority.

--
Event professional for 20+ years, travel & landscape enthusiast for 30+.
http://jacquescornell.photography
http://happening.photos

 

[RLight]

APFS can be significantly slower than HFS+ on spinners, but Bombich Software, developers of Carbon Copy Cloner, nonetheless recommend it for backup spinners, arguing that APFS' snapshot and other features offer useful benefits. For "cold storage", I don't imagine speed is going to be a high priority.

I'd agree with CCC partially; if it's a flash drive (all drives in a Mac these days), or, if it's a spinner, dedicated solely to Time Machine backups, and you're running Big Sur or later? Than yes. This is good advice, and this is the majority of end users.

That said, I did some significant research today... The short answer is allocate on write or AOW (HFS+ is AOW) was made with platters and actuators in mind, and hence its superior performance on a HDD. HFS+ is just older, hence why BTRFS is ideal as BTRFS combines the benefits of both Copy on Write (COW) and AOW, but, it's not available for MacOS making the choice between HFS+ and APFS (APFS is cow) one of do you need the features of APFS? If not? HFS+ is a journaled file system, handles large volumes, and is more efficient for lower-IOPS drives, in other words, spinners. I'm going to shorten my lengthy explanation for the sake of readers, and put this simply...

.

If you have a flash drive, and are running MacOS 10.13 (High Sierra) or later on all machines? APFS is your man.

If you have an HDD, and it's your boot drive (for whatever reason you haven't upgraded it to a SSD and you can run Big Sur...) in Big Sur or later? APFS. You're probably running Time Machine locally at least, whether you know it or not.

If, the HDD solely runs Time Machine on Big Sur or later machines? APFS. The initial sync will take longer, but subsequent syncs can leverage snapshots and clones.

If, the HDD is truly a "cold storage" hard drive where you just throw things on and off it? Example, RAW photo or video storage offloading, IE archival, what I intend? HFS+.

.

I also have a very much renewed appreciation for BTRFS which Synology uses, it's very efficient, very reliable and a good combination of AOW and COW (imagine that). Apple likely will never natively support it as a client OS manufacturer; they left the Server business over a decade ago now, that's a shame, and no modern Mac has a spinner, so there's no need to devote any time from MacOS or iOS developers to support a niche. But BTRFS makes a very strong case for spinners going forward and I very much applaud Synology for making it the default.

The only reason I'd recommend APFS over HFS+ on a spinner, is if you need snap clones, you need multiple encryption keys or something that needs shared process access; that is you're doing lots of data regression (think running VMs with snapshots, which you really should be running on flash or at least Hybrid) or backup software that leverages snapshot technology (I presume Time Machine does, but that's an assumption, it's a very logical one though). Otherwise? AOW and thus HFS+ is just going to win across the board. It was designed for mechanical drives, thus it serves "cold storage" much better on a spinner than COW aka APFS, does.

If anyone's interested I'll dig up the white papers on the subject. Interesting reading for those into that stuff.

Also in the same vein, I have a newfound appreciation for COW and APFS; if you have an SSD, and it's still formatted HFS+ and you're running a modern MacOS (High Sierra without Time Machine or Big Sur with Time Machine support of APFS) that supports APFS or supports APFS for Time Machine? You should take the time to migrate the data off, reformat / convert to APFS, as COW is just a MUCH better File System for Flash, and hence why they say it was made for flash.

I may run some detailed benchmarks in the future, time permitting, as I suspect based off my technical reading today, it's not just sequential writes that are better on AOW on a spinner, IOPS probably is a touch better too.

 

[Nick on the Baltic]

That's a lot more impressive than my quick 'research' - aka Google first page results :-D

I use APFS on HDDs that are for TimeMachine only - the last bit of performance doesn't bother me there, but future-proofing does.

White papers? Yes please!

 

[RLight]

That's a lot more impressive than my quick 'research' - aka Google first page results :-D

I use APFS on HDDs that are for TimeMachine only - the last bit of performance doesn't bother me there, but future-proofing does.

White papers? Yes please!

 

[Nick on the Baltic]

:-D I see what you did there.

Thanks

 

[RLight]

So, I have a "wrinkle" going on...

My Seagate EXOs? That I just took back out of the Synology DS224+? It scores, similarly, to my WD RED 18TB under same file formats, at least with Black Magic's test...

.

I'm going to have to break out the "big guns" and run multiple File Systems (in other words, add EXFAT to this conversation), unencrypted (as both HFS+ and APFS have encryption in these), and run multiple benchmarks.

Previously I'd established that the Seagate EXOs 18TB had faster writes, but the WD RED had faster reads... That just went out the window, at least for Black Magic.

This is going to take some time...

 

[RLight]

So, I've ripped both the WD RED Pro 18TB and Seagate EXOs 18TB out of the Synology DS224+ for testing locally against "Atlantis" aka my modified 2020 i9 iMac...

TL;DR...

The WD RED PRO 18TB is a touch faster at reads then the Seagate EXOs 18TB.

However, the Seagate EXOs 18TB is a touch faster at writes then the WD RED PRO 18TB.

.

Regarding file formats...

APFS is slower than HFS+, with no encryption, period. Not by a lot, but it's there.

Now, the Allocate on Write vs Copy on Write really shows up when you flip on encryption. HFS+ with Core Storage encryption, crushes, APFS with encryption on a spinner. So if you don't plan to encrypt? APFS, you're going to get snapshot support. But If you plan to encrypt (and you should)? See the previous discussion. HFS+ is the clear winner in performance on a spinner with encryption wrapping on the data with Allocate on Write. That's why the 30mb/sec difference in writes when encryption gets enabled on both APFS and HFS+; AOW vs COW)

Now ExFAT, it's a touch faster at writes than even HFS+, but that's because it's not journaled, at all. Oh, and you can't natively encrypt it.

In case anyone's interested in the data itself...



I ran each benchmark 3 times, aggregated and averaged. To say, pretty comprehensive. Between 3 different file formats, that means these drives have each been benchmarked plenty. And to have 2 different drives, the file systems have likewise been benchmarked, plenty, to decisively determine winners in both with a high degree of accuracy.

Just for kicks and grins, the built-in benchmarks on my Synology DS224+, agree with which drive is better at reads, and writes.

RED is for the WD Red Pro 18TB, EXO is for the Seagate Exos 18TB...

BM is for Black Magic

IOPS testing conducted by ATTO. Wouldn't run on ExFAT btw.

.

AJA EXO (Seagate EXOs 18TB)

• EXO Read APFS: 264.33 MB/s
• EXO Write APFS: 262.67 MB/s
• EXO Read HFS+: 263 MB/s
• EXO Write HFS+: 260 MB/s
• EXO Read ExFAT: 255 MB/s
• EXO Write EXFAT: 245.7 MB/s

RED (Western Digital Red Pro 18TB)

• RED Read APFS: 266.3 MB/s
• RED Write APFS: 266 MB/s
• RED Read HFS+: 271.33 MB/s
• RED Write HFS+: 270.67 MB/s
• RED Read ExFAT: 258.33 MB/s
• RED Write EXFAT: 258.67 MB/s

BM EXO (Seagate EXOs 18TB)

• EXO Read APFS: 263.43 MB/s
• EXO Write APFS: 262.9 MB/s
• EXO Read HFS+: 263.6 MB/s
• EXO Write HFS+: 261.3 MB/s
• EXO Read ExFAT: 264.97 MB/s
• EXO Write EXFAT: 262.27 MB/s

RED (Western Digital Red Pro 18TB)

• RED Read APFS: 269.37 MB/s
• RED Write APFS: 269 MB/s
• RED Read HFS+: 267.93 MB/s
• RED Write HFS+: 267 MB/s
• RED Read ExFAT: 261.37 MB/s
• RED Write EXFAT: 260.03 MB/s

ATTO EXO (Seagate EXOs 18TB)

• EXO Read APFS: 249.2 MB/s
• EXO Write APFS: 267.5 MB/s
• EXO Read APFS IOPS: 3.71
• EXO Write APFS IOPS: 3.99
• EXO Read HFS+: 258.1 MB/s
• EXO Write HFS+: 258.3 MB/s
• EXO Read HFS+ IOPS: 3.85
• EXO Write HFS+ IOPS: 3.85

RED (Western Digital Red Pro 18TB)

• RED Read APFS: 250.5 MB/s
• RED Write APFS: 251.3 MB/s
• RED Read APFS IOPS: 3.74
• RED Write APFS IOPS: 3.75
• RED Read HFS+: 274.35 MB/s
• RED Write HFS+: 274.36 MB/s
• RED Read HFS+ IOPS: 4.09
• RED Write HFS+ IOPS: 4.09

IOPS EXO (Seagate EXOs 18TB)

• EXO Read APFS: 3.91
• EXO Write APFS: 3.92
• EXO Read APFS IOPS: 3.78
• EXO Write APFS IOPS: 3.92
• EXO Read HFS+: 3.73
• EXO Write HFS+: 3.87
• EXO Read HFS+ IOPS: 3.97
• EXO Write HFS+ IOPS: 3.97

RED (Western Digital Red Pro 18TB)

• RED Read APFS: 3.91
• RED Write APFS: 3.92
• RED Read APFS IOPS: 3.78
• RED Write APFS IOPS: 3.92
• RED Read HFS+: 3.73
• RED Write HFS+: 3.87
• RED Read HFS+ IOPS: 3.97
• RED Write HFS+ IOPS: 3.97

 

[Nick on the Baltic]

All very interesting, but for my uses (backup, image file storage) I would never notice any speed difference, so I'll stick with APFS.

If I were running a database server it might be a different story...

 

[RLight]

All very interesting, but for my uses (backup, image file storage) I would never notice any speed difference, so I'll stick with APFS.

If I were running a database server it might be a different story...

 

[CAcreeks]

All very interesting, but for my uses (backup, image file storage) I would never notice any speed difference, so I'll stick with APFS.

If I were running a database server it might be a different story...

In addition to FreeFileSync to SSD, I do backups on external HDD that's formatted HFS+, so I can read it on Linux if the need arises. Last I heard, Linux FuSE does not fully support APFS.

 

[RLight]

All very interesting, but for my uses (backup, image file storage) I would never notice any speed difference, so I'll stick with APFS.

If I were running a database server it might be a different story...

In addition to FreeFileSync to SSD, I do backups on external HDD that's formatted HFS+, so I can read it on Linux if the need arises. Last I heard, Linux FuSE does not fully support APFS.

Correct, it does not.

Although I have no intent to sync between my Ubuntu system, I have however decided to go HFS+ for my "cold archive" drive, which I've also decided is going to be my WD RED PRO 18TB.

I'm going double Green in the Synology, that is putting both my Seagate EXOs 18TB, both in the Synology. The latest benchmark data I'm about to post on the PC talk, shows that my original hypotheses is correct, matching drives, do provide superior performance (in read as well).

Now what I'm puzzled about is the MD driver for Linux, does permit Read Balance, so obviously I'm missing something here on why having dual identical drives, with slightly lower read performance, would outgun a mix with a faster read drive, even though direct benchmarks confirm the data. Just means I don't understand why, or my tests are bunk, which I really pushed those benchmarks hard this round for that reason. Hard to mess up when you have a ton of repeatable data. The only thing I can guess, is what my other engineers have spoke of, that RAIDs operate "like a convoy" where it's only as fast as the slowest drive. That makes sense in writes, but reads? There's probably either some caching, or CPU or code overhead, or otherwise presentation difference I have yet to ascertain.

.

Largely I did this to rule out which drive would go in the Synology, which would go in the USB enclosure for "cold storage" and naturally producing file system benchmarks, also, produces an abundance of data to compare the two drives, as well.

 

[CAcreeks]

Although I have no intent to sync between my Ubuntu system, I have however decided to go HFS+ for my "cold archive" drive, which I've also decided is going to be my WD RED PRO 18TB.

Wow, that drive "goes up to 11" in other words 24TB in its largest size! Yours is less expensive though, $20.50 per TB instead of $23.70. WD 22TB is the least costly, $19.95 per TB.

I'm going double Green in the Synology, that is putting both my Seagate EXOs 18TB, both in the Synology. The latest benchmark data I'm about to post on the PC talk, shows that my original hypotheses is correct, matching drives, do provide superior performance (in read as well)...
The only thing I can guess, is what my other engineers have spoke of, that RAIDs operate "like a convoy" where it's only as fast as the slowest drive. That makes sense in writes, but reads? There's probably either some caching, or CPU or code overhead, or otherwise presentation difference I have yet to ascertain.

Could be the RAID controller. Does Synology make that themselves?

Sorry to say, I don't have that much data.

He or she who dies with the most data, wins. So far my wife is ahead.

 

[RLight]

Although I have no intent to sync between my Ubuntu system, I have however decided to go HFS+ for my "cold archive" drive, which I've also decided is going to be my WD RED PRO 18TB.

Wow, that drive "goes up to 11" in other words 24TB in its largest size! Yours is less expensive though, $20.50 per TB instead of $23.70. WD 22TB is the least costly, $19.95 per TB.

I buy whatever's cheapest; WD RED Pro was at the time, Seagate EXOs was at the time I purchased it too. The 18TB EXOs was 259.99 when I bought it, 14.38 per TiB. 284.99 for the WD Red Pro 18TB at the time, little more at 15.83 per TiB.

I'm going double Green in the Synology, that is putting both my Seagate EXOs 18TB, both in the Synology. The latest benchmark data I'm about to post on the PC talk, shows that my original hypotheses is correct, matching drives, do provide superior performance (in read as well)...
The only thing I can guess, is what my other engineers have spoke of, that RAIDs operate "like a convoy" where it's only as fast as the slowest drive. That makes sense in writes, but reads? There's probably either some caching, or CPU or code overhead, or otherwise presentation difference I have yet to ascertain.

Could be the RAID controller. Does Synology make that themselves?

MD; software RAID. I checked.

Sorry to say, I don't have that much data.

He or she who dies with the most data, wins. So far my wife is ahead.

Getting there, however that's part of my cold storage “plan” or regimen; keep only what's needed in my private cloud. Everything else, can go on an encrypted spinner in a shelf. You just gotta be careful of redundancy though and thus how valuable the data is that goes to archive.