Storage Spaces, exFAT, 512e/4Kn, ETL, Scripting

1. Long tuning with a new file server, Microsoft Windows Storage Spaces (@ Wikipedia) does not support shrinking the file systems? That's silly, because even traditional NTFS does allow that. I know it would be an intensive operation, but... So what? Creating a new smaller volume and copying everything... Awwww. To be honest, I find these limitations quite baffling and silly with modern storage platforms. They add "cool new layers" which are still inherently restricted by arbitrary limitations that could be technically solved quite easily. I know it's a slow and complex operation to shrink a system like that, but it's far from impossible as they claim. Duh.

2. Recovered a critical exFAT (@ Wikipedia) system for a friend. Don't ask me why they didn't have backups of the data on an external exFAT drive. Yet, I'm sure this is a story too many IT guys have heard too many times. In this case, the exFAT headers were corrupted; to slightly complicate and mislead the process the drive also contained a valid GPT backup, yet the drive was a naked but partially (header) corrupted exFAT system. First I checked the start of the disk using xxd directly, it looked good. It was obvious that it contained hallmarks of an exFAT file system as expected. No GPT (@ Wikipedia) or MBR information was there. But when I ran testdisk it immediately said that GPT is corrupted and it found a valid backup copy. Haha, nice trap! After a while wondering about the situation, I confirmed that the drive did contain only a single naked exFAT system, yet it wasn't mountable with Linux, Mac, or Windows. Nor the backup copy (6 KiB after the first (primary) 6 KiB of the data on the drive) of the exFAT was good. Don't ask me how that happens, I don't know what led to that situation. Anyway, after considering the situation a while, I took a full ddrescue copy of the drive and then extracted the data with RecoverExFat perfectly. I do happen to have adequate storage space for that. But for many users, it's a problem if a huge external storage drive needs to be recovered, and they have no space to store the data from the drive twice! i.e., first the FULL disk copy and then a COPY of the data from the copy to another destination. Yet the third copy could be the original drive, so basically you'll just need twice the space. Another "normal user" case is that they remember that X on the drive was important. After recovery they're like, ooh, you also recovered Y and Z, I thought I had already lost those ages ago. Yeah! Golden tip from an old IT pro. Remember, the customer / user always lies, doesn't know or remember and... Well, let's say "human". Haha. Anyway, I recovered everything requested successfully. And as a bonus a lot more stuff than the user even knew they had lost. Allocation bitmap, up-case table, and root directory are also easy to recognize and great data / file system markers.

3. Had a (too) long discussion about Avanced Format / 4Kn native (@ Wikipedia) drives. It seems that most drives are still defaulting with 512e (512-byte emulated) for compatibility reasons, and even new SSD drives, even if 4Kn (4096, 4 KiB) is "preferred", do not default to it unless you specifically change the setting. Duh. And as we know, 99.99% of users never do that. This seems to be one big "legacy" / future mess as well. Even if 4Kn is old stuff, legacy options are still preferred. Because nobody is using the new option. Classic. Also the fact that "new" data structures like GPT don't "account for" different sector sizes, is another classic fail. Interestingly exFAT falls for the same trap using padding to align data to 4 KiB sectors. At least NTFS, ext4, and Btrfs do not have that legacy design problem.

4. Thread priorities, about my ETL (@ Wikipedia) jobs and mass integrations, some admins have been wondering why my jobs do not cause significant performance impact, even if run during peak times. Well, that's because of: 1) I optimize the queries and make sure efficient indexes are available. 2) I minimize locking times. 3) Minimize shared state as much as possible. 4) I run bulk non-time-critical processing using threads and utilize lowered or even idle priority for the tasks. That's why most of my stuff runs completely unnoticed, even if those are quite heavy lifting batch jobs at times. Nothing new yet.

5. Improved many of my media handling scripts by adding persistent flags. Just like when doing unit tests, the tests / tasks need to be runnable independently. But for efficiency reasons, sometimes it makes sense to batch those without doing too extensive reset work in between. Now my media handling script contains a batch flag, which avoids unnecessary repeated operations like mount/unmount during the script as long as the batch flag exists. The batch flag is stored in the state management script, which is called by every unit independently. But now it's being ignored while in batch mode. This allows running any of the batched tasks independently without needing any manual state management, but makes the batch run much more efficient by removing unnecessary repeated steps. Oh well, nothing new, sure. But unfortunately often developers and admins skip these things, because "it doesn't really matter, and it works". Sure it does, but it's still not a smart approach!

2026-07-12