One of the cool things in the Torch Triple X emulator is having the floppy disk support working properly. The emulator can read and write floppy images (thanks to the Torch architecture using the SCSI interface for floppies so it’s the same code as for hard disks – yay, Torch!), and now I've even added a separate graphical disk manager to insert and eject floppy disks while the emulator is running. Ohh, get me.
 |
| Look at that there fancy disk manager window |
Everything appeared to be working rather well. And then I hit a snag.
The Torch Triple X runs its own version of System V Unix, and the emulator supports floppy disk images that use the correct format. These images can also be mounted directly under Linux, provided the Linux kernel supports the correct format which just so happens to be the old System V (sysv) filesystem. Most modern Linux kernels have dropped support for SysV, which is why I've had to install an older 6.14 kernel specifically for working with Torch Unix filesystems on my new laptop. (Did I mention I have a new laptop?)
This has been extremely useful as it allows me to copy files directly between my (new) laptop and the emulated Torch
At least, that was the theory.
While working with one of the floppy images, I discovered that attempting to copy another file onto it resulted in a rather unhelpful error: the disk was full. This was weird because, as far as I could see, the disk only a handful of very small files on it.
Running df under Linux produced something like this:
 |
| 'df' says that /mnt/flop has 5 blocks free?? |
 |
| And 'ls' shows about 6KB worth of file. WTF? |
But the directory listing only had three files listed occupying a grand total of about 6 KB. Where the fudge has the other 700 KB gone?
My initial suspicion was that something had gone wrong with the filesystem's allocation information. System V Unix uses a relatively simple filesystem structure compared with modern filesystems such as ext4. One of the main things it does it to keep track of which disk blocks are available for storing data. So, when a file is created, blocks are allocated to it and then when the file is deleted, those blocks are supposed to become available again.
If that process goes wrong then a disk can gradually fill up with blocks that are no longer associated with any existing file but that aren’t available to use for other files. Since the floppy image had been used several times for copying and deleting files, this seemed like it could be a possibility.
 |
| i-node, right? |
Looking at the inode information on the disk image in Torch unix was also confusing. The filesystem supported 174 inodes, but only five were in use. For anyone unfamiliar with Unix filesystems, an inode is essentially the record containing information about a file, including where its data is stored. So, from this, I think I could be fairly sure that there weren't hundreds of forgotten files hiding somewhere.
Something else must be happening.
Rather than continue to dig at an image that might have become corrupted through repeated use, I decided to start again to try an work out what the heck was causing the problem.
I created a brand-new 720 KB floppy image under Linux:
dd if=/dev/zero of=floppyblank.img bs=1024 count=720
This produces a file containing 737,280 bytes, all initially set to zero.
I then inserted the image into the Torch emulator and used my shiny ‘sh’ front end (pty) in Torch Unix to prepare it:
format /dev/fdsk/0
This formats the raw device and is basically a SCSI low level format.
mkfs /dev/fdsk/0 1440
This creates the filesystem as SysV with 1440 blocks of 512KB, giving the expected 720KB size.
mount /dev/fdsk/0 /mnt/flop
Finally, we mount the blank image so we can write to or read from it.
Running the equivalent ‘df’ command on the Torch showed 1412 blocks free. That means 512KB x 1412 giving 722,944 bytes or 706KB. So far so good and everything looked perfectly normal.
So then I unmounted the floppy, shut down the emulator and mounted the image under Linux (I’m using Pop OS).
sudo mount -t sysv -o loop,ro testimage.img /mnt/flop
This mounted the image at /mnt/flop. A quick df and, would you believe it, there it was again. Most of the free space was used! And I haven’t done a **** thing with the image at ALL! GAH!
 |
| WHERE IS MY FREE SPACE? |
At this point, I was thinking of two possibilities. Either the emulator was doing something particularly unpleasant to its floppy images when it was formatted so Linux was having difficulty interpreting them, or the emulator was fine but Linux was being a bit crap and was just having difficulty reading them.
So I went back to the Torch emulator to ask again how much free space was available. Even this turned into another minor adventure when the Torch df command initially produced absolutely no output whatsoever. Not an error. Not a warning. Nothing.
GAH!!!
After restarting the emulator and repeating the operation, however, it decided to work. (Isn’t that the definition of madness?)
The Torch reported:
/mnt/flop /dev/fdsk/0 1412 blocks 174 i-nodes
That meant (you can probably guess):
1412 × 512 = 722,944 bytes
Or approximately 706 KB free.
So now there were two operating systems looking at the same floppy image and seeing two totally different things. Torch says it’s empty, Linux says it’s full. Brilliant.
But both could read the directory with no issues.
This was beginning to look less like an emulator problem and more like a disagreement over the filesystem's internal structures. There’s only one way to find out. FIGHT! Ahem.
Anyway, after a lot of googling (it’s amazing how interesting filesystem formats can be) I thought I should take a closer look at the superblock. Apparently, every System V filesystem has one of these, and it contains lots of important information about the filesystem including its size, the inode information and (ta-daa) the number of free disk blocks.
So to have a look at the superblock in these images I wrote a python script to read it directly. Yay python!
The script looks at the possible byte orders and filesystem layouts, and it produces the following result:
SysV2 (Big endian)
First data block : 13
Total blocks : 720
Free cache count : 7
Total free blocks: 706
That looked promising. The filesystem contained 720 logical blocks, and its superblock said there were 706 free blocks. In other words, Torch's filesystem ‘metadata’ said exactly what would be expected for a regular blank floppy.
But there was another number that stuck out.
Free cache count: 7
This is suspiciously close to the seven kilobytes Linux claimed were available. Hmmm.
After a bit more reading it seems that System V filesystems use a linked free-block list to manage available disk space. In other words, rather than storing every free-block address in the superblock, they keep a small collection of addresses there. When those addresses are all used up, another disk block contains the next collection. This continues until the complete list has been followed all the way to the end.
Think of it as a series of pages in a notebook, each containing a list of free locations and a reference to the next page. Even my small brain managed to deal with that analogy..
The Torch superblock contained seven free-block entries:
35
32
29
26
23
20
17
The first entry, block 35, was especially important because it should lead to the next group of free blocks. So I extended the Python script to examine the contents of block 35.
The first 32 bytes looked like this:
00 00 00 32 00 00 00 b9
00 00 00 b6 00 00 00 b3
00 00 00 b0 00 00 00 ad
00 00 00 aa 00 00 00 a7
Initially, the script interpreted the first two bytes as the number of entries:
00 00 = 0
That would indicate an empty list. But this just wasn't right.
But then, if we interpreted the first four bytes as a 32-bit ‘big-endian’ number instead, we get:
00 00 00 32 = 50
And then all subsequent four-byte values looked like very plausible disk block numbers:
00 00 00 b9 = 185
00 00 00 b6 = 182
00 00 00 b3 = 179
00 00 00 b0 = 176
So, could it be that the Torch filesystem is storing the number of entries as a 32-bit value, but Linux was interpreting it as a 16-bit value? Hmmm.
To test the theory, I updated the Python script again. This time it treated the count at the beginning of each chained free-list block as a 32-bit value and followed the entire chain.
The output turned out to be the smoking gun:
Block 35: 50 entries
Block 185: 50 entries
Block 150: 50 entries
Block 112: 50 entries
Block 262: 50 entries
Block 212: 50 entries
Block 362: 50 entries
Block 312: 50 entries
Block 462: 50 entries
Block 412: 50 entries
Block 562: 50 entries
Block 512: 50 entries
Block 662: 50 entries
Block 690: 50 entries
Fourteen linked blocks with each containing 50 entries.
When combined with the seven entries in the superblock, that gave:
7 + (14 × 50) = 707
That was frustratingly close to the superblock's reported total of 706 free blocks. Looking at the last free-list block though, block 690, its first entry was zero. So this was the terminating marker at the end of the list, not actually a free block.
So in reality the count was:
7 + (14 × 50) - 1 = 706
Exactly 706 free blocks. Not approximately 706. Not somewhere in the region of 706. Exactly the number recorded in the superblock. GET IN!
Every supposedly missing block accounted for. Nice.
And then Linux confessed to something that I should have spotted at the start of this impromptu investigation. I had a look at ‘dmesg’ i.e. the Linux kernel messages with and guess what I found:
sysv_count_free_blocks:
free block count was 706, correcting to 7
That message explained exactly what the heck Linux was doing. It read the superblock and found the recorded count of 706 free blocks. And then it then attempted to traverse the free-block list itself. But because it expected a 16-bit count where the Torch had written a 32-bit count, it interpreted the first chained block as containing zero entries. And so it stopped.
Linux then decided that the superblock's count must be wrong and used its own calculated value of seven. This explained why df had been insisting that the floppy was almost full even though nothing had been written to it.
So it wasn't a problem with the emulator losing data, it wasn't caused by deleted files failing to release their blocks, and it wasn't a completely corrupt filesystem. It was an obscure difference in how the two Unix implementations represented the linked free-block list.
Now I know what is happening, the next question was what to do about it. What I really wanted was the ability to copy files to an image under Linux and then insert that same image into the Torch emulator.
And so the solution was another trip into Python.
Rather than modify the emulator or rebuild the Linux kernel (ain’t nobody got time for that), I created another python script that converts the relevant free-block structures between the two layouts.
For Torch-formatted images, it changes the linked free-list blocks from a 32-bit count to the 16-bit count expected by Linux. And for images being returned to Torch, it performs the reverse conversion. Everything else, the actual files, directories and inodes are all left untouched.
And to make sure there’s no unfortunate data loss (due to almost certain bad programming or operator error), the utility creates a separate output file rather than modifying the original image.
To convert from Torch to Linux:
python3 torch_sysv_convert.py --from torch testdisk.img testdisk_linux.img
The resulting converted image can then be mounted under Linux:
sudo mount -t sysv -o loop testdisk_linux.img /mnt/flop
Files can be then be copied on to it normally.
Once finished, the image is unmounted:
sync
sudo umount /mnt/flop
And converted back:
python3 torch_sysv_convert.py --from linux testdisk_linux.img testdisk_torch.img
The final image can then be inserted directly back into the emulator.
The python script also validates the free-block chain, checks for invalid or duplicate block addresses, and verifies the resulting image. I love python. :)
To make sure everything was working correctly I took a freshly formatted, empty Torch floppy image, converted it to Linux format and mounted it.
This time, Linux reported the expected free space.
Success!
But reading a blank disk is one thing, successfully creating files and then using them in the original operating system is another. So a more comprehensive test was required...
Starting with the same blank Torch image, I copied the Torch Unix kernel onto it. Then I converted the image to Linux format and mounted it on the laptop. Next, I created two additional files, each 100 KB in size. That forced Linux to allocate a decent number of blocks and update the filesystem's free-space attributes.
After un-mounting the image, I converted it back into Torch format and inserted it into the emulator.
And everything worked. Perfectly. The files were present, the filesystem mounted correctly, and the space allocation was as expected.
It does add an extra step in the conversion of disk images but at least I know what the heck was going on. I don’t like not knowing why something is the way it is.
What did we learn from all this?
Firstly, the emulator's floppy disk implementation appears to be doing its job correctly. The disk image contained the expected filesystem information, and the free-block chain was coherent and intact.
Secondly, just because two operating systems recognise the same filesystem doesn't necessarily mean they interpret every internal structure identically. This is apparently important when dealing with historical Unix implementations. Presumably, in the early days different vendors introduced their own variations, just to annoy vintage computer enthusiasts 40 years later..
Thirdly, filesystem corruption isn't always what it first appears to be. In this case, the symptoms lead me down the path of a damaged allocation structure, but the information was actually still present and consistent and it was just a misunderstanding of the block list values.
And finally, a relatively small python script can solve a compatibility problem that would otherwise have involved me trying to modify the Linux kernel (bleurgh) or changing the emulator (not necessary as it turned out).
Sometimes I wonder whether retro computing is a hobby or a particularly elaborate form of self-inflicted punishment.
And the floppy image that started all this? It was both full and empty. It just depended on which operating system was observing it…
(Insert Schroedinger joke here...)
