[Maarten Andreas]

Anyone that got their recording back by means of PhotoRec or any of the many commercial or spyware-loaden tools on the internet, they just got really lucky.

Why?

What happened in those lucky cases is the undelete/recovery program recovering a file that existed prior your most recent recording. And this now undeleted file that predates your 0-bytes recording happened to be in the same location as were your latest lost 0-bytes recording was written. Or maybe partially overlap with it.

FAT16/32 operate like a book index, stored as a tree: At the front of the drive there’s a root node that always exists, which points to other nodes, which point to other nodes etc. much like branches of a tree. Now when you delete a file, only the chain of connections from this root node to the node holding the file information is severed somewhere. But even though the branch is no longer connected to the root, it remains there, floating around, until it gets overwritten by something else.

So on deleting a file the node holding the file information and the pointers to where the data is stored is still there, it’s just no longer connected to the root node. Both this node holding information and the data elsewhere pointed to by this node remain on the drive until they are overwritten by the creation of a new file.

It is these severed nodes that recovery programs go look for. They have predictable shapes & sizes, hence an automated process can spot them.

When you hit record on the Zoom, none of these structures are written. The recorder starts writing data to disk but there’s no pointer to it yet. Headers & pointers are all written when you hit the stop-button. But during the recording, the device does write data already somewhere on the disk, but no node describing a start & end exists.

If PhotoRec or something else recovers anything, you just got lucky because it found a set of nodes that were describing a file you have created & deleted before. It has nothing to do with the 0-byte file(s). Just by sheer luck, the data of this prior created & deleted file happened to be stored in the location your new but unfinished/lost recording was written to.

If this prior created & deleted file started at the exact same spot and is equal or greater in length than your lost recording, you get your data back. If it’s shorter or starting points not match, you only recover part of your lost recording.

If you get lucky, your tool recovers a node of a much older and already deleted recording that by sheer coincidence partially overlaps with your latest but corrupted recording. It not actually recovered your lost recording, it recovered an older file that was in the process of being overwritten by your last recording.

I can say with certainty, none of these tools will really work for this particular problem.

For more information & howto recover your data, read here: https://github.com/merethan/zoom-H4n-recovery

[Maarten Andreas]

Hey all,

Took me a while as life got busy, but today I followed up on my promise to deliver a complete story with a complete howto. I put it up on GitHub including the code:

https://github.com/merethan/zoom-H4n-recovery

I’ll update my tool tomorrow or somewhere next week to simplify use.

As described on GitHub is sufficient to get the job done for at the very least all 16-bit resolution recordings. I’m now working on 24-bit, if stuff comes up it will be described & updated in my tool on GitHub. The process however will be similar.

If you’re in need of help, keep an eye on my GitHub page!

[Maarten Andreas]

Guys & gals, I had the same problem and I figured it out!

My Zoom H4n (firmware v1.90) had its power cord pulled while recording internal mics & line inputs (4 channels), which netted me a file 0 bytes in size. And I recovered it!

The bad news: I can say with certainty that no file recovery program on the internet is going to help you in the event of power loss (net adapter pulled, batteries dead), pulling the card too soon, or any other reason for not having closed down the recording properly (which includes a faulty SD card). Don’t spend your time on it, don’t risk installing spyware and what not. Recovery is a manual process, of which I have not (yet?) dreamed up an automated mouseclick mouseclick done process for. Recovery is a manual process involving you.

The good news: The reason I can say the former with certainty is because I figured out how the thing works, and how to recover data.

Summary (TL;DR):
You probably read this before, but to repeat it once again: If you experience a failure of any kind to gracefully stop the recording, your data is still there, somewhere, on the SD card. DO NOT START ANOTHER RECORDING AFTER YOU HAVE EXPERIENCED YOUR FAILURE! If you’re in the situation where your file shows 0 bytes in size when you put the card in your PC, data is still there, somewhere, provided it was not overwritten by you starting another recording. If such happens, swap out the SD card for another, go figure out the recovery later. Or make what’s called a disk image of the first and only partition (or entire SD card if you can’t figure it out) and get into the disk image later. (More on what a disk image is and why further below.)

For the low price of me getting to keep your SD card and you sending two special beers from your country along, I’ll do it for you. Contact me on chef@edgewanderer.com

TL;DR technical details for the leet geeks:
Zoom H4n firmware formats cards as FAT32, uses 512 byte sectors, 64 sector clusters, writes chunks of data alternating between 16 and 17 clusters in size. In 4CH mode it writes two .wav files, which it interleaves/multiplexes 2×16 clusters (16 clusters for each file), 2×17, 2×16, 2×17 etc. sequentially in one lengthy strip of data. Until a proper stop is done, no pointers in the File Allocation Table where the stream of data begins, no RIFF/Wav header prefixed to the stream; all written on hitting the stop button. So to find the start of the stream load the entire image of the partition (not disk, partition! much easier calculating) into for example Audacity by using the load raw data function (44.1kHz., 16bit, little endian, stereo), find the start of the audio you’re after, and calculate the start point in seconds into a byte offset in the disk image. All recordings start on a cluster boundary, so pick the nearest start of a cluster when you calculated your byte offset. Extract the data from your now precise on the cluster boundary-offset to another file (hex editor, dd, sleuth kit whatever), and demultiplex that with the C++ program pasted at the end. You now have your two stereo .wav files.

The story:
There’s a very sad design choice in the firmware of the Zoom H4n (other models are probably very similar) in that it writes all metadata the moment you hit stop. There’s nothing but raw (random looking) data written to the SD card, until you hit stop. No magic recovery program on the internet can help you, as there’s nothing an automated process can go look for. It will find old recordings you deleted before, but not the one you just lost.

By saying metadata, I refer to any sort of location information in the File Allocation Table (FAT) system or any sort of RIFF/Wav header. I spend a lot of time looking for it, there’s none. Zero. All there is, is a filename in the index, with no data assigned to it, hence it being 0 bytes in size. All the metadata stuff is written when you hit stop, or when the file limit size is reached and a new file is created. Your recording is somewhere on the SD card, but no pointers to it, no header in front of it.

Which means it is very hard to write a computer program to recognize where your data is. Writing such a program would involve it being able to differentiate a voice or music instruments from noise and clicks. And being able to do so at different audio settings (sample rate, resolution, channels). But locating the data is possible by means of utilizing a super flexible & programmable neural network! The key tool employed in this recovery is you yourself, your brain.

But first, some on how the writing process works: As mentioned, the Zoom H4n (mine has fw v1.90) basically dumps raw data from the ADC’s (analog-digital converters) straight to the SD card. But, as I found out, it does so in a predictable pattern. When the device is in 4CH mode, it writes two stereo .wav files. Those two files are written almost like the SD card is a tape-deck, in that it writes a chunk of the first file, a chunk of the second file, a chunk of the first file etc. in a continuous strip. So it writes two chunks (one of each file) continuously while recording. Once you hit stop, it writes to the File Allocation Table (FAT) what chunk belongs to the 1st file, what chunks to the 2nd.

If there’s an idea popping up in your mind now going “so, something somehow even numbers is first file, uneven something must be second file!”, then you are absolutely right.

Before we go into this pattern, a few details on how the FAT32 filesystem works: The Zoom H4n wants SD cards formatted with a FAT32 filesystem and it very much likes to do the formatting itself. Reason for this, is that FAT32 can be used with multiple parameters, but the Zoom firmware wants a specific set of them due to how it writes data to SD card. Also it wants a very specific set of directory names to be there before commencing recording, which it makes during formatting.

The main parameters it cares about are sector size and cluster size. Any storage medium, which includes hard disks and SD cards, is essentially like a tape deck in that it has a start and an end, with places in between. All any storage media can do is save & read what’s called bytes. Zeroes and ones. We call a group of 8 zeroes or ones a byte. A byte is the most elementary unit of data in a computer.

What filesystems like FAT32 do is providing a mechanism to manage this enormous strip of bytes, into what we now know as files and directories. A filesystem functions like an index, for example bytes 956 to byte 3065 are /groceries/shoppinglist.txt, and bytes 5364 to 5631 is /school/todo.txt

In FAT16/FAT32, those bytes are not referred to individually, but by what Microsoft has named secors and clusters. How big they are is configurable when you format a drive/card/thumbstick with your PC. The Zoom H4n wants to format SD cards itself because it wants a sector to be precisely 512 bytes and a cluster to be precisely 64 sectors (which makes a cluster 32768 bytes in size).

When the Zoom H4n is recording 4 channels at 44.1kHz samplerate and 16bit resolution, as mentioned above it writes two stereo .wav files interleaved. It writes 16 clusters of data for the first .wav file, 16 clusters of data for the second .wav file, 17 clusters of data for the first file, 17 clusters of data for the second file, then the cycle start over.

16 clusters is 524288 bytes. When sampling at 44100 times a second, with 16bits (two bytes) of resolution times two because stereo, each second of audio takes 2 bytes * 2 channels * 44100 samples = 176400 bytes. If we divide 524288 bytes by 176400 bytes we find that each chunk (16 clusters) is about ~2.97215 seconds worth of data.

For those of you who already managed to load their disk image into Audacity and listen to their lost recording that way, you noticed the audio cuts back and forth between mics and line-in every 3 seconds or so. Now you know why: You are listening to a strip of stored data which is the two .wav files interleaved (multiplexed).

So, how to get this strip of data and demultiplex it?

I’m a little short on time now so I will edit & expand this section later (and deliver on my promise I would tell you what disk images are), but for now, you’ll have to do with the TL;DR (too long; didn’t read) summaries above. Then use the C++ program I wrote below to cut up your multiplexed stream of data into two files, which you can then load as raw data into Audacity. Find a programmer friend or ask your neighbour if you can’t figure it out! For the low price of me getting to keep your SD card and you sending two special beers from your country along, I’ll do it for you. Contact me on chef@edgewanderer.com

I have to get to work now cya later!

#include <iostream>
#include <fstream>
#include <string>
#include <cstring> // memset()

#define ZOOM_FAT32_SECTOR_SIZE 512 // In bytes
#define ZOOM_FAT32_CLUSTER_SIZE 64 // In sectors
#define ZOOM_INTERLEAVE_SIZE 16 // In clusters
#define ZOOM_INTERLEAVE_SIZE_DEVIATION 1 // Zoom H4n v1.90 alternates between writing 16 clusters of each file, writing 17 clusters of each etc.

using namespace std;

int main(int argc, char *argv[])
{
string infile;
string outfile[2];

// Count of arguments is the program name itself, plus the arguments to it;
// from argv[1] and onward is interesting to us
switch(argc)
{
case 2: // One argument in addition to the program name
infile = argv[1];
outfile[0] = infile + “.m.wav”;
outfile[1] = infile + “.i.wav”;
break;
case 4: // Three arguments in addition to the program name
outfile[0] = argv[1];
outfile[1] = argv[2];
infile = argv[3];
break;
default:
cout << “Usage: [outfile_mics outfile_line] infile” << endl;
return 1;
}

cout << “Demuxing file ” << infile << endl;
cout << “to files” << endl << outfile[0] << endl << outfile[1] << endl;

ifstream interleaved(infile);
ofstream deinterleaved1(outfile[0], ios::out | ios::app | ios::binary); // Input-output stream,
ofstream deinterleaved2(outfile[1], ios::out | ios::app | ios::binary); // out, append, binary

for(uint32_t i = 0; !interleaved.eof(); i++)
{
uint32_t chunksize = ZOOM_FAT32_SECTOR_SIZE * ZOOM_FAT32_CLUSTER_SIZE * ZOOM_INTERLEAVE_SIZE;
if((i / 2) % 2 == 1)
chunksize += ZOOM_FAT32_SECTOR_SIZE * ZOOM_FAT32_CLUSTER_SIZE * ZOOM_INTERLEAVE_SIZE_DEVIATION;

char buffer[chunksize];
memset(buffer, 0, chunksize);

cout << “Writing chunk ” << dec << noshowbase << i; // No endl
cout << ” starting at offset ” << hex << showbase << interleaved.tellg() << endl;

interleaved.read(buffer, chunksize);

if(i % 2 == 0)
deinterleaved1.write(buffer, chunksize);
else
deinterleaved2.write(buffer, chunksize);

cout << “Deinterleaved ” << dec << noshowbase << interleaved.gcount() << ” bytes” << endl;
}

return 0;
}

That’s it for now folks! Soon up on github too. But got to get to work now.