Clarke's Third Law...
…goes: Any sufficiently advanced technology is indistinguishable from magic. source. He’s dead, by the way. Recently, too.
Yes, so, I’ve kept this quote in the back of my mind for a while now, and I wanted to break it out on a worthy occasion. It’s arguable whether this is that occasion. I’m not sure.
—
Undoubtably you’ve heard about Amazon S3, or Simple Storage Service. If you haven’t, here’s a sufficient summary:
- They let you store infinite data
- They’re always up (well almost)
- It costs, like, nothing.
People have done several cool things with this, I’m sure you can imagine a few.
One of the things this made possible, was a “virtual file system” via FUSE in the Linux kernel. This means you can have a mount point on your computer that reads and writes directly from S3. So you got infinite storage. Cool, eh? You’re limited to transfer speed, of course, but that’s in the details.
At first, I tried s3fs (courtesy of Google), and it does the job. Horrendously slow, too. Which, makes sense too, if you think about it, kinda. When you write ls it has to connect to S3, get a dir listing, and present it to you. For each operation, it has to connect to S3. Or, well, theoretically. What you can do, and what something like Brackup does, is to use an inventory file. Kind of a “manifest” of who’s on the plane, so to speak.
I haven’t looked in any kind of implementation details, but I’d imagine this is what PersistentFS does. I jacked in to the raw S3, and checked out the files, and they’re divided/sharded in a very non-obvious way. s3fs on the other hand, would just store it verbatim.
—
Wait, the impact didn’t come across strongly enough there. Lets try again: PersistentFS! Besides the fact that their website marginally resembles that of MSDN a few years back (I think I even saw it on OSWD), their solution is pure-breed magic. It’s fast like a race horse just after it had its vitamins and all your files are right there when you remount it. It’s truly indistinguishable from the black arts.
Don’t go and ruin it by commenting on how it probably has some kind of smart caching going on, just let me enjoy it for a while. Heck, go try it yourself. It’s a lot of fun, if you’re a nerd. Excellent piece of engineering.