Building a filesystem
I started building a file system today, which will help us solve various problems related to [nfs, data volume, number of files, backup and snapshot functionality, ..]. I am toying with the idea of implementing it directly as a kernel module as opposed to using FUSE and run it on the user-space, however it is quite early to decide about it.
The system will support up to 1T per filesystem instance, because we don't need more than that, per case. In addition to that its going to allow for hundred of thousand of files within the same directory and handle such cases with ease ( a single disk-seek and a binary sort will be more than enough to look up an entry for a given directory, unless its cached in memory ), because some old (legacy) applications we have to support suffer from those kind of bad design decisions. The kind of decisions we no longer make, thank God. In addition to that, its going to be journaled, distributed over various tablespaces ( that is, the filesystem data will be spread over a few files, each file containing chunks of files etc ) and centralized, so that we can mount it on various nodes with ease.
More planned features include snapcloning, rolling backups ( keep track of modified pages and replay them over the previously created clone, that is ), manageability ( operations and information data ) exposed to the various tools through an API and caching.
I was going to go for a Google FS approach, that is break down file data into chunks, have a metadata server which would manage various chunk servers and all that, but we can't afford to spend that much time (+ resources ) on such a project, whereas the unified master + chunkserver approach would make things simpler and would reduce the implementation time by an order of magnitude.
On unrelated note, politics ( and practitioners of this trade ) suck! Almost as much as the tv personas and so called journalists infesting the TV sets, radios and news papers. How most people can stand them and even admire them, its beyond my comprehension levels.
