Update on CloudDS
Here is a progress update to my current main project (we call it 'CloudDS' which stands for cloud data store which is a silly name but it will have to do until we can find a replacement ).
I have been working on the data store component of the service. It has taken at least x4 as much time and effort as I thought it would. A prime reason for underestimating the time requirements is that the initial features list I wanted to implement doubled in size. In addition to that, testing for most of the possible logic paths that could result to failure also took a long time - even if some of that testing was automated, not all of it was and validating results is harder than setting up the test environment.
In such a service, it matters little if most of underlying components fail (I/O and tasks scheduler, garbage collector, cache subsystem, etc) as long as the data management component is not affected. Suffering from a service outage is bad, suffering data corruption and/or data loss is something that has to be prevented by any means necessary.
As it stands, that said component now deals fine with reads and writes, self-healing, caching and performs faster than I hoped it would. The data model is based on BigTable, Dynamo, Cassandra and some earlier prototypes/projects we toyed with in the past. It borrows Cassandra's ColumnFamily/SuperColumn/Column key value representation model. Data are pushed into MemTables and an append only commit log, memtables are flushed into SSTables to disk.
The GCollector merges SSTables whenever required to reclaim space, resolve conflicts and extract a single value out of multiple versions, etc. All operations supported by Cassandra are implemented (query by path, predicate, column names, key ranges, etc ) and CloudDS clients/users will also be able to use a scripting language to describe explicitly down to bytes what they need(i.e give me the first couple of bytes for those values, or gimme a concatenation of those values, etc etc).
Now that that component is out of the way, I can move on to the rest; those are relatively straight forward to implement ( the tasks scheduler and the network I/O subsystems are mostly done ).
Friday, 26 March 2010 9:19 pm
mySQL, noSQL, and Key Value datastores
Monolithic RDBMs are losing ground to key-value data stores, particularly persistent distributed in nature. mySQL mounting problems was perhaps the key reason (pun intended) people looked elsewhere. Google's brilliant engineers realized that a key/value data model can satisfy the needs of almost every class of application that needs a datastore backend.
Key/value datastores are simple to build, easy to understand, easy to optimize, easy to scale. The, now famous, CAP theorem states that it is not practically possible to guarantee consistency, availability and partitioning resilience/tolerance all at the same time; one of those traits has to be sacrificed. Again, most applications really do not require all three to function. The CAP theorem is most likely derived from the Project Triangle mode.
Most web-based applications are built on simple data models. Most web-applications eventually suffer from service capacity and availability issues(i.e scalability woes). It is trivial to scale out(vertically) application logic processing(application servers), HTTP requests processing(web servers, load balancers).
It is not easy to scale out an RDBMS. Some expensive systems(Oracle, etc) provide ways to address those issues (e.g Oracle RAC) but its expensive to deploy them, and most of them rely on a shared everything setup which just doesn't work in the long run. (Shared nothing is really the way to go).
Google released a bunch of papers ( actually, a bazillion of papers ), many of them defining and shaping the development of future related technologies. Namely, the papers describing GFS, BigTable, MapReduce (and of course, the paper the changed everything, "The Anatomy of a Large-Scale Hypertextual Web Search Engine" ) steered everyone to the right direction.
In the datastores domain, Hadoop/HBase, Radix, Cassandra and others, based on BigTable and Amazon's Dynamo papers, all relying on the simple key/value datastore model, are gaining market share - rightly so. Coupled with Memmache and similar services(in-memory key/value stores) they are solving the problems of service capacity and availability. This is a paradigm shift. Its a downhill for heavy-footprint, complex and inflexible datastore systems. They wont go away but will not be such a valuable(pun intended) component in tomorrow's technology landscape.
We are going to gradually migrate from RDBMs - though, we are not relying that much on them nowadays - to a key/value datastore (we are currently building one, also based on BigTable and Dynamo ). If nothing else, those simple systems are both simple and beautiful (for the most part).
Saturday, 13 March 2010 8:14 pm
on Apple's iPad launch : the Aftermath
The iPad was launched yesterday. What was, perhaps, the most hyped and anticipated product of this eon, finally became known as Steve Jobs held it proudly in his hands. These kind of products usually wind up being the recipients of all kinds of crazy expectations. Everyone wants the next-big-thing to do everything, better, in new radical ways and, if that is not enough, they expect even more in the end (in the case of Apple, that comes after Job's "one more thing" statement). All in all, most people are not pleased with the iPad. I am not one of those people. Here is my take on (some of) the 'issues' raised so far.
The name
'iPad' indeed feels wrong. I was expecting it to be called 'tablet' or 'Applet tablet' or something along those lines. Of course, I failed to consider the fact that this would not be a tablet, according to Apple at least. This is not a netbook either. In fact, this is a product that carves a new niche and defines it. So Apple didn't use tablet in the title. I remember when the first Macbook came out. People didn't like the name. They don't mind about it now - in fact they may even come to like it.
Multitasking
The iPad does not allow multiple apps to run at the same time. This made much sense on the iPhone, but would make little to no sense to a laptop or netbook. Well, again, this is neither of those. My gut feeling tells me the reason Apple chose to go with this is threefold.
- The existing iPhone apps are built and executed in a way that just doesn't translate well to a multitasking environment. It is not impossible of course. Apple decided to play safe with this one.
- The RAM on this thing would not allow for a multiple applications to run in parallel efficiently. Lets just say that Apple, again, decided to play safe with this one too.
- Apple's vision for the iPad is very specific. A consumption device that does some things extremely well, one thing at a time.
Cameras
I would have loved it if iPad had a front-facing camera so that I could video chat with my brother. Well, actually, I never use video chat, but that need is valid for enough folks to make it an important omission. Apple has been keen on adding an iSight camera on just about every Mac product it has released. iMacs, Macbooks, the cinema displays, you name it. They love it when people video chat over iChat, use PhotoBooth to go silly and whatnot. What became apparent with the iPhone and even more apparent with the iPad is that Apple, presumably, has a very good set of reasons that led it to (at least for now) make it impossible (one way or another) to do so on this class of devices. It is likely that AT&T and other carriers are to blame here. Apple is giving up some features in exchange for others(better deals with the carriers?). Win some, lose some.
Adobe Flash support
Apple doesn't like Flash. It can't be more obvious than that. They could list a number of technical and semi-valid reasons as to why this is a bad thing, but none of that would matter. I personally couldn't care less about Flash support on Safari, but the vast majority of potential users would, especially the folks in the US where, I hear, Hulu has become the go-to site for all things entertainment there. Unless things go way south for Apple, I don't see it changing its stance on the subject.
On screen keyboard
When the iPhone came out, naysayers and pessimists sure had lots to say about the onscreen virtual keyboard. Nowadays though, people seem to actually prefer those kind of keyboards to the traditional physical ones, me included. Why waste device physical space, weight and looks for a 'real' keyboard, which is there even if you don't want it, when this new virtual keyboard works for you? I have been trying to type using both hands for the past few days. I can type now at least x2 many words/minute than I was able to do so in the past, when I was using just my thumb and I hope, expect, to get better at it. On the iPad, which features even bigger keys, things should be even better for me.
Books and magazines
I wanted an ebooks reader for quite some time now. I was hoping for an ebooks store tightly integrated to iTunes and the iTunes store, the ability to subscribe to magazines (Wired, Edge, ..) and have them delivered to my 'subscriptions inbox' ( with a nice badge indicating new subscriptions count; me and Stelios would sure love that ) and a reader that would provide all benefits PDF readers come with, but with even more thrown in. Well, it won't really work like that on the iPad, at least for now, but this iBooks application sure looks sweet and well done. Apparently, each magazine, newspaper or other content provider will come up with its own solution to the digital content challenge ( NYTimes app demoed was pretty impressive ), which is one way to do it. Apple is playing safe there, again. There is no subscriptions aggregation place/app, or anything like that. Wired will need to build its own app and same will be true for everyone else. I can't wait to see what they will come up with.
All in all, this is v1.0 of a new product that, again, occupies and defines a new category on its own. Recall v1.0 of OS X, the iMac, the iPod, or any other product, produced by any company. Most v1.0 products are there to establish a baseline. Evolution bless them with more feature in later releases. This will certainly be the case with the iPad too. I never needed a tablet device ( my MBP 17" is everything I would 'ever' need ), but I am so buying one for me and Dora ( and for my brother and Dimitris if they themselves wont get one ). Exciting times (ahead).
Thursday, 28 January 2010 8:14 pm
