Wednesday 5 November 2008

To shard or not to shard

I remember back in the day it was simple, HTML --> Server Side Scripting --> DB. Thats it. It was easy.

It seems that as with the gaming industry driving the development of hardware to handle the increased requirements of the gaming software, data is driving the development of frameworks and designs to solve one issue;

"How do we create reliable architecture to scale with increased storage and retrieval requirements". That to me is the bottom line. To further segment those requirements with proposed solutions

Storage
  1. More Disk Space
  2. More Servers
  3. Sharding
  4. Archiving
Retrieval
  1. In Memory caches
  2. Indexing(DB)
  3. Indexing(Application)
  4. Sharding
There are more but these seem to be the most prevelant. The performance and scalablity gains are paid for in complexity, which in turn excacerbates maintenance issues, but I suppose that is the price of progress.
In our teams latest prototype, we have created a prototype which works extremely well using a java suite of framworkes.
Briefly if uses Spring MCV and IOC,Hibernate Core,Hibernate Search,Memcached and MYSQL.
We have a DAO design pattern with Hibernate implementations, A single Hibernate Search(Lucene Index) as well as a Memcached object cache with an indexed mysql DB being used fro persistent storage.
After hammering the web application using JMETER, I have come to the conclusion that it performs very well.
Now enter Hibernate Sharding and the wheels are coming off!

My first blow came from my extensive use of DetachedCriteria.

My design all stems from a base object with a shardId in all object hierarchies, and a constraint ensuring all objects descended from that base object will be in the same shard. My ShardSelectionStrategy takes care of that based on entity type
So I subclassed DetachedCriteria and using a factory based on entity type I can retrieve the shard I need, which is restrictive but works for me.

I am now busy with the FullTextSession and I am running into big trouble.

Basically EventSource isnt supported by shards and a ShardedSessionImplementor and a SessionImplementor are quite different. I can created shard aware classes for all the main players.
  1. FullTextSession
  2. FullTextQuery
  3. AbstractQuery
In a nutshell, I have got to point, as expected where it returns the correct hits from the index, but since I needed to reference a single shard(The master shard) in order to do this, it will only return the objects on that shard. So my result size may be 3 but my list of objects will only contain the objects on that shard.
Now without rewriting the whole implementation, and going through that pain, how do I solve the issue of implementing a distributed fetch?