It seems that as with the gaming industry driving the development of hardware to handle the increased requirements of the gaming software, data is driving the development of frameworks and designs to solve one issue;
"How do we create reliable architecture to scale with increased storage and retrieval requirements". That to me is the bottom line. To further segment those requirements with proposed solutions
Storage
- More Disk Space
- More Servers
- Sharding
- Archiving
- In Memory caches
- Indexing(DB)
- Indexing(Application)
- Sharding
In our teams latest prototype, we have created a prototype which works extremely well using a java suite of framworkes.
Briefly if uses Spring MCV and IOC,Hibernate Core,Hibernate Search,Memcached and MYSQL.
We have a DAO design pattern with Hibernate implementations, A single Hibernate Search(Lucene Index) as well as a Memcached object cache with an indexed mysql DB being used fro persistent storage.
After hammering the web application using JMETER, I have come to the conclusion that it performs very well.
Now enter Hibernate Sharding and the wheels are coming off!
My first blow came from my extensive use of DetachedCriteria.
My design all stems from a base object with a shardId in all object hierarchies, and a constraint ensuring all objects descended from that base object will be in the same shard. My ShardSelectionStrategy takes care of that based on entity type
So I subclassed DetachedCriteria and using a factory based on entity type I can retrieve the shard I need, which is restrictive but works for me.
I am now busy with the FullTextSession and I am running into big trouble.
Basically EventSource isnt supported by shards and a ShardedSessionImplementor and a SessionImplementor are quite different. I can created shard aware classes for all the main players.
- FullTextSession
- FullTextQuery
- AbstractQuery
Now without rewriting the whole implementation, and going through that pain, how do I solve the issue of implementing a distributed fetch?