Groovy Precompiling

Maybe anybody who is interested in Groovy can make a few comments about this.

The requirement is that we offer functionality that extends what exists in core functionality of a java web application surrounding the persistence lifecycle.

So I added functonality which, will execute a groovy script which has been associated with an event and object type EG preupdate and Widget. The script also needs to be updateble within the session lifecycle.

This works well and adds loads of flexiblity but there will be an obvious performance hit owing to the creation of a groovy shell and execution of the script during the persistence lifecycle.

So my collegue suggested we look into precompilation of the scripts into classes and executing these in a native java environment rather than creating a Groovy shell and running the script, which will need to be dynamically compiled every time the script executes.

This was achieved in essence by creating an instance of the compiler , compiling the script to a class file and storing it somewhere pre-execution, and then when the script needed to be executed, Loading the class (Will always be a subclass of the Script class) and calling the run() method


So to compile we we do this and the resultant output is a .class file.


CompilerConfiguration cc = new CompilerConfiguration();
cc.setTargetDirectory(destdir);
cc.setScriptBaseClass("groovy.lang.Script");

Compiler c = new Compiler(cc);

c.compile(myClassName,myScript);


Now we need to retrieve the class and for that we need a ClassLoader, so I extended the ClassLoader to create a FileClassLoader which finds the file on the file system, creates a byte array from the file, which it passes into the defineClass method of the ClassLoader class definition.

We create the script by creating a new instance of the class and clalling the run() method on it:

FileClassLoader fcl = new FileClassLoader();

start = System.currentTimeMillis();

Class cls = fcl.loadFromFile(DIR + "//DIR//" + classname + ".class", classname);

Binding b = new Binding();
b.setProperty("obj","compiled");

Script scr = (Script) cls.newInstance();
scr.setBinding(b);
scr.run();

The result of this is the as doing this :

GroovyShell gs = new GroovyShell();
gs.setVariable("obj","interpreted");
gs.evaluate(script);

The theory is that every time you pass the script in a string variable to a shell it needs to compile and then execute the script and precompiling will prevent the need for this, thus providing a performance gain as essentially, it not executing a script which needs to be dynamically compiled, but is executing precompiled code.

I tested for performance and since I have the groovy plugin installed in my Eclipse IDE, the script is compiled and cached even when using the shell so there is not real performance gain when I test in the IDE, but will this be a more efficient way of using Groovy on the webserver?

Will update this post when I have an answer

To shard or not to shard

I remember back in the day it was simple, HTML --> Server Side Scripting --> DB. Thats it. It was easy.

It seems that as with the gaming industry driving the development of hardware to handle the increased requirements of the gaming software, data is driving the development of frameworks and designs to solve one issue;

"How do we create reliable architecture to scale with increased storage and retrieval requirements". That to me is the bottom line. To further segment those requirements with proposed solutions

Storage

1. More Disk Space
2. More Servers
3. Sharding
4. Archiving

Retrieval

1. In Memory caches
2. Indexing(DB)
3. Indexing(Application)
4. Sharding

There are more but these seem to be the most prevelant. The performance and scalablity gains are paid for in complexity, which in turn excacerbates maintenance issues, but I suppose that is the price of progress.
In our teams latest prototype, we have created a prototype which works extremely well using a java suite of framworkes.
Briefly if uses Spring MCV and IOC,Hibernate Core,Hibernate Search,Memcached and MYSQL.
We have a DAO design pattern with Hibernate implementations, A single Hibernate Search(Lucene Index) as well as a Memcached object cache with an indexed mysql DB being used fro persistent storage.
After hammering the web application using JMETER, I have come to the conclusion that it performs very well.
Now enter Hibernate Sharding and the wheels are coming off!

My first blow came from my extensive use of DetachedCriteria.

My design all stems from a base object with a shardId in all object hierarchies, and a constraint ensuring all objects descended from that base object will be in the same shard. My ShardSelectionStrategy takes care of that based on entity type
So I subclassed DetachedCriteria and using a factory based on entity type I can retrieve the shard I need, which is restrictive but works for me.

I am now busy with the FullTextSession and I am running into big trouble.

Basically EventSource isnt supported by shards and a ShardedSessionImplementor and a SessionImplementor are quite different. I can created shard aware classes for all the main players.

1. FullTextSession
2. FullTextQuery
3. AbstractQuery

In a nutshell, I have got to point, as expected where it returns the correct hits from the index, but since I needed to reference a single shard(The master shard) in order to do this, it will only return the objects on that shard. So my result size may be 3 but my list of objects will only contain the objects on that shard.
Now without rewriting the whole implementation, and going through that pain, how do I solve the issue of implementing a distributed fetch?