My first Google engine application

Yesterday I've created my first application using Google App Engine.

I've been thinking about the possibilities of using a cloud computing environment for quite some time now. Google brings the advantage that you can use their services for free up to a given threshold of resources.

When I first heard about google app engine, it was some years ago and it was only supported on Python. At the time, phyton didn't exactly rock my world so I just walked away and kept on doing other things.

A few months ago, Java support was announced. My wild guess is that Phyton was not exactly a popular choice amongst many developers but Java is a whole different case.

So, I went on with my trial of their (free) service.

The Java environment that is officially supported by Google is based on the Eclipse IDE, since my preferences on coding fall more on the NetBeans side of life, I've "googled" for this alternative.

It turns out to be a quite simple process of configuration.

The first step is downloading and unpacking the Google App Engine SDK to some folder inside your disk - http://code.google.com/intl/en/appengine/downloads.html

No fancy install involved, just unzip and place it somewhere nice.

-------------------

Then, we open NetBeans. Google support is added as a plugin for the NetBeans IDE. There is a very good page that details these steps: http://kenai.com/projects/nbappengine/pages/NBInstall

When configuring the google plugin, I didn't selected the Hibernate Support and didn't seemed to matter much in either case.

After all said and done, you're only missing to add google app engine as a server inside your environment and this is also a simple task.

Click on the "Services" tab (that is next to "Projects" and "Files" on the right side of the IDE) and right click on the "Servers" item.

All that is missing is add a new server - select the "Google App Engine".

---------------

You should now have everything set up.

To start a web project, create a new project and select "Java Web" --> "Web Application". It will create you a template project that will be saying "Hello world!"

When you're ready to publish the project on the cloud computing environment, just right click on the project and choose "Deploy to google app engine".


Don't forget that you need to have a project already configured on the side of the google app website. After this you should be able to visit your URL and try out the results.

You can also view a full video of all these steps at the following location: http://kenai.com/projects/nbappengine/pages/Home

---------

Hope you have fun with cloud computing, this is a really simple and interesting concept at a very affordable cost.

:)

Implementing a customized Hot folder tracking with Java 1.5

One of the features required for the new project is to track changes made on specific folders.

This is the commonly called "hot folder". In the newer java versions this feature already comes implemented using the OS API whenever available.

On my case, being stuck with Java 1.5 there is not other built-in option other than pooling a folder and check for changes.

However, just checking one folder and not going deeper is still kind of limited so we need to stretch this concept further and use the imagination to check other folders without draining away all of the memory available for the java application nor hogging the CPU resources.

This was a nice challenge, used some of the learnings from the Models of Software Systems class with Pedro Bizarro, guess he'd be proud to see a finite state machine diagram being used to describe this concept. It was indeed a good tool to work out the details before moving to the implementation.

I include a small screenshot of the sketch, some changes were made on the final implementation. It's working really fast and indexing all sub-folders at blazing speed with resort to a database to save the precious RAM memory.




Let's move onto the next challenge! :)

Surviving to java.lang.OutOfMemoryError: Java heap space

Memory leaks happen.

You might be a neat and tidy person that tries to ensure that no waste is made since the start of your project coding but leaks happen.

I fell into this trap.

Was feeling nice and comfortable believing that all my code was behaving nice and using a database was keeping the memory consumption low.

Everything was working perfectly when working with a few hundred files.

However, when dealing with little over 50 thousand files the picture turned out to be grey and the evil java.lang.OutOfMemoryError: Java heap space appears.

I tried my best to pinpoint the evil doers but I couldn't really point my finger at any guilty party since none seemed to blame.

Looking for solutions, most people recommend increasing the size of the virtual machine to support the stress. But as mentioned on the opcode website: http://blogs.opcodesolutions.com/roller/java/entry/solve_java_lang_outofmemoryerror_java, adding more memory only hides the problem, doesn't really bring a scalable solution.

That blog post in particular turned out to be very useful. A good approach is using profiling tools that "police" my application and see how it behaves and which parts are not behaving nicely.

NetBeans is remarkable in that sense.

It already comes with a profiling tool that is built-in and quite simple to use.

Just click on "Profile" --> "Main Project" and follow the directions.

It will launch your project and allow you to track what is really going on underneath the hood while the program is running.

Far better than following your program from the external task manager.

I'm attaching a screenshot so that you can see how it looks like:


My application was bursting at 100Mb, with profiling I was able to track down the reason of the leak: the log entries were consuming too many resources.

Disabling the log entries you can see on the screenshot how it goes well above the previous limit using less than 10Mb of memory to index 100 000 files.

Still a lot of things to improve on my code but this nifty tool sure helps to make it possible.

:)

Java: Getting the localized path to the user's Desktop under Windows

Getting the path to the user's desktop under Windows is tricky if you're not using Java 1.6 and above.

Looking for a solution around the web, I see many people recommending to use a wrap around the Win32 API using SHGetSpecialFolderPath or reading the value directly from the host registry system as suggested here: http://stackoverflow.com/questions/1080634/how-to-get-the-desktop-path-in-java

But I'm not very fond of these solutions and found one that is simple and seems to work fairly well although it doesn't seem to be documented (yet).

Just try this snippet under a Windows machine:

FileSystemView filesys = FileSystemView.getFileSystemView();
File[] roots = filesys.getRoots();
filesys.getHomeDirectory()

And you should be able to get the desktop folder as expected. Under my language, this folder is called "Ambiente de trabalho" so it worked like a charm.

This tip comes from Russ Bradberry: http://stackoverflow.com/questions/570401/in-java-under-windows-how-do-i-find-a-redirected-desktop-folder

:)

Reading the file version from Windows EXE and DLL files.

One of my goals for this week was reading the version of Windows executable files. The code should be implemented in Java and avoid resorting to any native calls.

So, a solution that seemed simple and straightforward would be reading directly the PE header from these files and read the version number for our use.

As many things in life, it's easier said than done.

The troubles started with the code required to handle binary files. I'm using Java and had none of my trustworthy code from previous projects to read files using specific x86 sizes for DWORD, WORD and Unicode strings with a limited size.

To my rescue, I've found the nifty binary file class from Jeff Heaton: http://www.heatonresearch.com/articles/22/page2.html

It's simple and perfect. Far better than the code I used in my previous projects. To handle Unicode strings I read each byte of the Unicode string onto a byte buffer and then use a Java command to convert it properly:

Grabbing the byte sequence of the string:

rgbData = new byte[Data];
for (int i = 0; i < rgbData.length; i++){
rgbData[i] = (byte) bin.readByte();
if ((rgbData[i]==0)&&(rgbData[i-1]==0))
break;
}

Converting to plain string:

String output = new String(rgbData, "UTF-8");

--------------------------

But my biggest trouble was the fact that the file version was not kept inside the PE header itself. The file version for DLL, EXE, OCX, DRV, SCR and similar files is kept inside a resource on the file. (Thanks to TheK on boot-land for helping me sort this detail: http://www.boot-land.net/forums/index.php?showtopic=11890)

So, besides implementing the PE header reading part, it was also necessary to implement all the logic to correctly interpret resources inside executables.

For my luck, this format is extensively documented around the Internet and even MS itself has released official documentation that explains (to some extent) how the structures should be read.

Nevertheless, it took me far longer than initially expected. I had planned for a full day of work and ended up working 3 days to achieve this goal. The code itself is not optimized for speed but for the moment it will suffice the needs of the prototype.

I've tested both with DLL's from the Windows kernel and custom executables from other compilers that had inclusively been modified with UPX - the exe compressor.

It was a good adventure. I've learned far more than what I originally knew about the binary format of windows executable files and this added knowledge might certainly open the "window" for other adventures in the future.

:)