The importance of database isolation

This week I've continued to progress on the new project.

One of the worries that has been roaming my head was the isolation of data between different components. On the current concept, components are isolated from each other to some extent but they still share some common resources like the database.

Below is a simple diagram that depicts the difference between shared and isolated approaches to database storage (credits to Mary Taylor @ IBM).

It was a relevant point to ensure that this resource would also stop being shared for the following reasons:

• One faulty component could wreck the database, causing other components to also fail
• 10 components can share a single database, could 100 or even 1000 use the same model?
• No duplicate table names on the database could exist, this issue is aggravated when sharing this resource with other components
• One single point of failure, if the database goes offline then all others will also fail to recover themselves

And some advantages also surfaced:

• Each component can define custom login/passwords to access their database
• Full control over the data that is stored, no other components change the data by accident
• Provides choice between running their HSQL database based on a disk file for optimum storage or stored in volatile RAM for optimum speed
• Each component can now individually store up to 16Gb of data using HSQL

I suspected that transitioning from a shared database model onto an isolated model would be somewhat troublesome, but the implementation was (fortunately) completed in a half a day and the effort was surely worthwhile. The end result passed all test cases and works exactly as intended.

Lesson learned:

When designing/implementing an architecture, do take into consideration the importance of keeping these resources isolated from each component as early as possible. In the long run you are improving the robustness of the system that is being designed.

Editing large text files (above 1Gb of size)

Recently I've began working with considerably sized text files that cannot be edited with the regular text editors such as plain notepad or notepad plus.

To keep on working, I found Cream (http://cream.sourceforge.net). This editor is free and handles fairly well files of larger dimension. It is supported across multiple Operative Systems, I could use VIM but I really prefer to use big round buttons and menus to get my work done.

Albeit the cream editor allows you to change big files, it is not a magic bullet and you should be prepared to wait some time until your changes are saved back to the file on disk.

Having said this, I think the tool is really worth to try out.

:)

Old school programming

This weekend was intense.

While working on my current project, I've slowly reached the conclusion across the past week that many things were wrong on the adopted architecture and that a simpler solution could be adopted.

Reminding myself of the words from a wise software engineer about efficiency of the good enough I could have kept on moving, but it would pain my soul knowing that things were not optimal. If there was a time to make changes, the time was now.

Couldn't afford more schedule slips as seen for the month of the March. So I've decided to plunge myself into a coding marathon to rearrange the code and do things right.

Started of at Friday after work around 17:00 and stopped around 04:00 Saturday morning to grab a snack and some sleep. Saturday and Sunday followed similar hour addiction for code cranking the intended result. The outcome was fabulous!

Not only the new framework is simpler in terms of architecture, it also looks great. The web user interface provides a neat appearance and makes life a lot easier than creating a Swing based interface for each component of the system.

Many other nice-to-have features were also added this way. People can use browsers or third-party developers can create wrappers that make use of the provided web services.

I know that anything coded during a hackathon weekend will haunt me with defects across the next times. That is a risk. But it is still better than living with an architecture that that would only add layers of unnecessary complexity and my own (human) resources are too limited to cope with them.

As result, there is absolutely no need to adopt a Google web toolkit, hibernate, SEAM, JBoss or ActiveMQ COTS for the moment.

The current result can rightfully be called old school programming (compared to current trends of course). And I say this perfectly aware of all advantages and disadvantages that come from this design decision. But nevertheless all that, there is elegance in simplicity and this is the route that I prefer to follow whenever possible.

And speaking of small, the entire framework when including the database (HSQL), message queue, process manager, web interface and remaining components is still using some whooping 3Mb of disk space.

Below is demonstration screenshot of the current status. The page demonstrates two applications. The first one is a simple file browser and the second demonstrates a simple page with text. Each application can host child applications and these are automatically added on the page tabs.

Tough weekend but happy results.

:)

Last night a findbugs saved my life

Do you know what FindBugs is?

It's a neat tool for Java that helps coders find part of your software that are plain redundant or might even run into "bugs" in the future (I prefer to call them "defects" rather than plain "bugs").

I had already used FindBugs in the Eclipse IDE last year, but only recently started using this tool for the NetBeans IDE and now I really appreciate its value.

To install this tool as an integrated plugin of Netbeans is fairly simple and covered widely across the Internet (here's an example).

When I started applying FindBugs on my current project, I noticed that my code was syntactically correct and ran as intended on the test cases, but FindBugs flagged some relevant concerns.

For example, it would flag a warning when new objects were created in a redundant manner or even when a given result might not be handled properly (null handling).

But what I found to be the most useful feature of them all, is the help for coding systems where multiple concurrent threads share information between them.

I mean, a developer already has some notion of practices necessary to prevent thread starvation and other concurrency errors but there exists a good amount of human effort and possibility of human flaw that frustrate your coding progress.

This is where FindBugs came to the rescue. While coding it was consistently able of reporting to me which methods were required to be synchronized in order to prevent thread locking and inherent locking of the outputs expected by other threads.

I would have been able of solving the synchronization issues on my own but having this tool available to help was a real time saver.

If you're still resistant to the idea of using FindBugs after reading all this, do try it by yourself at least once and then let me know what you think.

Take care!

:)

Simple and fast recursive find files for Java

This week I needed a simple routine to find files inside a given folder and respective sub folders.

There are plenty of solutions around the web, but these solutions typically use a second method or even more, instead of a single recursive method.

Below is the code that I've written, it is not perfect nor fully tested but it is simple and fast on my Toshiba R630 Win7 laptop (indexed 152450 files in 30Gb of data under 12 seconds), so I'm happy and decided to share this code snippet with the rest of the world.

It will output an array list composed of File objects. You only need to define where (the starting folder) and maxDeep (the level of subfolders that you want to crawl).

If you have suggestions for future improvement, please do mention them and I'll update the snippet along with placing your name on the credits. I liked this solution but I'm by no means a coding guru.

Have fun!

You find the snippet at this doc: http://goo.gl/pKq0D

--- update 1 (23th March 2011), the code was made shorter and simpler. Please update.

--- Benchmarks

Testing on drive D: with 30Gb of data took an average 12 seconds for 152 450 files

Testing on drive C: with 92Gb of data and 476 789 files started with 829 seconds on the initial scan and on posterior scans dropped to an average of 140 seconds.

Machine of these testings was a Toshiba R630, equipped with 4Gb of RAM, an i7 core and Windows 7 x64 bits.

:)

How to change the default icon on a Java Frame

When you create a Java frame, it usually comes with the default Java icon on the window title.

It is not a bad icon, but you might want to customize that portion after a while.

All you need to do is place this code snippet on the initialization part of your frame:

this.setIconImage(Toolkit.getDefaultToolkit().getImage("./media/network.png"));

I've simply placed a .png image on a folder called "media" on the root of my project.

Have fun.

A day in my life

Well, are you curious about knowing how a day on my life is?

The answer is quite simple:

function starNormalDay()
begin
  // initiate procedure to move outside of bed
While    normalDay()
 begin
  wakeUp()
  getToBathroom()
  getBreakfast()
  goToWork()
  goHome()
  sleep(7 hours);
end
 
end

/*
* We wake up at each day, this is life
*/ 

function wakeUp()
begin
  if isMorning()
   then
      if weekDay is (saturday or sunday)
          then
              while (time.now < 10h00)
                sleep(10 minutes)
          else
              getOutOfBed()
end

You see? Sometimes life is easy to explain. What do you think?

:)

The end of March

The end of this month marks several deadlines that need to be meet.

A new project will be presented, this follows a lot of new troubles and issues that need to be dealt until the deadline expires.

We are still running short on resources. Things are certainly not going as planned and development seems to progress far slower than desired.

Nevertheless these nuisances, the progress made during the past two months is certainly something that leaves me proud to think about. We've managed to create a slim framework that combines essential components for any application to survive the upcoming decades.

We've created a Message Queue to pass messages between applications, a Process Manager to store informations about each applications, a dynamic database storage that can adopt any other technology in the future and a network component that extends the local message queue to interact with remote message queues in other instances in a fully transparent and asynchronous manner.

All of this is certainly not something to undergo with light head. A lot of effort, commitment, rewriting and brainstorming was needed until we were proud of the result.

And the result is nice. We've built a functional enterprise framework that is probably one of the smallest in the world. Using no more than a total of 2Mb and little above 10Mb of RAM while running at full speed.

Unfortunately, the deadline is indeed tight. It takes time to write test cases, to write documentation, to fine tune our code to work as intended, but things still move on.

We've been thinking about the licensing for this new framework and that is one of my biggest headaches. I want this new tool to be free and follow the traditions of all my previous works. I would also like to try out the open source way of doing things.

But several issues concern my imagination:

- If an idea is innovative, how can one prevent large corporations from creating their own "better" versions. I say this because we don't have the muscle or desire to compete against big players but they'd have a galore of fun just looking at the code and throwing all our effort into oblivion

- Open source allows branches and more often than not some wild-coder will just pick the code, create a new branch and then advertise it as better instead of contributing to a longer standing project, effectively fragmenting and eroding any community built around the tool or killing any will of the original developers to fell motivated and follow progress

These are some of the issues that our time constraints urge to solve. Personally, I prefer to look on the bright side of things and adopt an "I don't care" approach. If these are the problems that we cannot find a proper answer, then I guess that it will be better to follow with the same licensing model used with success for Winbuilder.

Completely free but closed source. On the other hand, I would really like to try out the open source for a change. I am just not ready to open my hand after so much hard work over the past 14 months to see it "used" as a commercial product by someone of ill intentions.

Time to think more about the licensing mode.

DoDAF wisdom

The documentation for version 2.0 of DoDAF is a bit lengthy but contains some nice pearls of wisdom.

I personally like this quote:

The central core of DoDAF V2.0 is a data-centric approach where the creation of architectures to support decision-making is secondary to the collection, storage, and maintenance of data needed for efficient and effective decisions.

While some decades ago I'd reckon that the effort should emphasize in achieving better solutions to reach a given result, at our current state we should have learned by now that any new tool will soon grown outdated. So, wise decision to shift the focus in order to preserve the data and knowledge that is gathered in between for the future times.

Decision making tools are easy to find, what is not so easy to find are the data tidybits that help you make an informed decision.

My first MQ

I've finally got around to build my own MQ (Message Queue) service.

Since early January that I've been trying and looking for MQ's already implemented by someone else but they all seemed unadjusted to what I had in mind.

So, to keep things simple I've written a message queue server using a database running on HSQL that is also embedded inside the application.

It was a bit of a marathon, since early design up to implementation it took less than one day and a half but I'm happy that it is available and working as intended.

The advantage of using a built-in message queue is the ease in maintaining several disperse services working with each other in asynchronous manner. For example, A wants B to receive an update. So, A leaves the message on the queue that B will receive whenever becomes possible.

This is handy for cases where a massive amount of information needs to be handled by B from multiple sources, becoming a simple tool that smooths the peak usage from daytime activities that would otherwise require more hardware to cope with the resource demand.

So, the next step is implementing a process manager using the same technique. The overall idea is to implement a publisher-subcriber-alike feature at the running processes. This way, process A is running while processes B, C, D (..) can check the ongoing status of process A since he only needs to update his process status on the process manager.

This way we provide a real abstraction layer between the presentation and business logic of our system, allowing to expand a given application to other platforms and interfaces, let them be web-based, GUI based on even plain command line.

----

Still, I would have preferred to use a COTS or open source project that could save my time and effort to implement these features. There's only so much that one or two developers can do on their own.

I believe that the limitation of working in a small team can also bring advantages, for example, we can't afford to support the bulk associated with typical enterprise-level frameworks and we are constantly reminded that any given implementation needs to be simple or we'll risk not reaching our goals.

Not using enterprise-level frameworks, doesn't necessarily translate to "it won't scale". The idea is to start little and add abstraction at critical locations from the start, so that we can replace components by others with more power (and resource demands) as we progress across the years. So, start small, keep growing as needed.

After all this is done, the final step is moving a bit higher in the ladder and allow connectivity between several clients on a given network using the same application. Let's move forward.

:)

4th semester MSE, the End Of Semester Presentation

This is the Fiber team video at the final EOSP session that was recorded during last December.

The video showcases our MSE project that was developed during the program, along with some reflections and decisions made along the way.

You can view this presentation online at http://goo.gl/c2ImF

The video quality is not top notch, but considering how difficult it was to get all sort of different equipment working together, I already feel happy to see we managed to get the session recorded.

Hope you enjoy the presentation.

Creating test cases

Recently I had to brush up my memory of how test cases for Java are created.

By default, using JUnit is a simple and straightforward way of creating a test case that either validates your code or assures that things are working as you need.

To get started, I recommend looking at the following (quick) tutorial: http://www.cavdar.net/2008/07/21/junit-4-in-60-seconds/

It's from 2008 but nicely explained and still up to date.

:)

Inaccuracy on LZX documentation

A few more days of debugging passed.

I really appreciate to have documentation available from Microsoft so that I can give somewhat correct names to the binary structures.

But progress moves slowly as the available pages only provide twitter-style sentences to explain how each block tie together and even these are sometimes misleading. Nevertheless, I can't complain much as progress is slow but not frozen.

Right now I've reached to a point where an implementation of the Microsoft LZX-2 algorithm is required to compress data. Looking at alternative implementations made by others such as the WINE team, I found this funny comment:

/* LZX decruncher */

/* Microsoft's LZX document and their implementation of the

* com.ms.util.cab Java package do not concur.

*

* In the LZX document, there is a table showing the correlation between

* window size and the number of position slots. It states that the 1MB

* window = 40 slots and the 2MB window = 42 slots. In the implementation,

* 1MB = 42 slots, 2MB = 50 slots. The actual calculation is 'find the

* first slot whose position base is equal to or more than the required

* window size'. This would explain why other tables in the document refer

* to 50 slots rather than 42.

*

* The constant NUM_PRIMARY_LENGTHS used in the decompression pseudocode

* is not defined in the specification.

*

* The LZX document does not state the uncompressed block has an

* uncompressed length field. Where does this length field come from, so

* we can know how large the block is? The implementation has it as the 24

* bits following after the 3 blocktype bits, before the alignment

* padding.

*

* The LZX document states that aligned offset blocks have their aligned

* offset huffman tree AFTER the main and length trees. The implementation

* suggests that the aligned offset tree is BEFORE the main and length

* trees.

*

* The LZX document decoding algorithm states that, in an aligned offset

* block, if an extra_bits value is 1, 2 or 3, then that number of bits

* should be read and the result added to the match offset. This is

* correct for 1 and 2, but not 3, where just a huffman symbol (using the

* aligned tree) should be read.

*

* Regarding the E8 preprocessing, the LZX document states 'No translation

* may be performed on the last 6 bytes of the input block'. This is

* correct. However, the pseudocode provided checks for the *E8 leader*

* up to the last 6 bytes. If the leader appears between -10 and -7 bytes

* from the end, this would cause the next four bytes to be modified, at

* least one of which would be in the last 6 bytes, which is not allowed

* according to the spec.

*

* The specification states that the huffman trees must always contain at

* least one element. However, many CAB files contain blocks where the

* length tree is completely empty (because there are no matches), and

* this is expected to succeed.

*/

http://source.winehq.org/source/dlls/itss/lzx.c

Funny because a decade has passed and still we see specifications for other formats to contain lapses, mistakes and misplacements on official docs from the MS corporation. Even the patent claim they made for the WIM specification contains inaccuracies.

Would be nice to see things change.

Inaccuracy on WIM documentation

For those brave souls in the future, trying to interpret the WIM headers using the documentation provided by Microsoft: there is a typo on the declaration of data structure for RESHDR_BASE_DISK.

Where one reads:

typedef struct _RESHDR_BASE_DISK

{

ULONGLONG ullSize;

BYTE sizebytes[7];

LARGE_INTEGER liOffset;

}

It should actually be read as:

typedef struct _RESHDR_BASE_DISK

{

BYTE bFlags;

BYTE sizebytes[7];

LARGE_INTEGER liOffset;

}

The only difference is replacing ullSize by bFlags. If you don't, the difference is that a BYTE on this case is only sized in 8 bits whereas ULONGLONG is sized in 64 bits.

If you're trying to read the header from a binary file then you'd be stuck with the wrong results.

I had actually noted this detail over a year ago. Now I was looking at this again and had to spent around two days doing the math and printing the hex dump to see why things were not looking right (and get some grey hairs).

So, now I've decided to write it once for all in the blog so that it won't get forgotten again. If it helped you, do let me know.

Happy Christmas!

:)

Fresh start on the Alexa rank

I like watching the Alexa rank to get a notion of how different sites fare in terms of mainstream audience.

Wonko, "the Sane" has a very different opinion about the usefulness of this rank. Yet, it is indeed interesting to observe how we are starting from a fresh new domain that was ranked in 2 million about a week ago and right now is already breaking the top 30 000 barrier.

Not something that we get to see very often.

More interesting to note that over the past weeks our rank had been around 60 000 and that moving to a new domain has also impacted our relevance to search engines as an old domain has far more trust when compared to a brand new one, bringing less visitors from google and the sort.

I'm just glad that this transition is going so smoothly. A lot of things could have been gone wrong and still a lot more in need to be fixed but so far, I'm really happy to see how we are going.

:)

Support for Android and Iphone

One of the advantages from upgrading the board is that we also enjoy some of the recent changes.

Here is one of the nice surprises, support for mobile browsing already available by default:

The Reboot empire.

Boot Land was rebooted, why?

Over the past few months, anyone could note that despite our growth in terms of popularity, page views and visits - we were no longer working true to our founding principles.

Our once peaceful netizens turned the public forums onto a circus stage/arena for proceeding in never-ending quarrels.

While looking back, I see how much energy was wasted in defending or destroying opposing positions between aristocracy members without practical results. Instead of seeing progress, I'd risk stating that we actually saw regress and crisis to install in our development/research projects across 2010.

-------------------------

The Boot Land republic

The flagship weapon of our community, Winbuilder, saw script warlords requesting so many new features to a script engine craftman that seldom times said no. Encouraged with a multitude of feedback and requests, he began an effort to instantiate syntax correctness that would last the following two years.

From my perspective, this craftman was (and is) well intended. But each new version would disrupt scripts coded in older versions of the weapon. This forced warlords to re-train themselves and update all scripts in weaponry stock.

The script warlords were (and are) well intended. They know that winbuilder is one of the strongest weapon on their arsenal. They desire new specialized features that may give them an advantage on battlefields not just in open plains as before, but also in mountains, swamps and tropical environments.

I would have preferred to see other weapons being used together with winbuilder to achieve optimal performance in combat, rather than seeing both the engine craftman and these warlords creating a tool that served their specific situation alone. Since design simplicity was no longer present, we really lost the single most important combat advantage.

The engine kept on convolving to a reality each time farther apart from practical reality in present battle fields. While warlords of conquered domains kept on using older versions and consider other weapons for conquering new territories, new versions became ignored or deeply criticized.

Personal conflicts escalated to unprecedented levels of animosity to find guilt in others.

Rage settled in, projects are removed from public sight, opposing parties verbally attack each other on sight of public movement. This discourages thousands of netizens from joining the public forum. Vengeance, rather than reason, becomes a frequent dish served at public gatherings that are now only frequented by a few surviving senators that observe, but seldom times intervene in fear of retaliation.

Ironically enough, due to the plural investment in many other projects instead of just being a winbuilder centric community, we also saw the Boot Land domain escalate to an unparalleled growth across the boot disk universe at the Internet.

We see for the first time the barrier of 700 000 page views being broken and also celebrate the success of many excellent projects promoted by unaligned brave souls at our community, that put their heart in the work and move the boot disk state of the art by themselves.

Conflicts between aristocratic members stale any decision or course of action for the future.

Our online republic fails.

-------------------------

The reboot empire begins

Situations of this kind are not uncommon to occur at any community of reasonable dimension. I remember clearly a sequence of similar events that took place at 911CD.net some years ago. Bart, the author of BartPE superseded by far in popularity the work of DoctorXP, author of the 911CD project.

DoctorXP stepped down from public activity and BartPE became the defacto tool in coming years. Bart lost interest after some years and his work was left to other initiatives such as Reatogo and ubcd4win. Conflicts soon started to fringe the once peaceful environment at 911CD.net to a stage of pandemonium until everyone was unhappy.

As time passed, our state of conflict starts to resemble each time closer to the one observed at 911CD.

Since nobody was accepted as right by others nor admitted wrong doing on their side, decisions still need to be made and the state of Imperium was declared.

The reboot empire comes to life.

Martial law is instantiated to restore a sense of order amidst the political chaos. Those who cross the line of civilized manners are handled summarily regardless of their rank in society. A sad period but necessary to prevent our public forum from returning onto a public arena.

We live in the age of pax romana.

The goal is clear. We work to rebuild stability, to define the new milestones of expansion for our domain and to ensure that our society regains once again it's own balance to conquer new territories. These decisions will surely not please everyone but we are a breed of fighters.

We reboot.

RawReg included in pwning bootkit

Just of hearing the name of RawReg brings back some really good memories from the attic.

So, it was kind of fun to read IceCube mentioning that it was included by default on the Stoned bootkit, a project described by the author as:

Stoned Bootkit is a new Windows bootkit which attacks all Windows versions from 2000 up to 7. It is loaded before Windows starts and is memory resident up to the Windows kernel. Thus Stoned gains access to the entire system. It has exciting features like integrated file system drivers, automatic Windows pwning, plugins, boot applications and much much more. The project is partly published as open source under the European Union Public License. Like in 1987, "Your PC is now Stoned! ..again".

Peter Kleissner, Software Developer in Vienna

The project can be found at http://stoned-vienna.com/

-------------

Well, this certainly brought me back good memories when I didn't worried about the integration of enterprise applications and their survivability in the long term.

Here is a screenshot of the "about" screen on rawreg, while running under Wine on the Mac OS.

Looking forth to the future, many plans lay ahead. However, I still question every day if there will ever be time and commitment to see them through.

700 000 page views

For the first time since its inception in 2006, Boot Land has reached a new record with more than 700 000 page views served on a single month.

This means that over the course of 11 months, we have successfully grown more than 77% in terms of popularity when compared to last year.

Over the same period of time, the daily consumption of bandwidth on the server has surpassed 100Gb per day while also reducing our RAM usage to little above 4Gb.

Things are looking bright when looking at these numbers and indeed there are many good reasons to be proud about, however, not all news are roses as seen on the case of the winbuilder wars.

The good part is that our community has finally outgrown this state of warfare and other projects are also giving very solid signs of growth as well. Across 2010 we can see grub4dos, Sardu, multiPE, Wimb's work and many other good projects rising to become top tools in this industry. These projects are indeed becoming the defacto tools on this arena.

At this rate, 2011 is already promising to become a really interesting year in terms of community achievements.

:)

To EJB, or not to EJB?

I've found myself asking this question, what advantages does EJB bring?

Googled a lot, found a lot some bulleted lists repeated across many sites, but not so easy to find the real reasons that might drive a person to consider EJB in regard to other options.

Finally, I've found a really a good article from Humphrey Sheil that was written at the year of 2000. It encompasses the fundamental questions that one should ask ourselves while looking at this technology even after a decade has passed.

Below is the introduction:

To EJB, or not to EJB: that is the question.

Whether 'tis nobler in the mind, to suffer

The slings and arrows of outrageous licensing;

Or to take arms against a sea of potential overheads and features,

And by opposing end them? To roll your own: to reinvent the wheel;

No more; and by reinvent, to say, we continue

The heart-ache of low-level systems maintained in-house,

and the thousand natural shocks

That flesh is heir to; 'tis a consummation

Devoutly to be avoided.

You can read the whole article at this location: http://www.javaworld.com/javaworld/jw-12-2001/jw-1207-yesnoejb.html?page=1