Today I am very happy. Solved a challenge that had been a concern for almost a year.
I needed to find a way of keeping two databases (more specifically two tables) in different machines synchronized with each other. It was simple enough to solve this task when designing the system.
As depicted on the image, the proposed concept was clean to understand and perfectly logical on paper. But in implementation this solution was not clean at all.
Maintaining two databases synchronized was not difficult, the difficulty appeared when the pairs of databases to be maintained starting multiplying and the conscience worries of knowing that even more databases would appear in the future.
When designing system architectures, one can imagine in advance several problems that will be encountered. But when building a brand new system, there exists scarce preparation to get things up and running as intended from a blue print. The proposed solution was too complicated for getting data exchanged. Common sense dictates that the more complex a system becomes, the more prone it becomes to errors and difficult to improve in the future.
I wasn’t happy. The implementation looked “ugly” from all possible perspectives, and to my despair it wouldn’t even scale to the exchange of gigabytes between remote machines as necessary.
After five months of frustrating effort to make the designed concept work in a simple manner, it was time to throw the towel and re-think the attitude to solve the problem.
Gathered my notes about what had been learned from past months, locked myself in the house for two days and only stopped when it was finally working: the new solution is simple and functional.
Too many data containers need synchronism, the server part had to manage each one manually and the whole thing grown into a huge puddle of mess.
The new solution is imaginative and breaks away from a rigorous concept of client-server onto a mixed model of peer to peer data exchange.
It is so simple that required only two methods inside a single class to work as intended and scale to whatever size of data in need of being exchanged up to the tens of gigabytes.
In resume, we start with the traditional client-server approach to discover who is authorized to exchange data, after the initial handshake protocol we ask each database container to “talk” with their counterpart database container on the server side.
This method drastically reduced the administration burden. Each client container will update with new information their server container when it becomes available and vice-versa. We maintain a supervising entity of transactions to provide security while allowing each container to ensure that they correctly pass data onto the other side.
So, I no longer need to care about individual instances of containers and if more instances are added to the bulk, they will follow the same rules and patterns as the previous ones.
Breaking the data exchange problem onto little pieces has helped to break the complexity of this problem into a manageable solution.
Simple solutions are nowhere simple to be found. They take a lot of effort, time and concentration to be found. They truly bear a heavy price to achieve.
My advice is that you should make your best to run that extra mile and find them.
In the end, they are worth the effort.
(screenshot image credits to http://goo.gl/1a40b)
No comments:
Post a Comment