Do you ever have those software ‘solutions’ that you have to babysit day in and day out just to keep them going? You have to coerce them to keep running. “Please, please, pretty please” you say “keep working”. You reboot them because of memory leaks. You make sure servers come up in particular orders. You reboot them because they broke and you don’t know why.
The waves just keep coming in on your technological sandcastle.
You try all of the tricks. You dig moats around your sandcastle, you eventually build an island just for the sandcastle to live on. The ocean doesn’t care, the waves are relentless.
In desperation you start slopping large piles of wet sand on top of where your sandcastle used to be, hey at least the ocean won’t win right? And that is what your solution ends up being, a large pile of slop.
Ok, lets switch analogies. Plate Spinners.
With a small IT department (Ok, probably with any sized department) you typically have a fixed amount of personnel. You can image each one of these people as a plate spinner. Each time you add a new system a member or member(s) of your troop has to start spinning a plate. Eventually you reach the point where you have so many plates spinning that you can’t add any more plates. Your operations have overcome your ability to implement. What can you do?
If you could go back to the very beginning (which is impossible, but good to think of how you should have done something) you could implement or architecture things in such a manner to alleviate or eliminate the necessity of spinning that plate. There is much to be gained by doing things right in the first place. Increasing stability, reliability, and alerting will always pay off in the long run.
I think the smaller your group is, the more time should be spent on increasing your systems stability, reliability and alerting. This will be the only way to survive.
When you are spinning a lot of plates, it’s easy to think that you are just too busy to take any time to increase the stability, reliability, or alerting capabilities of a system. This is the exact moment that you need to do so, you need to spend the time now in order to reap the benefits later. If you need to take extra time, or even work a little over time to learn a new tool, you must do it. Always create as firm of a foundation as possible and continue to improve your foundation. Always take time to “sharpen the saw” instead of just sawing faster with that dull blade.