When deciding to multi-tenant on the cloud, please do not put services that depend on each other on the same auto scaling group. It’s just a recipe for disaster.
So, if you’ve decided against all my advice and reasoning not going multi tenancy here, then for the love of God don’t put a service that rely on another in on the same auto scaling group.
The reason for this is simple, Code Deploy does not have an ordered list (nor should it) when deploying applications on a newly spun instance. Now, if you’ve done the sensible thing about connecting all of them via a load balanced DNS; then you’ll be safe(r).
I was investigating a problem earlier this morning about an instance in a perpetual recycle (scale up, kill self, scale again) thinking it was the classic 5 minute multi tenancy timeout issue. Lo and behold I found that it was an auto scaling group of one (it’s a test environment), and as part of our installation script we check that it can talk to all its dependencies. Of course, this dependency has by that time been destroyed and by chance has not spun up “yet”. That’s several thousand pounds worth of lesson.
I’m anticipating questions about containers. Yes, containers will resolve a lot of the issues we’re facing. Unfortunately we’re a .net shop and containers are still extremely immature. Furthermore, we’re talking about a mixture of new and very, very old components (some with shelf life for over a decade and half).
So kids, moral of the story? Don’t do multi-tenancy. If you do, then read all my whinge about it so you’ll know what NOT to do.
logically illogical 