On your point about 3, my experience has been that customers tend to have unrealistic expectations about performance of their new site. I have come across a client who wanted their new site with lots of dynamic content and personalisation to offer the same or better performance levels than their old site, which was entirely static (i.e. HTML) and delivered through the web server. I agree with you that it is best to establish the baseline and tune the system until acceptable performance levels are reached or the system reaches its breaking point.

On your point about using live production for performance testing, I will seriously consider using live production for some performance testing in the future. I was fortunate enough to work with a client recently who had invested in a replica test environment that was sized and configured to be the same as production but was offline as it was to be used for DR, so it ended up being used for performance and load testing for new functional releases. It is very unusual though especially in the current economic climate of cut backs!

Thanks again for your comments.

Parv

Very good article! I would add that the pace of delivery and potentially huge scale of the modern web make all of this difficult, but critical. A few comments:

To your opening sentence, we’ve actually found in our work with thousands of sites across every market segment that an alarming number of sites actually DO go live without ANY load or performance testing taking place – up to 75%. This MUST change!

on 3) The challenge here is that clients often really don’t know what their high level or peak load expectations are. Considering the scale of many sites, and the multiplier of mobile access, it’s unclear what “success” will bring. It’s best to work upwards from the baseline (point 4) to understand the upper limits. We suggest establishing the baselines, then iterating higher scale tests as you tune the whole system until, 1) You reach limits that everyone agrees are acceptable, or 2) You do hit a pre-established limit with acceptable performance. From there, test even more, so you really do know where the “top” (breaking point) is!

on 5) Adjusting and extrapolating performance numbers are too often way off or wrong and therefore invalid. Lab testing for scalability loses much credibility with dev and ops team if the numbers are not “real”. This alone helps the case to test in production environments and full scale, either (or both) in maintenance windows and live.

on 6) When you test in prod, you will have NO PROBLEM getting everyone involved during the tests! It makes the sessions incredibly fun and dynamic, too! Many Issues get fixed during the session…not a day or 3 later.

on 9) Of course, and running in the prod env to collect comparative reports of pre and post prod performance.

on 10) This is the key. If the approach or tool does not enable you to have clear and correlated views of all the collected perf metrics, so that all the constituents can react to and implement changes — or completely control the test as it runs (i.e. stop, pause, adjust load, adjust scenarios)…your risks in production testing are higher.

In response to your comment back to Scott, again, it’s the visibility and control you have of the test platform running the test as well as the speed and completeness of the views of all the monitored data that assure you will be able to react to any impact the test has – whether infra, code or database – on real users that would risk their experience. We have been pleasantly surprised at how common it is to see large (albeit sophisticated, and often cutting edge) companies who either have been testing in live production for years, or begin doing so once they see the rapid return and relatively low risk of the practice.

To quote a leading retail customer of ours: “We’ve never had this happen during live prod testing, but even if we crashed the live system in off hours and affected 100 online users, it beats crashing on Cyber Monday with 350,000!”

Brad
(SOASTA CloudTest)

While I understand and agree with your intention on #5, my experience in web dev has shown me that a “replica” never is truly the same as production. There are too many tiny variables to get exactly right. It’s hard to get hundreds of hardware and software configurations to be the same.

Your point is still correct. My suggestion is to always run a performance test after you release code to production. That is the only way you know for sure. I’ve made the mistake of “extrapolation” a few times…it’s a pain when the expectations fall apart and you get a call from the CEO!

Thanks,
Scott