Does quantifying performance in the Cloud drive you mad?

Both of us were familiar with traditional performance testing. However we also believed that those practices are only partially applicable to understanding cloud computing performance. We agreed that in order to fully understand performance in the cloud one cannot rely on a static report based a snapshot in place and time. The cloud changes too rapidly: New products, new locations, new software updates and new hardware emerge daily and are affecting performance accordingly.We needed to do something quite different: to do it right one would have to measure all instances, all the time and everywhere. Unfortunately that is a lot of testing. Fortunately the cloud also presented the solution to the economic constraint: on demand computing.The result of the collaboration with Rice is the industry’s first comprehensive and continuous price-performance benchmark. Burstorm tests all instances of our target providers, several times at random days during the week, continually (to emulate all the time), at locations in Europe, the USA and Asia (to emulate everywhere).Using a high degree of automation the most recent scope shows the 7 major suppliers representing  153 instance types. Each instance type is tested in 3 locations for a total of 459 test runs per cycle. You might think this is a lot but we currently have 1075 product sets in our database with over 40,000 different products. We are just getting started.