In this post, I’ll go through some interesting hints around the “performance data reporting”. Even if this is not a fancy or trending topic the solutions proposed address issues that are quite common (and often underestimated) for most performance engineer. Due to the increase of complexity, data sources and therefore KPI over last years, to assess or just to monitor a service (web or mobile application) became a nightmare: find the “rotten apple” is definitely not an easy task.
The two speeches that I picked-up provide different point of view to the “performance data reporting”. From one side there is the pure theoretical & mathematical approach provided by Heinrich Hartmann (Chief Data Scientist at Circonus – ) in Statistics for Engineers. On the other side, we have the customer centric view offered by Simon Hearne (Senior web performance technical consultant at NCC Group – ) in Beyond the waterfall – meaningful web performance visualizations.
One of the most interesting step of the Heinrich speech has been on a quite common problem: how can we provide a synthetic view of the available data without losing relevant information? Or, as he more accurately said – given a bunch of data points how can we:
- characterize the distribution in one or two values;
- have a characterization robust to outliers.
The following charts provide a clear example to this kind of problem: peak erosion caused by aggregation.
In the two graph, the same peak is shown with height 0.08 and 0.03 due to the aggregation applied to chart the same data over different time spans.
Two the solutions presented by Heinrich:
1- Histogram Aggregation
2- Max Aggregation
A second interesting hint provided by the data scientist is about the use and abuse of the standard deviation as typical displacement from the mean value. Std. dev. is popular because everybody learns about it at school and it has extremely nice mathematical properties as:
- 68% of data falls within 1 std-dev of the mean;
- 95% falls within 2 std-dev of the mean;
- 99.7 falls within 3 std-dev of the mean.
Problem is “this is utter nonsense” since all these properties are true only for normally distributed data! When was the last time that you check about your data distribution? Moreover std. dev. is definitely not good to measure outliers.
So how can we represent the displacement from the mean value? Heinrich suggest to keep in mind that we have other options… do you know the mean abs. dev.?
EDIT 01/27/2016: As suggested by Heinrich Hartmann in the comments, the best alternative to std. deviation measures are provided by quantiles (e.g. 99%-quantile) or quantile-based measures like the inter quartile range (span between 75% and 25% percentile).
As anticipated, the speech from Simon Hearne, provides a more customer centric approach, based on the assumption that as performance consultant: our job is to find the best visualizations to support our story; in order to present and share our findings with not technical people in an easy and understandable way.
The following chart provides a Simons analysis of the common data visualization approaches for web performance data.
In his opinion what are we missing is an intuitive way to provide live info about the performance of a whole site.
His solution is the “rum wall” (following chart), a data visualization based on the common google analytics data. Each box of the wall represents a site’s page: the length of the box is the page exit rate, the height is the page views count, while the color is given by the relative performances (resp. time increases from green to red).
A “rum wall” is hence able to show if and where you have to focus your analysis: the page linked to a big red box has for sure a problem!
If you use google analytics in your site you can try the rum wall using the following link: rumwall.webperf.tools …. but don’t forget to thanks Simon for sharing his ideas and tools!
Attend the Velocity 2015 conference in Amsterdam has been an amazing experience. In just 3 days I collected tons of information and ideas directly from the most innovative and smart people active on topics like web operations, performance and DevOps. If you want to build & deliver sites, apps and services that are fast, scalable, resilient, and highly available to be at the Velocity conference is a must!