One of our major projects for the next year is the responsive relaunch of all of our sites. As any major site redesign is a risky business, we've decided to start on one of our small sites, applying everything we learn from that site to our main UK websites. The site we have started on is the single-site offering called "Branch309", which has never had any mobile or tablet specific UI treatments applied. This is in contrast to our main UK offering which has desktop, mobile and tablet specific interfaces. This post about reporting mostly applies to the planned main UK offering.
Why is reporting an issue?
Currently our internal reporting systems report the desktop, mobile and tablet sites as separate branches. We then create sales budgets for each of those branches. Using any of the common KPIs (visits, revenue, conversion, etc) we see quite different behaviours between each of those sites. It would therefore seem sensible to write distinct budgets for them. It's a fairly neat picture with a one-to-one relationship between sites and budgets. It is also a fairly obvious question to ask: how is the mobile site doing today? A single responsive site changes this neat picture.
It's one site, so should it be one budget?
We don't think so. The behaviours seen over the different device types are sufficiently varied that a single budget really would paint a very false picture of the true activity happening on the sites. For example, PPC has about half the ROI on mobile devices as it does on desktop. By having it's own budget we can monitor, understand and manage that ROI. We still need the budgeting granularity.
So continue to report as it has always been reported?
Perhaps. However, I'm not entirely sure yet. For one thing, it looks like tablets and desktops are converging in terms of their KPIs, however nobody knows how this will turn out. I think the rejoining of the three sites is an opportunity to look further at our reporting; to look and see whether we are doing the right thing. Here is how I plan to approach it.
Let's take a step back and think about the fundamentals of why we break down the reporting and budgeting in to sub-branches. Apart from the nicety of being able to ask "How is the mobile site doing today?", we do it to try and budget and plan as accurately as possible as that helps run a profitable and effective business. The complete set of ecommerce site activity is sufficiently non-uniform that accurate forecasting would be unrealistic. Instead, it is useful to break the complete set up into subsets which are more uniform. It is then easier to forecast about these subsets. However, it is counterproductive to break the complete set up in to too many subsets, the cumulative error of all the subsets may exceed the error of the complete set; also, if the cardinality of the subset is too low, you're wasting a lot of time budgeting for little benefit.
For me, the next step will be to look at the complete set, segment it by a number of dimensions and look at the uniformity of the subsets produced. I'll also seek to discard any dimensions that produce no meaningful distinction between subsets. The end goal will be to have a small (probably no more than 5) number of subsets which are reasonably uniform, such as would make prediction of future performance more accurate, this accuracy must exceed the accuracy achievable from looking at the complete set.
Perhaps it will turn out that segmenting the activity by the experience they receive, which is centered around device width, will be the right approach. We have designed the site such that those with the smallest screens are assumed to be the most mobile and hence will have a preference for certain features. Note: This is not user-agent based. Our working assumption is that those who have a small tablet are more likely to be mobile than those with a large tablet, even though they may declare themselves to be the same user agent.
I think user context is what is important, not specific devices, and screen sizes (portability) will to a large part inform that context. If your screen size means that it is too big to carry around, then you won't be using it to carry out the tasks that you might want to do if you are mobile. The performance indicators of that experience will be around a stationary user context. The data will guide us, if we only care to look.