Why you shouldn’t trust device categories

So you’ve finally gotten around to getting a responsive website, good for you! You’ve sat down with a designer and/or a developer and they’ve mentioned something along the lines of a desktop and a mobile view. You’re all excited and check your Analytics to see how both the views are performing. But the data you see in Analytics can not be trusted, and here’s why…

First, let’s define what a mobile device is

Most tools like Google Analytics (GA) categorize devices based on the userAgent. Your userAgent right now is:

 

This data should represent the device you’re using right now. It should show your browser, platform and a whole lot of version information. What GA does, is read this information and decide what kind of device you have. For instance, if the userAgent contains ‘iPhone’, guess what, you’re a mobile device.

There is an obvious pitfall here, and that’s when a new userAgent is introduced. If Google’s classifier is made to look for certain words, and those words are not in your userAgent, the script has no idea what you are viewing the website on. A simple example of this was when the first iPad was introduced; most of the platforms looked for ‘iPhone’, but not ‘iPad’. This still happens, because each platform has their own script to read the userAgent. Results will vary.

So why shouldn’t you trust this categorization?

The minor deviations platforms have while reading the userAgent are not that big of a problem. The big problem is that your website uses a completely different metric to define if users are on a mobile or dekstop device. Your website probably uses breakpoints; certain specific widths that trigger certain views. Most of the popular frameworks like Bootstrap use this method, and it’s gaining adoption as we speak.

You can easily test your breakpoints by resizing your (desktop) browser window and watching content appear, disappear and generally jump around. You’ll also see your mobile view if you make your window small enough. If you’ve been paying attention, you’ve just noticed the problem.

Breakpoints don’t care about devices

You’ve just seen your sites mobile view on your desktop. You as a user saw the mobile view, but in Google Analytics, you’re a desktop-user. This works both ways, because there are also larger phones that will show the desktop view, making you a mobile-user viewing the desktop site. This happens because your sites breakpoints don’t care about what kind of device you’re using, they just react to the size of your viewport. You can read my earlier article of viewport metrics here, but this image kind of says it all:

viewport-example

Now you know that your metrics and views don’t line up, you can’t make any significant calls about the device reports you see in your analytics platform. The fun really starts when you turn everything around. Recently, I needed to run an A/B test on an element in Visual Website Optimizer (just an example, most tools use the same userAgent for device classification). The element was a button in the mobile menu, the menu that appears when the breakpoint for mobile is triggered by the viewport. My only option to test this element was to segment the test to ‘mobile users’, based on, you guessed it, the userAgent.

I had no means to test the specific element, because I had no idea if the mobile breakpoint has triggered for a user. To put things in perspective, I created a test to see the differences in device classification and breakpoint triggers. The results:

device-category-defintions

 

  1. There is a small difference in Google Analytics’ definition of ‘Tablet’ versus Visual Website Optimizer’s definition of Tablet. When checking the deviation in Analytics, it appears that VWO hasn’t updated their classifier in a while, because ‘Galaxy Tab’ is set as ‘mobile’, while these are obviously tablets.
  2. This is what you’ve just tried yourself; it’s possible to see a mobile view on your desktop. Practically, this probably won’t happen a lot.
  3. If you have a desktop and a mobile view, there might not be a tablet view. The regular breakpoint to switch to the mobile view is 768px wide, meaning that a tablet in portrait mode is a mobile device, but if you turn it to landscape, you’ve magically changed to a desktop.
  4. This is the same issue as #3, showing the impact of larger mobile devices. Again; the 768px breakpoint is pretty much in the middle of the tablet device category.

I have not mentioned any percentages here, because that will be specific for your audience.

When checking your data in GA, expect a deviation of around 25%.

Just let that sink in for a while.

 

So how do we fix this?

The logical step to find the data you are looking for would be to segment your visitors by screen resolution, but that would still be flawed data. The most reliable method of segmenting your users based on what version of the site they see is to simply tell GA what it is. I’ll show you how to do this using the default Bootstrap breakpoints and Universal Analytics. If you’re using a different system, it shouldn’t be too hard to translate this to your system. Here we go:

Step 1: figuring out what breakpoint you’re seeing

Check out the Bootstrap Grid System, which uses 4 different breakpoints by default; large, medium, small and extra small. Now you could be all fancy and make a script that reads these breakpoints, but for the sake of brevity, lets just use the default pixel widths. When you use the grid system, you’ll always have some sort of container to hold everything. Lets find out how wide that container is with a basic example:

function getBreakpoint(){
  switch($('.container').css('width')){
    case '1170px':
      return 'L';
      break;
    case '970px':
      return 'M';
      break;
    case '750px':
      return 'S';
      break;
    default:
     return 'XS';
  }
}

We can now call getBreakpoint();, which will return the current breakpoint, and thus the correct version of the site the user is seeing, regardless of device. I recommend checking the users breakpoint on page load, on resize and on orientation change.

Step 2: send that data to Google Analytics

If the breakpoint has changed (no point of sending the data otherwise), you want to get that data into Google Analytics. I’d use a custom dimension piggybacked onto an event.

ga('set', 'dimension1', getBreakpoint());
ga('send', 'event', 'metrics', 'breakpoint', getBreakpoint(), {'nonInteraction': 1});

Step 3: create a custom dimension to work with the data

In Google Analytics, set up your dimension. I recommend setting it to session, and calling it only once per session. (Yes there are resizers and orientation switchers, but let’s keep it simple for now.)

Step 4: report!

All you need to do now is segment your visitors in GA by Breakpoint, allowing you to report accurately. Deviation, if any, will only be the set of users that have switched breakpoint during their session, which is way less than the deviation you had when you reported on device categories. Happy reporting!

image credit

Header image: Jeremy Keith on Flickr