Optimizing site speed is hard. Like really really hard. So hard that there’s an entire industry around web performance optimization. There’s not one single metric that measures site speed. In fact, there are many. And there’s not one single tool that can evaluate site speed. There’s a bunch of those too.

It’s so hard and yet so important, that some have caught on that you can fool certain speed testing tools into thinking and reporting a site loads faster than it really does. It’s a lot easier to cheat than to do an honest job, like so many other facets of life.

Yeah, you may have come across this before. A speed report from PageSpeed Insights or Lighthouse showing a nice big green score of nearly 100 out of 100. It often looks like this:

Not only do the fake scores do nothing to improve the customer experience or search rankings, they may be making your site even slower, by sacrificing a better loading experience to achieve better scores.

Luckily, Shopify has just released a fresh update to their performance dashboard that helps prevent speed score cheating. In this guide, we’ll go over how cheaters cheat and how to correctly measure site speed with Shopify’s new speed dashboard.

How the cheaters cheat: Emissions cheating for site speed

Remember the VW emissions scandal that happened back in 2015? Volkswagen added something to their cars that detected when their cars were being emissions tested, lowering their NOx output to fool the tests into thinking their cars were meeting standards, when in fact they were 40x worse! Boy golly did that piss off the EPA. Site speed cheating generally works the same way but unfortunately there’s not some governing body looking out for site speed cheaters, at the time of this writing.

The first step for cheaters is to detect when they’re being tested. Some speed testing tools like Lighthouse, PageSpeed Insights, GTmetrix, and WebPageTest give hints that they’re actively testing a website. Speed cheaters use these hints to know when they’re being evaluated.

Once detected, the second step is to fool the speed testing tools. There’s many ways to do this, but the quickest and most common way to cheat speed tests is to simply prevent JavaScript code from loading. There’s other ways of doing it, sometimes foregoing the first step entirely (we’ve talked about this in the past), but this is the most common we’ve seen.

And that’s it. Code is added to a website that detects a speed test, then changes the website temporarily so that it makes some sweet scores. 

Who’s doing this? Honestly it can be anyone: individual freelance developers, “site speed experts”, agencies, theme publishers, and tech companies selling site speed cures. We’ve seen it all, unfortunately. Goodhart’s Law couldn’t be more applicable in this case: When a measure becomes a target, it ceases to be a good measure.

Luckily, there’s better ways to measure site speed to help avoid falling victim to these scams.

How to really measure speed, and avoid scams

Now that we’ve established that tools like PageSpeed Insights, Lighthouse, GTmetrix, and other synthetic testing tools can be cheated rather easily, what’s left to evaluate site speed reliably? Stay with me, because there’s light at the end of this tunnel, I promise!

At last, we arrive at the right way to test speed. The most reliable way, anyway. How can you be sure your visitors are experiencing a faster site? Well, how about they tell you! Today’s modern browsers can report how a website loads and this data can be collected. In the industry, we call this Real User Monitoring (RUM) and luckily there’s free tools available to do this!

The first thing we want to talk about is Shopify’s brand spankin’ new web performance dashboard. We highly recommend this as your first line of defense for Shopify sites. Here’s what it looks like:

How do you access this new report in Shopify? Within the Shopify dashboard, visit the Online Store channel and click on the “Loading speed” metric at the top of the Themes section:

This report contains a few metrics, but the main one we always recommend focusing on first is “Largest Contentful Paint”, which Shopify is calling “Loading Speed” in their new report. We talk about how important this metric is here and this is what we typically refer to as “site speed”. Basically, keep this metric under 2.5 seconds, and you’re doing good! You can read more about all the features of their new dashboard here.

Another tool you can use is the Core Web Vitals report in Google’s Search Console:

Simply look for pages that have “issues” or “need improvement” in this report. If you have a bunch of those, that means your site speed may not be fast (generally look for Largest Contentful Paint issues specifically).

There’s more nuance to that, like you need to have enough visitors to begin with for this report to be generated (it comes from anonymized Chrome sessions) and the data is a rolling 28-day average, so it’s always a bit delayed from where your actual site speed is. If you don’t have enough visitors for this report, then you’re probably addressing site speed issues too early honestly (or maybe you don’t have any visitors because your speed is so terrible?).

There’s one limitation to these reports, which is they both rely on data being collected first. That means you won’t know if any optimizations you made are effective until after they’re launched and you have data collected. If you need to know sooner, you can use a less-easy-to-fool tool like WebPageTest, which is also free. WebPageTest is still prone to being cheated, but it has one BIG advantage that I mentioned earlier: it gives you a full filmstrip and video of how your website loads. That way you can see how the page loads visually, instead of just looking at metrics.

And that’s it! Stick to these tools, and you will have a much better chance of avoiding speed scams. Don’t pay for any 100-speed-score-work until you’ve seen the results in Shopify’s new web performance report or Google’s Core Web Vitals report!