Investigation of Hermes UK reviews on Trustpilot

I recently sent a package containing my old iPhone to my mum using the Hermes UK delivery service. Since Hermes ask what your package contains I stated 'mobile phone'. I don't know whether Hermes staff see this description or not, but after being tracked to one of their depots the package mysteriously disappeared. After "an extensive investigation" Hermes told me "we have not been able locate your parcel". Hmmm... I wondered how it's possible for a delivery company to actually lose a package. Could it be lying somewhere in their depot, hidden out of sight under a pallet? Or did it fall down into the footwell or crevice of a delivery van, and is still there, undiscovered.

Maybe. Or perhaps some thieving b*stard stole it. A few years ago I'd had a brand new iPhone stolen by a delivery driver right outside my office building after he signed for it as delivered himself (luckily this was caught on CCTV as the phone company were insisting it must be in my possession). I decided to check online reviews of Hermes to see if this was an unusual occurrence. Perhaps I was being a little premature and jumping to conclusions.

So I went to the Trustpilot page for Hermes UK. At first glance it looks pretty good. They get a 4 star "Great" rating on 69,000 or so reviews. This would probably be enough to reassure me normally that they must be fine, as surely it wouldn't be possible to get a great rating on so many reviews if there were major problems?



But looking further down the page I noticed lots and lots of 1 star reviews from some very angry customers. One from 'Angela' headed Re previous HERMES review concerning parcels 'disappearing into thin air' caught my eye as it seemed very similar to my experience. Her parcel had disappeared en-route between a collection depot and a hub. Angela had obviously had several previous email conversations with Hermes support staff and was very suspicious of their explanations: 'I ask again: was my parcel stolen while entrusted to Hermes?' she asked.

The recent reviews seemed to be overwhelmingly negative with complaints about parcels not being delivered, lies told about delivery attempts, parcels being thrown over fences, and one reviewer said of his 'beautiful guitar' that 'some neanderthal decided to cut all the tape, break the lock, open the case, snap the neck and only after much complaining was it eventually returned to me in a broken useless state'.

However, as I went though the reviews I noticed that there seemed to be hundreds of reviews per day as I paged back into January. Whereas in February there were typically 20-30 reviews per day, on the 28 January 2019 there were over one thousand! Why would the volume of reviews change so dramatically? Is there some kind of parcel sending festival in January I'm not aware of?

Being curious I decided to investigate. I downloaded over 2900 pages of Hermes reviews from Trustpilot in the terminal and for each page I grepped out from the source a nice JSON object that contained all the reviews on that page. A bit of fiddling around with sed and jq eventually gave me a CSV file containing the author, headline, date published and rating for over 59,000 reviews going back to October 2016.

Great, So first let's look at the number of reviews per month for Hermes UK:

Wow! There are some absolutely huge peaks with over 20,000 reviews published in Jan 2019. Most months have less than 1000 reviews so this is a big difference. Also the peaks seem to be made up of lots of 5 star reviews. Let's look at the percentage of reviews that get different ratings by month to get a clearer view of this:

OK so now we can see that in Jan 2019 nearly 90% were 5 star reviews. This compares to 16% the next month in February and 23% in November 2018. So it looks like this big peak was accompanied by a huge boost in the percentage of 5 star reviews. We see something similar, though not as dramatic, in February and May 2017, and May to July 2018. My theory about a January parcel sending festival looks unlikely now.

Another way of looking at this is to plot average (mean) rating by month along with the volume of reviews:

So we see average ratings dramatically increasing from around 2 in months with a typical number of reviews, up to around 4 or even close to 5 in months when there is a surge in the number of reviews. Let's plot the average rating by volume to see the relationship:

So that looks pretty clear cut - the rating goes up with the number of reviews. It's even more obvious if we plot it on a log scale:

In fact the Spearman rank correlation coefficient is 0.87 (1.0 being perfect correlation and 0 being no correlation).

So there is overwhelming evidence that in months when more reviews are published the average rating improves. If we divide the months into those with more than 1000 reviews and those with fewer we see these results:

Review volume	Number of months	Total reviews	Mean rating	Median rating
1 - 1000	21	10,352	2.1	1
>1000	8	48,549	4.3	5

So if we exclude the months when a very large number of reviews were placed we estimate the rating at 2.1. This would be labelled 'Poor' on Trustpilot which gives a very different impression to 'Great'.

Why would a company suddenly receive so many reviews? Well perhaps there are some months when Hermes goes out of it's way to ask customers to leave a review. It doesn't mean the reviews are faked - but the evidence would suggest the reviews are strongly biased towards 5 stars. Hermes could, for instance, only solicit reviews from customers who received their parcels on time and didn't need to contact their support channels. My own experience is that they did not ask me to review them on Trustpilot. My parcel was never marked as delivered so perhaps my case never got to the stage where a review is requested.

A less charitable interpretation is that Hermes are soliciting fake reviews on an industrial scale when they see their average rating dipping below 4. The average rating in my dataset before Dec 2018 was 2.9 which would have been labelled 'Average' on Trustpilot. After the unprecedented volume of reviews in December 2018 and January 2019 (over 33,000 reviews which is nearly half the total number of reviews they have ever received), their average had been boosted to 3.9.

Either way I think the evidence points to an upwards bias on the true rating of Hermes UK either through 'unintentionally' requesting reviews from satisfied customers or through thousands of fake reviews.

You'll need to make your own mind up but I, for one, will never use them again if I can help it.

If you want to do your own analysis you can download the CSV file I used here: https://nofile.io/f/NtVHPCyq7ZO/reviews.csv