Saturday, July 17, 2010

8 out of 10 prefer Google

For some time (as covered in previous blogs) we have been running a project to move our consumer e-mail services over to the Google Apps platform.

New customers have been provisioned onto the Google platform from mid last year, and we are now in the final stages of migrating the last of 6.8m existing customer mailboxes over to Google with the imminent completion of the @ntlworld.com Google migration - after which we open up @virginmedia.com e-mail addresses to existing customers who wish to adopt them and close the project down.

In April we undertook some questionnaire based research of customers who had been affected by the project in order to see how we've done and to make any changes needed for the remainder of the migrations - although clearly we have been tracking calls and feedback to our customer care teams on a daily basis throughout the programme.

We surveyed a statistically significant 975 customers, who were split as such:
  • 14% 'new' customers (who joined after we launched @virginmedia.com addresses for new customers and have only ever been on the Google platform for their e-mail)
  • 31% migrated National customers with @virgin.net e-mail addresses
  • 55% migrated cable customers with @blueyonder.co.uk e-mail addresses
The first question we asked was a general status check as to what people thought of the new webmail - and over 80% of them were positive about it:
Fig 1: Overall webmail rating of new and migrated customers. Click on image for a larger version.

One thing we particularly wanted to test was how the migrated webmail users reacted to the new Google platform, given that it is significantly different from the older interfaces that customers were on before they moved to Google.

Encouragingly three quarters (well, 74%) of them found the Google platform to be an improvement:
Fig 2: Comparison of previous webmail service with Google platform for migrated customers

A big consideration we had in moving customers to Google was the customers who still choose to use client software (such as Outlook Express or Windows Mail) rather than webmail - and that we wanted to minimise the impact to them, bearing in mind that some would still need to make changes in the interim and that all would in the longer term as we adopt the Google client settings.

We put a number of tools in place to minimise this impact, and are encouraged by how few client users noticed the migration - with many of those telling us that it was only because we both e-mailed them beforehand to let them know it was coming and afterwards to confirm their mailbox had been moved:
Fig 3 : Impact of migration on client software users

This is backed up by 9 out of 10 customers (89%) telling us that the detail of the e-mail communication was "just enough" for them and that 70% of users did not need to use the help pages:
Fig 4 : Customer usage of Virgin Media Mail help

We've been doing some usability work to address customers who found it difficult to find the help content, but encouragingly they managed to get by with very few customers needing to contact us compared with our forecasts:
Fig 5 : Actual customer contacts compared with forecasts. Scale removed as commercially sensitive.

We also asked customers if they wanted to share any free-form comments on how their experience has been, and I thought some might be interested in a selection of them:
Fig 6 : Free-form customer feedback about Virgin Media Mail

With the project coming to a close, our team are now working on future developments including improved mobile functionality - which is something our customers have been telling us they would like to see.

I had the privilege of being the commercial owner of the project throughout many of the key timescales, and would like to congratulate and thank the project team on doing such a good job throughout (you know who you all are), and thank my successor Giorgi and Francesca from our research team for the data and the nicely reusable PowerPoint slides :)

14 comments:

  1. It's refreshing to see that all the blood, sweat and tears that this project has required is paying off and that the response is favourable towards the new platform.

    ReplyDelete
  2. Impressive work by project team - especially consdering the scale of the task and importance of raliable email service for users.

    ReplyDelete
  3. Alex is in cloud cuckoo land. You only have to look at the online support forum to see how many problems there have been. We all know about lies damn lies and statistics. Alex's post is based on the results of 975 people out of 6.8 million users!

    ReplyDelete
  4. Hi 'Anonymous'

    We've migrated 6.8m mailboxes, not users - each household can have more than one mailbox.

    I've covered some of the issues we've found and what we've done about them on previous blogs, but overall customer reaction has been excellent with significantly less contacts from them than we had forecast and excellent feedback in the research undertaken.

    If you've got any specific issues am happy to get them looked into for you if you can get me some details.

    Thanks




    Alex

    ReplyDelete
  5. Hi Alex - I'm thinking of signing up to VM. Can you tell me how intrusive gmail is? Will google scrape my mail content, identifying key words in order to advertise using targeting strategies?

    Cheers

    Bob

    ReplyDelete
  6. Bob

    Hi - Customers on our implementation of the Google Apps platform don't have their content scanned for any advertising purposes and there is no targeted advertising deployed on the Virgin platform.

    HTH



    Alex

    ReplyDelete
  7. Hi Alex,

    Obviously, the emphasis of the upgrade is on webmail users - and for them, the new system may well be better.

    However, for everyone that uses an email client (Outlook, Outlook Express, Windows Mail, etc), this upgrade has been a nightmare.

    The forums are full of angry people who's service has been compromised with settings that were working stopping and then working again, plus complete system failure on a number of occassions - most recently this week.

    Added to the fact that several settings can only be changed via webmail (e.g. spam) and you can see why huge numbers of email client users are less than happy at what is being done in the name of something they never want to use (webmail). For them, you've taken a simple working service and made it complicated and unreliable.

    ReplyDelete
  8. Hi Geoff

    Actually the emphasis was on both, with us investing much time and effort on some tools in the network to minimise impact for client software users given that the Google platform needed some different credentials ... and the 81% of client software users that didn't notice the change shows it generally works, as well as most users being happy with the migrations.

    We've had a few platform issues in the last couple of weeks which hasn't helped and have been working on preventing a recurrence of those.

    Not saying we haven't had a few issues in the last couple of weeks as we have, but they've not been representative of the overall experience and generally it's gone very well - as shown by all the metrics we measure, and the survey you commented on.

    Thanks for the comment.




    Alex

    ReplyDelete
  9. Hi Alex,

    I am aware that when you migrated the accounts, you put in place measures to allow people to continue using their original settings and log-ins (without the domain) and standard port settings. That bit did go pretty well.

    But then when you changed the port settings, started using SMTP authenication, SSL and specified that log-ins needed to have the domain as well - well, that's when it started to go Pete Tong.

    If you read the forums, you can see where people had settings that worked one day and not the next (e.g. log-ins with the domain stopped working and then started again, SSL and authentication the same, etc).

    Quite a number of people I know have not yet made the setting changes (so they are still relying on your translation measures). But a lot, myself included, have - and have been 'rewarded' by shakey and unreliable performance.

    There are a number of "quirks" in the new system (such as virginmedia.com addresses having their sent mail appearing in their Inbox too (due to the conversation threading used by Google), which are plain annoying for those using email clients.

    The fact that you can only review spam in webmail (and POP3 email client users don't even get a "you have spam for review" notification) is also irritating.

    As a matter of interest, do you have figures for the split between those who use webmail and those who use clients?

    Currently, I see very few posts in the forums from webmail users and whole bucket loads from email client users. It would be interesting to see the figures for the proportion of users using webmail vs. email clients against the proportion of people with problems with webmail vs. email clients.

    Geoff

    ReplyDelete
  10. Geoff

    Hi, thanks for the note. We do measure client v webmail usage and have stats on that, but I can't share these with you sorry as they are commercially sensitive.

    On the webmail point, it really is the best way to get the best out of the Google platform for an end user in terms of functionality, features and interface and there's no current plans to stop client access - but I would recommend using webmail ... understanding that of course it's your choice :)

    Apologies again for the issues of the last couple of weeks, the project team are re-assessing things right now to learn from the problems we have had.




    Alex

    ReplyDelete
  11. Hi Alex,

    There are a couple of points you make which are rather revealing:

    1) "There are no current plans to stop client access". Current? Is that a politically expedient use of the word, or have there been active dicussions on abandoning client access, which just haven't been taken forward - yet?

    2) "I would recommend using Webmail..." I use Outlook 2007, with calendar, scheduling, tasks, MS Office integration, etc, etc. Please tell me why webmail would be better than that?

    I use Outlook at work too and we also have Outlook Web Access (OWA), but no-one uses OWA unless they have no alternative for some reason.

    Maybe I'm old fashioned, but I have an aversion to trusting my private correspondence and data to a third party, which I can only access as long as I am a subcriber and have a working Internet connection. Too many opportunities for security breaches. You only have to look at the Government to see their record in looking after peoples' data - appalling. Even Google haven't been immune.

    I know that there are reasons for promoting webmail - for a start it makes people less likely to leave... who'd fancy transfering 7GB of data from one ISP to another ;-) Although you don't "currently", you can also use webmail to place adverts in front of people... and so on.

    Overall though, I stick to my view that this change to the email system was done mainly for the benefit of webmail users and those with email clients have "currently" got the rough end of the stick.

    ReplyDelete
  12. Hi again Geoff,

    No, withdrawing client access is not on the radar and hasn't been discussed - just that I can't say what will happen in what is a fast moving world of technology between now and the end of time! There are no current plans to make any changes on this front.

    The point on the webmail usage was around thr advanced search tools, the vastly better UI (in my view), access to chat etc - but as I say it's clearly your choice.

    Thanks for the input.




    Alex

    ReplyDelete
  13. Hi Alex,

    Hmmm... let me see ... The Virgin Webmail UI is better than Outlook 2007?

    Of course, you are prefectly entitled to your view, but I have to beg to differ. It may be better than what was there before (I never used that either), but better than Outlook 2007... Sorry ;-)

    I never use chat on webmail or facebook or several others. I use Skype. It does IM, VOIP and video, works on my PC and on my mobile.. and I can chat to anyone anywhere (not just other Virgin webmail users).

    For email, I just want a simple reliable transport medium - and that is the most important thing. Email is useless if it isn't reliable. You wouldn't use a mobile that dropped calls all over the place would you ... well unless you have an iPhone 4 of course - but you wouldn't want to get me started on Apple ;-)

    Geoff

    ReplyDelete
  14. Of course you can chat with any other Google account holder.

    And we'll agree on the Apple front, I really must do a blog about it sometime!

    Have a top weekend.



    Alex

    ReplyDelete