Saturday, October 24, 2009

One down, two to go

This week we hit a major milestone on our project to move our ISP e-mail services over to the Google Apps platform - the completion of the first migration to Google.

To recap, all new Virgin Media customers setting up e-mail with us have been put onto the Google platform (with @virginmedia.com e-mail addresses) for some time, and this week's completion of the first migration means that all National (ADSL) customers with @virgin.net addresses now have their mailboxes on the Google platform too.

This means that customers have much bigger mailboxes, as well as access to features such as chat and search within the webmail interface. Customers have been moved over without any change to their e-mail address (they will be able to choose a @virginmedia.com address at a later date if they wish to) - this is just a change of the platform.

The biggest challenge we have expected (and planned for) within the migrations is how customers will react to what is a very different webmail interface from what they were on previously, and we have focused our help and customer communications around this, as well as training up our technical support teams on what some of the most common queries would be.

So far we have seen what we expected, with some 80% (well, 81.23% to be precise) of the calls we've had classified in this 'customer education' category as it is known internally. In short, this means helping customers out with how to use the service, where things are, what they are called and the likes.

The remainder of the calls have been in a number of different categories - password resets, some of the data integrity work we had to do (which involved some customer communications to anyone affected) and a couple of the minor bugs we found and have plans to resolve ahead of the next migration if not already resolved.

The learning curve for using the Google service seems to be around 2-3 weeks for customers migrating to it (and I recall it taking me a couple of weeks to get used to it) going by the enquiries we've handled (Day 0 = day customer was migrated to Google):
[Click for a larger version, scale removed as commercially sensitive]

We had actually planned for it to take around four weeks, so we're ahead of expectations.

In fact we're ahead of expectations generally - the National migration finished 8 days early and all the work is now happening ahead of the next migration, ex-Telewest customers with @blueyonder.co.uk e-mail addresses. In fact the team are already talking about the National migration in the past tense!

This weekend a number of technical changes are happening, that enable us to move into some final production testing and then a staff trial ahead of the migrations starting, which is due to happen this side of Christmas and run into next year, when the ex-NTL customers (with @ntlworld.com e-mail addresses) will also move over to Google.

Now that we're a good chunk of the way through the 6m mailboxes we are moving to Google and it's going well, it's full steam ahead on the next phase. A big thanks to the exceptional work from our project team, and also to those customers who have taken the time to give us feedback already.

Thursday, October 15, 2009

National e-mail migrations - almost done

As of this morning we're well over 90% of the way through the migration of National (ADSL) customers with @virgin.net e-mail addresses to the Google Apps platform, with all new customers having been on the platform for some time.

As with any migration of this scale - remembering that we have 6m mailboxes overall to move to Google from the legacy platforms - it's had teething problems, but generally it's gone very well and we've had some excellent feedback from our customer care & technical support teams, other staff members and, most importantly, customers.

Given that we're only about 6 weeks in since we migrated the first mailbox and 5 since we started ramping the volumes up, we've got this done very quickly and expect to finish the National migration imminently.

We've attempted to migrate every mailbox at least once so far, and are now working through those that didn't work first time and fixing them before re-migrating them. The reasons they didn't migrate first time are due to a multitude of different scenarios that we are working through one by one, but include:
  • Corrupt mailboxes
  • Password mismatches (data integrity issues we've resolved)
  • Particularly large mailboxes (resulting in mailbox time thresholds being hit, and hence we rollback to ensure people don't lose access to their e-mail for a long period of time)
  • Too many messages in inbox after migrated (a difference in the count of messages on the old platform versus Google, usually due to some spam hitting the mailbox during the migration process)
While none of these are customer affecting and the mailbox is migrated back so the team can fix it if they fail first time, unfortunately we have had a few cases (a low number but we know these are frustrating to those affected) where we have seen some customer impacts, which has in the main affected those with secondary mailboxes on the old platform when the primary has gone to Google.

Our focus has been on fixing these and re-migrating the secondaries as quickly as possible, and the last big batch of these was done last night. I can only apologise for the inconvenience this has caused those few customers affected, as the team have been working all hours to fix these cases when they have happened.

As you can see the faults (in the main calls) we have had in from customers have been well within what we expected, and we're very happy with this given the scale of the migration we have been working on:

[Click on image for a larger version, scale removed as it is commercially sensitive]

The slight rise towards the end of the migration has been as we have tackled some of the more problematic mailboxes and finished our data cleanse programme, which required some customers to change their passwords.

Most people contacting us have been those getting to grips with the Google-based interface, which is quite different to what they were used to previously, and our technical support teams have been helping out when customers do contact us (and more help can be found here).

So where do we go from here?

The next key steps are to finish up the National migration and review it to see what we need to do differently ahead of the next migration in order to improve the customer experience. Next up we have the ex-Telewest customers with @blueyonder.co.uk e-mail addresses, to be followed next year by the ex-NTL customers with @ntlworld.com addresses.

There's also a nifty little tool coming over the next few weeks to help customers who use client software (like Outlook Express) update their settings for what we need them to be in the longer term.

Monday, October 12, 2009

Ukraine vs England – the ISP view

This weekend’s Ukraine vs England game was a landmark as (apparently) the biggest pay per view event on the web in the UK so far, and as I mentioned in my other blog post it seemed to be a good customer experience to me as an end web user.

As an ISP, we had a very strong virtual team working across all of last week to ensure that we had no negative customer impact as a result of extra demand for the network, and I’m pleased to say that we’ve not experienced any issues due to the hard work everyone has done.

We made some changes to optimise traffic management (for both our cable and National [DSL] customers) and prepared our contact centres for any enquiries, as well as having much contingency planning in case should any negative impacts occur.

So what were the impacts?

On our edge network for cable (the bit that customers connect to before the traffic traverses the huge routers that link our network to the outside world), we saw a 9.9% jump in demand in the 2 hour period from 1700 to 1900, when there was most interest in the match:
Comparative edge network downstream network traffic, 3/10 vs 10/10 (scale removed as commercially sensitive)

As you can see, things were busy during the time of the match, but only towards the levels of the normal peak on a Saturday afternoon – so, despite a change to the typical usage pattern for a Saturday the network coped well. The National service also did not experience any problems.

This is also reflected in the enquiries to our technical support team, who didn’t experience any increase in problems reported from customers (although we did have additional resource mobilised just in case) and actually saw lower call volumes than expected during the match – no doubt with people watching the game!

Saturday, October 10, 2009

Ukraine vs England - a good experience

I've been doing some work with a fabulous virtual team across Virgin Media this week to ensure that we were ready for today's Ukraine vs England match, which was shown online only (well, in cinemas also - but the point being it wasn't on TV).

We were all ready so I was able to relax and watch the match, which went very well for this Virgin Media fibreoptic cable broadband user in Surrey anyway!

The picture generally looked very good:
[Screenshot]

[Picture taken of my Acer AL1912 19" monitor from my smartphone's camera]

I watched the game on the 'High' stream (which was streamed at 800Kbps) and didn't spot any buffering, lost connections or anything. It was good on both the standard window and the full screen, and the sound quality was also excellent.

Overall a very good customer experience.

One unexpected side effect was my own version of multi-room, where I was able to watch the cricket at the same time:
[Cricket on my LCD TV on the left, football on 19" monitor on the right]

And that was a very exciting finish, so a good Saturday of sport all round.

I installed the ThinkBroadband Broadband Meter before the match to see what usage would look like, and almost immediately as soon as I closed the window this popped up:
As I installed it about 1 minute before the match, it's safe to say it used up about 1GB of traffic then.

Some more stats and graphs from the ThinkBroadband tool:
It all went very smoothly really, which is less than you can say for Andriy Shevchenko's penalty taking!