SM4 area very poor performance

[ccarmock]

Is anyone else here int he SM4 area havign bad performance this weekend?

I have a 50 Mb/s business connectiont hat has been all but unusable. Initially business faults suspected a duff Business Superhub and were going to send the on-call engineer to me yesterday evening.

This was after they tried to move me from one upstream to another, which didn't help.

I then got a call back to say that one of his colleages has seen very high network utilisation and this is almost certainly the problem.

This is todays Thinkbroadband graph:-

http://www.thinkbroadband.com/ping/share-thumb/c94863f9c99f520f47cfd33b285a1aee-04-03-2012.png (http://www.thinkbroadband.com/ping/share/c94863f9c99f520f47cfd33b285a1aee-04-03-2012.html)


This is yesterdays:-

http://www.thinkbroadband.com/ping/share-thumb/4c2135df9cdf6b9aa674d0faa1d44705-03-03-2012.png (http://www.thinkbroadband.com/ping/share/4c2135df9cdf6b9aa674d0faa1d44705-03-03-2012.html)

[craigj2k12]

holy s.............. -uperhub

and I thought the business package got prioritised!

proves one thing though - VM are able to manually switch upstreams, just need to get a tech with half a brain. Is the business support team different to the normal one?

[ccarmock]

As far as I am aware there is no prioritisation of business traffic. The SLA is different (24 hours) and they offer static IP addresses and a different support team.

Yes he switched me to a different upstream while I was on the phone. The Superhub didn't change the 'channel' associated with the upstream, but the upstream power level did change.

The business support team is different - all support is handled in the UK and often they will get you through to an engineer that will log in to the CMTS right there and then and check things out for you.

Usually performance is good - but something has happened recently that has totally killed it durign evenings & weekends. 100 Mb/s has just become available in my area so maybe the torrent addicts are at work.

[Andrewcrawford23]

holy s.............. -uperhub

and I thought the business package got prioritised!

proves one thing though - VM are able to manually switch upstreams, just need to get a tech with half a brain. Is the business support team different to the normal one?

buisiness tech support is compeltely serperate i thin they have acces to other tools or can do more than what tech support for residential usere can do, second lien can swithc upstream but they genreally dnt want to do it

[qasdfdsaq]

Just the last of a number of very similar looking graphs this week.

[craigj2k12]

maybe they are shoving everyone in an area onto one upstream to free the other upstream(s) for bonding?

[Andrewcrawford23]

maybe they are shoving everyone in an area onto one upstream to free the other upstream(s) for bonding?

i thought it was they just had to do a lien card change so yeah you might get switched but once the psyhcila card was changed ythey could mvoe oyu back ingition would know the answer to that one better wherever he is

[craigj2k12]

i think you misunderstood what i said

[ccarmock]

Well another evening of unusable business service. Was getting around 0.5 Mb/s earlier, but no service at all right now.

Business faults tell me this is now with the edge network planning team and to call back tomorrow

Hhm we'll see!

[MagicUK]

no probs within sm5 if that helps.

https://www.cableforum.co.uk/images/local/2012/03/55.png (http://www.thinkbroadband.com/ping/share/e3a04604a77c6be15178718980047675.html)

---------- Post added at 18:19 ---------- Previous post was at 18:17 ----------

My Broadband Ping (http://www.thinkbroadband.com/ping/share/e3a04604a77c6be15178718980047675.html)

[ccarmock]

That looks good Magic - hopefully it'll stay that way for you. Still dire here - though it's actually working so far this evening - getting around 4-6 Mb/s on the 50 Mb/s business service. Still with IP edge network planning I'm told.

[ccarmock]

Well Virgin Media Business still haven't addressed the issues in SM4:-


http://www.thinkbroadband.com/ping/share-thumb/405d789172395fb797ccc3846dcd0eba-12-03-2012.png (http://www.thinkbroadband.com/ping/share/405d789172395fb797ccc3846dcd0eba-12-03-2012.html)

They tell me the 24 hour SLA for fix doesn't apply to this... hm selective SLA then!

[leexgx]

not bad that thinkbroadband ping log service, from the way it starts to lag up from 4:30pm looks like overloaded part of the network your on in your area not just SM4 as that cover an wide area (does virgin have any type of QOS on the nodes so it drops speed instead of packets like BT do)

[MagicUK]

just upgraded to 100mb speeds and speeds are still flying.

[ccarmock]

Magic are you also in SM4? Maybe it's just the CMTS I am on.

Does anyone know - is a given CMTS tied to serving customers on a given group of cabinets? I guess congestion could be either on the 'cable' side of the CMTS or it's connection into the Virgin network could become congested.

What I was able to glean from the faults engineer that did the tests for me I am on CMTS nmal18-2-0 and on cable 8/0/0. Would this serve one optical node that would then feed a few cabinets in a small region? I was on Upstream 1 and was moved to upstream 2 as that was slightly less congested, though I believe it's downstream congestion I am suffering. When I can do a speed test I find at peak times I will get around 0.5 to say 4 Mb/s down but around 5.5 Mb/s up. Being on the business service for 50 Mb/s my upstream profile is set to 5.7 Mb/s.

After escalating to the CEO's office after a week of 'no ETA' I am now told that they have escalated this and planning will be complete tomorrow with an ETA to be implemented by Friday.

[qasdfdsaq]

Ignition would know ;)

Based on recent events it would seem a particular CMTS has lots of line cards and lots and lots of gigabit ethernet ports. Making distinctions on the CMTS level aren't really useful. Links from the CMTS to the core are rarely if ever congested, because there is so much more of it than capacity from the CMTS to the end user.

What you really want to look at is line cards and congestion at that level. There may be multiple line cards serving an optical node and subcabinets from there. Not every customer on any given cabinet will be connected to any given line card, a line card's capacity may be spread over several cabinets, and have several line cards serving each cabinet. How granular this actually gets I don't know. But it's not the case where everyone on one cabinet => one set of capacity.

[ccarmock]

Interesting - thanks for that. How is the mapping for a given customer to line card done?

I had assumed that a given line card and port would provide a given set of 4 downstreams (or more when enabled) that all modems would connect to on cabinets served by that port.

Or should I think more in terms that a given port on a line card will generate a set of downstreams and several sets of downstreams map to a given node and cabinets and the modems config files determine which is used?

If this latter way is the method would it not be possible to change the line card a given user mapped to by changing a config file?

[qasdfdsaq]

Just noticed the base ping on your TBB graph - your traffic is taking a very roundabout route, via Manchester. Your base ping should be <8ms. It's only affecting a few routes though, so not a significant part of the bigger problem

[ccarmock]

This is because I have the Virgin Media Business servcie with a subnet of public IP addresses (6 in fact). In order to achieve that the business Superhub (same hardware different firmware) forms an L2TP tunnel into the VM network so that it can obtain truely static IP addresses. This means that all traffic has to traverse this tunnel and breakout from the core of the network.

So I tarcert to www.bbc.oc.uk looks a bit different on this service:-

C:\>tracert www.bbc.co.uk

Tracing route to www.bbc.net.uk [212.58.246.94]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms gw [10.10.1.1]
2 46 ms 61 ms 48 ms brad-bam-1.network.virginmedia.net [194.145.148.
188]
3 42 ms 37 ms 42 ms brad-core-1b-ge48.network.virginmedia.net [213.1
05.159.42]
4 49 ms 84 ms 18 ms brad-core-2b-ae2-0.network.virginmedia.net [212.
43.163.237]
5 26 ms 43 ms 35 ms leed-bb-1b-ae6-0.network.virginmedia.net [81.97.
80.121]
6 55 ms 38 ms 51 ms leed-bb-1a-ae0-0.network.virginmedia.net [62.253
.187.185]
7 32 ms 70 ms 69 ms nrth-bb-1b-as2-0.network.virginmedia.net [62.253
.185.101]
8 34 ms 23 ms 31 ms tele-ic-4-ae0-0.network.virginmedia.net [62.253.
174.18]
9 43 ms 64 ms 33 ms pos6-1.rt0.thdo.bbc.co.uk [212.58.239.237]
10 * * * Request timed out.
11 68 ms 94 ms 32 ms ae0.er01.cwwtf.bbc.co.uk [132.185.254.93]
12 29 ms 39 ms 63 ms 132.185.255.165
13 102 ms 88 ms 32 ms bbc-vip015.cwwtf.bbc.co.uk [212.58.246.94]

Trace complete.

Notice the first hop after the local router isn't the local CMTS at all but the distant end of the L2TP tunnel.

I agree though that the base latency is higher than I hoped on this service.

[qasdfdsaq]

That's not it though - I'm tracing to the raw IP (the one you PM'd me) of the Superhub itself - from one location (my NL datacentre) it takes the following route:

traceroute to <snip>, 30 hops max, 60 byte packets
1 hosted.by.leaseweb.com (95.211.44.252) 0.494 ms 0.565 ms 0.609 ms
2 te4-3.sr1.evo.leaseweb.net (85.17.129.209) 0.592 ms 0.634 ms 0.735 ms
3 te5-2.sr2.evo.leaseweb.net (85.17.129.194) 0.528 ms 0.606 ms 0.708 ms
4 be12.crs.tc2.leaseweb.net (62.212.80.73) 0.696 ms 0.921 ms 0.902 ms
5 195.69.145.253 (195.69.145.253) 11.247 ms 11.699 ms 11.209 ms
6 manc-bb-1a-ae9-0.network.virginmedia.net (62.253.188.157) 20.775 ms 20.528 ms 39.580 ms
7 popl-bb-1b-as3-0.network.virginmedia.net (62.252.192.158) 28.069 ms 28.051 ms 18.474 ms
8 croy-core-2b-ae0-0.network.virginmedia.net (81.96.226.6) 28.433 ms 19.399 ms 18.723 ms
9 croy-core-1b-pc200.network.virginmedia.net (195.182.178.85) 29.164 ms 18.789 ms 19.245 ms
10 nmal-geam-1b-pc112.network.virginmedia.net (62.30.242.54) 29.149 ms 28.859 ms 27.891 ms
11 nmal-cmts-18-ge330.network.virginmedia.net (81.96.225.198) 27.873 ms 35.660 ms 35.740 ms
(and then times out)

Whereas a trace from my London datacentre goes the sensible way:
traceroute to <snip>, 30 hops max, 60 byte packets
1 ge-0-0-5-4.cs0.thw.uk.goscomb.net (93.89.92.1) 1.301 ms 2.173 ms 2.011 ms
2 xe-0-0-0.rt0.the.uk.goscomb.net (93.89.91.13) 40.619 ms 40.347 ms 40.243 ms
3 tele-ic-5-xe-020-0.network.virginmedia.net (62.253.174.69) 0.519 ms 0.420 ms 0.309 ms
4 popl-bb-1b-ae12-0.network.virginmedia.net (213.105.159.118) 0.554 ms 0.445 ms 0.346 ms
5 croy-core-2b-ae0-0.network.virginmedia.net (81.96.226.6) 1.231 ms 1.007 ms 0.897 ms
6 croy-core-1b-pc200.network.virginmedia.net (195.182.178.85) 1.624 ms 1.418 ms 1.774 ms
7 nmal-geam-1b-pc112.network.virginmedia.net (62.30.242.54) 2.084 ms 2.022 ms 2.020 ms
8 nmal-cmts-18-ge334.network.virginmedia.net (81.96.225.214) 1.859 ms 1.869 ms 1.744 ms^C

Anyway, I don't think the routing's at fault for once.

[ccarmock]

Hi,
The raw IP will take the normal route (ie no via the L2TP tunnel). I have PMd you the IP that traffic actually uses....

[qasdfdsaq]

Yeah, as I said, the "normal" route without the tunnel is still going via Manchester for some sources... Odd.

In any case I've drawn up a script that pings every (VM) IP in New Malden in turn and calculats jitter on it. It's a dirty hack but I'm going to leave it running overnight to see if I can localize the problem.

[ccarmock]

thanks - I'll be interested in your thgouhts

[qasdfdsaq]

Almost exactly 24 hours on, I'm still half asleep and haven't had a chance to look at it yet - sorry.

[ccarmock]

Things seem much better tonight in this area. I had escalated this to the CEO's office and have a person checking this out for me. She tells me that IP edge planning have been told to prioritise the work and that a planner was assigned to work on this yesterday with a likely implementation by Friday.

Anyway I have noticed that at periods of high congestion the superhub's top blue light will stop flashing and stay on solid blue. Sometimes if the congestion is very prolonged it won't recover and requires a reboot.

Now tonight speeds were fine but the blue light stayed on solid resuting in no internet access. rebooting fixed it. However I don't think there was congestion tonight.

I am wondering if there is a fault with this Superhub.... has anyone else seen the blue light stay on solid?

Also I note that the upstream channel has changed - it was always ch2 before but for the first time it has changed to Ch5 after the reboot.

[qasdfdsaq]

On the Superhub? No - I don't use one. But on the VMNG300 I frequently saw similar behaviour, where all the blue lights would stop flashing and stay steady blue. Usually required a reboot, also occurred more often when the network was more congested (month-wise not hour-wise though)

Oh, and had a look at the ping figures for New Malden a few nights ago (which I probably should have just left running 24/7). 2166 responding IPs.

Mean jitter was 9.8ms, median 9.3, 57 people had 2ms or less. Yours is currently ~9.6, which ain't too bad. Can't remember what it was two days ago though.

[ccarmock]

Interesting - thanks so it maybe that it's a 'feature' of these devices then as opposed to a bug. Seems more frequent these days, and as you say seems to coincide with congestion, though that wasn't the case tonight when it went solid blue.

I also see sometimes when i go to http://192.168.0.1 I get a partial login page - the icons are not there but the username & password boxes are. It will never allow me to login when that happens, but if I want 30 mins I can then login OK, as if something times out.

[qasdfdsaq]

Curious. I don't see it as a feature, I see it as my modem crashing... However VM always quietly sidestepped the question when I asked if it was possibly a faulty modem, and always blamed it on the congestion.

Anyhow, some charts for your perusal:

http://qasdfdsaq.com/images/nm-j.pnghttps://www.cableforum.co.uk/images/local/2012/03/28.png
https://www.cableforum.co.uk/images/local/2012/03/29.png
https://www.cableforum.co.uk/images/local/2012/03/30.png

You should probably subtract ~15ms from your pings to account for the fact it's going through Manchester and back
http://qasdfdsaq.com/images/nm-p.png

[ccarmock]

Well I had an email from the person looking into my complaint in VMB today to say relief work was carried out at 9:15 this morning.

I was sent a graph showing number of devices on each of the 4 upstreams on the CMTS I am on. US 0 & 1 were showing 200 users each and US 3 & 4 100 each on average then at 9:15 US 1 dropped to 100 users.

I guess they moved 100 users to a totally different CMTS since neither of the upstreams showed an increase to ofset those moved. They left US 0 at 200 users though.

My issue was downstream congestion though, but VM always seem to focus on upstream for some reason.

So far tonight it is much better though.

[qasdfdsaq]

Difference is downstream channels are bonded and upstream aren't.

Oddly I'm seeing 19ms jitter on your line atm, i.e. more than yesterday, how's your thinkbroadband graph looking?

[ccarmock]

here it is:-

http://www.thinkbroadband.com/ping/share-thumb/26a756b31a9e5316d23f4e3a41ea246b-15-03-2012.png (http://www.thinkbroadband.com/ping/share/26a756b31a9e5316d23f4e3a41ea246b-15-03-2012.html)

Ironically yesterday evening was fine too, so I am still unsure if they have made any true difference. As they made the change at 9:15 I need to give it a few days I suspect

Understood re bonded d/s but when I was testing when it was poor I was getting something like 1.5 Mb/s down and 5.3 Mb/s up, yet they still focus on the upstream....

[qasdfdsaq]

Doesn't look like any big change happened at 9:15, that's for sure.

Thing with the download is as it's bonded, everyone's already load-balanced across all available channels (probably). Hence they can't really do anything about downstream congestion short of installing new equipment (costly, long lead times). Upstream is easier as they can just shift people about.

Ignition would know better, but one of the mods ****ed him off and he's left the forum again :(

By the way, I recently had a graph that looked like yours, and had similar symptoms - but it turns out it could have been upstream congestion all along. Despite getting pathetic download speeds (5-10mbps) and full upload speeds, as soon as they doubled the upload capacity my download speeds shot up to the full 50mbps. Can't really explain it. Something to do with ACK suppression is my guess.

[ccarmock]

This is the graph I was sent:-
http://img210.imageshack.us/img210/1268/vmcmts.th.jpg (http://imageshack.us/photo/my-images/210/vmcmts.jpg/)

It shows a change to US 1 at 9:15, in that the number of users on it have dropped. I guess I might be still on it just that they moved others off to somewhere else.

Certainly things are fine tonight, but they weren't bad last night either, before this change.

Yikes Igni has gone from here - what a loss!

[qasdfdsaq]

ChartDirector (unregistered)??? Lol.

I really need to study on DOCSIS/HFC network architecture before I can say more. You did have a fairly low jitter last night (as I say, just <10ms).

There's a test somewhere that measures upstream and downstream jitter independently so you can tell which direction the congestion is in... I'll post it if I can dig it out. Like I say, even if it *looks* like downstream congestion - maybe it is - it sometimes isn't.

[ccarmock]

This area has *just* had the upstream bandwidth upgraded and 100 Mb/s released here. Also soon to be speed doubled, so this coudl all get worse again!

At the moment there are still 4 downstreams and one upstream.

Sadly the business services are not being speed doubled.