Network problems Tonight ?

IanUK · 04-08-2005, 19:13

Getting very bad lag on browsing tonight to many sites - Traces seem to show why

Target Name: www.3dgamers.com
IP: 209.8.17.25
Date/Time: 04/08/2005 20:07:23
2 6 ms 7 ms 8 ms 13 ms 7 ms 6 ms wapk-t2cam1-a-ge-wan41-112.inet.ntl.com [80.1.170.57]
3 16 ms 7 ms 6 ms 7 ms 8 ms 15 ms popl-t2core-a-ge-wan62.inet.ntl.com [62.255.81.45]
4 7 ms 7 ms 12 ms 7 ms 7 ms 7 ms pop-bb-a-so-330-0.inet.ntl.com [213.105.174.113]
5 285 ms 270 ms 231 ms 271 ms 258 ms 266 ms tele-ic-2-so-010-0.inet.ntl.com [62.253.185.86]
6 8 ms 8 ms 10 ms 9 ms 23 ms 9 ms cr02.ldn01.pccwbtn.net [195.66.224.167]
7 108 ms 106 ms 106 ms 107 ms 109 ms 107 ms www6.3dgamers.com [209.8.17.25]

Target Name: www.ev1servers.net
IP: 207.218.223.142
Date/Time: 04/08/2005 20:12:20
2 5 ms 7 ms 6 ms 6 ms 6 ms wapk-t2cam1-a-ge-wan41-112.inet.ntl.com [80.1.170.57]
3 170 ms 6 ms 7 ms 8 ms 7 ms popl-t2core-a-ge-wan62.inet.ntl.com [62.255.81.45]
4 7 ms 7 ms 6 ms 7 ms 7 ms pop-bb-a-so-330-0.inet.ntl.com [213.105.174.113]
5 13 ms 8 ms 6 ms 8 ms 7 ms pop-bb-b-ae0-0.inet.ntl.com [213.105.172.77]
6 8 ms 9 ms 17 ms 9 ms 9 ms tele-ic-1-so-010-0.inet.ntl.com [62.253.185.82]
7 314 ms 326 ms 318 ms 321 ms 323 ms [212.250.14.62]
8 390 ms 386 ms 389 ms 393 ms 393 ms p10-0.core01.bos01.atlas.cogentco.com [130.117.0.46]
9 409 ms 407 ms 409 ms 417 ms 406 ms p5-0.core01.ord01.atlas.cogentco.com [66.28.4.110]
10 408 ms 407 ms 411 ms 413 ms 406 ms p15-0.core02.ord01.atlas.cogentco.com [66.28.4.62]
11 419 ms 415 ms 422 ms 426 ms 416 ms p12-0.core01.mci01.atlas.cogentco.com [66.28.4.33]
12 432 ms 430 ms 437 ms 438 ms 436 ms p12-0.core01.iah01.atlas.cogentco.com [66.28.4.105]
13 119 ms 122 ms 119 ms 120 ms 120 ms everyonesinternet.demarc.cogentco.com [38.112.12.178]
14 * * * * * [-]

Target Name: www.bbc.co.uk
IP: 212.58.224.81
Date/Time: 04/08/2005 20:12:51
2 9 ms 7 ms wapk-t2cam1-a-ge-wan41-112.inet.ntl.com [80.1.170.57]
3 11 ms 7 ms popl-t2core-a-ge-wan64.inet.ntl.com [62.255.81.65]
4 6 ms 6 ms pop-bb-a-so-330-0.inet.ntl.com [213.105.174.113]
5 232 ms 252 ms tele-ic-2-so-010-0.inet.ntl.com [62.253.185.86]
6 8 ms 8 ms ntl-ge2-8.prt0.thdo.bbc.co.uk [212.58.239.217]
7 13 ms 8 ms [212.58.238.153]
8 8 ms 9 ms www30.thdo.bbc.co.uk [212.58.224.81]

Every one has a massive spike as it leaves NTL it seems.

Anyone else seeing this ?

cheers

Ian

Roy MM · 04-08-2005, 19:17

Browsing normally here N Manchester.

04-08-2005, 19:30

believe me, I am no expert, but IIRC the tele-ic routers favour 'normal' traffic over pings, so hop times will appear to vary with load. I was always told it is the final hop that shows the true speed - and from those traces the first two are poor, but the one to the BBC is spot-on.

But having said that, the BBC trace went through 'wan64.inet', rather than 'wan62.inet', so maybe there is a local routing problem?

like I say I'm a

at this

Code:

08/04/05 20:34:36 Fast traceroute http://www.3dgamers.com/
Trace http://www.3dgamers.com/ (209.8.17.24) ...
 
 3 80.4.47.161      24ms   13ms   13ms  TTL:  0  (nott-t2cam1-a-v128.inet.ntl.com ok)
 4 80.1.79.77       13ms   18ms   84ms  TTL:  0  (nott-t2core-a-ge-wan73.inet.ntl.com ok)
 5 62.253.188.33    21ms   65ms   17ms  TTL:  0  (lee-bb-a-so-200-0.inet.ntl.com ok)
 6 62.253.185.238   22ms   20ms   24ms  TTL:  0  (pop-bb-b-so-100-0.inet.ntl.com ok)
 7 62.253.185.82    26ms   19ms   20ms  TTL:  0  (tele-ic-1-so-010-0.inet.ntl.com ok)
 8 195.66.224.167  162ms   42ms   21ms  TTL:  0  (cr02.ldn01.pccwbtn.net fraudulent rDNS)
 9 209.8.17.24     116ms  123ms  174ms  TTL: 54  (www5.3dgamers.com ok)

IanUK · 04-08-2005, 19:38

Pingplotter is now showing up to 50% packet loss on those traces..others sites affected are:

Target Name: forum.eagames.co.uk
IP: 159.153.177.38
Date/Time: 04/08/2005 20:32:56
2 7 ms 7 ms 5 ms 9 ms 12 ms 8 ms 7 ms 8 ms 7 ms 6 ms wapk-t2cam1-a-ge-wan41-112.inet.ntl.com [80.1.170.57]
3 7 ms 6 ms 7 ms 7 ms 6 ms 11 ms 7 ms 8 ms 6 ms 6 ms popl-t2core-a-ge-wan64.inet.ntl.com [62.255.81.65]
4 7 ms 6 ms 6 ms 6 ms 7 ms 7 ms 6 ms 6 ms 6 ms 7 ms pop-bb-a-so-330-0.inet.ntl.com [213.105.174.113]
5 241 ms 230 ms 225 ms * 208 ms 240 ms * 248 ms 251 ms 270 ms tele-ic-2-so-010-0.inet.ntl.com [62.253.185.86]
6 * 9 ms 8 ms * 7 ms 8 ms 8 ms 9 ms 9 ms 8 ms [212.250.14.54]
7 10 ms 8 ms 8 ms 8 ms 224 ms 13 ms 8 ms 10 ms 8 ms 10 ms gi3-0-sar2.lon.router.colt.net [212.74.64.130]
8 9 ms 9 ms 9 ms 8 ms 9 ms 10 ms 9 ms 9 ms 9 ms 9 ms [80.169.175.2]
9 9 ms 9 ms 13 ms 13 ms 9 ms 13 ms 10 ms 16 ms 8 ms 11 ms [159.153.156.3]
10 8 ms 9 ms 17 ms 9 ms 9 ms 10 ms 10 ms 9 ms 9 ms 9 ms 4v4.easports.com [159.153.177.38]

Target Name: jolt.co.uk
IP: 82.133.85.65
Date/Time: 04/08/2005 20:36:13
2 7 ms 6 ms 6 ms wapk-t2cam1-b-ge-wan41-112.inet.ntl.com [80.1.170.185]
3 7 ms 7 ms 7 ms popl-t2core-b-ge-wan63.inet.ntl.com [62.255.81.189]
4 13 ms 8 ms 8 ms pop-bb-b-so-320-0.inet.ntl.com [62.253.185.69]
5 7 ms 7 ms 7 ms pop-bb-a-ae0-0.inet.ntl.com [213.105.172.78]
6 249 ms 250 ms 266 ms tele-ic-2-so-010-0.inet.ntl.com [62.253.185.86]
7 9 ms 10 ms 10 ms lon1-10.nildram.net [213.208.106.169]
8 9 ms 10 ms 16 ms [84.12.225.130]
9 9 ms 9 ms 9 ms jolt-gw.nildram.net [195.149.20.126]
10 9 ms 9 ms 10 ms [82.133.85.65]

Looks like a problem

__________________

Quote:

believe me, I am no expert, but IIRC the tele-ic routers favour 'normal' traffic over pings, so hop times will appear to vary with load. I was always told it is the final hop that shows the true speed - and from those traces the first two are poor, but the one to the BBC is spot-on.

Yeah, I agree, but in this case my web browsing to those sites is awful (any many others) and download speeds are terrible (sub 20k)

04-08-2005, 19:52

Quote:

Originally Posted by IanUK

Yeah, I agree, but in this case my web browsing to those sites is awful (any many others) and download speeds are terrible (sub 20k)

I just got the same spike on that tele-ic link

Code:

08/04/05 20:40:22 Fast traceroute http://www.jolt.co.uk
Trace http://www.jolt.co.uk (82.133.85.65) ...
 1 10.189.183.254   11ms   11ms   37ms  TTL:  0  (No rDNS)
 2 10.189.183.254   11ms    9ms   32ms  TTL:  0  (No rDNS)
 3 80.4.47.161      12ms   14ms   55ms  TTL:  0  (nott-t2cam1-a-v128.inet.ntl.com ok)
 4 80.1.79.61       12ms   13ms   14ms  TTL:  0  (nott-t2core-a-ge-wan72.inet.ntl.com ok)
 5 62.253.188.33    14ms   15ms   15ms  TTL:  0  (lee-bb-a-so-200-0.inet.ntl.com ok)
 6 62.253.185.238   20ms   22ms   20ms  TTL:  0  (pop-bb-b-so-100-0.inet.ntl.com ok)
 7 213.105.172.78   19ms   28ms   19ms  TTL:  0  (pop-bb-a-ae0-0.inet.ntl.com ok)
 8 62.253.185.86   250ms  282ms  393ms  TTL:  0  (tele-ic-2-so-010-0.inet.ntl.com ok)
 9 213.208.106.169  19ms   32ms   36ms  TTL:  0  (lon1-10.nildram.net probable bogus rDNS: No DNS)
10 84.12.225.130    59ms  120ms   66ms  TTL:  0  (No rDNS)
11 195.149.20.126   26ms   37ms   36ms  TTL:  0  (jolt-gw.nildram.net fraudulent rDNS)
12 82.133.85.65     20ms   33ms   21ms  TTL: 53  (No rDNS)

I have no other issues, when normally surfing, but if you are trying to game it may be a problem ????

can't offer any solutions, sorry ..

Chrysalis · 04-08-2005, 20:12

It is true icmp is normally low priority but it should only show maybe the odd jump not persistant high ms, that usually indicates some sort of problem on the hop.

there was 2 problems with my connection yesterday, the one on the first hop I got fixed by my contact in ntl was routing through a bad router, the other is on the edge of the ntl network.

1 7 ms 8 ms 7 ms 10.8.143.254
2 6 ms 6 ms 7 ms leic-t2cam1-b-ge914.inet.ntl.com [82.3.35.213]
3 6 ms 12 ms 6 ms leic-t2core-b-ge-220-0.inet.ntl.com [82.3.33.138
]
4 8 ms 11 ms 9 ms nth-bb-b-so-230-0.inet.ntl.com [62.253.188.114]

5 11 ms 14 ms 9 ms pop-bb-a-so-100-0.inet.ntl.com [213.105.172.14]

6 275 ms 264 ms 264 ms tele-ic-2-so-010-0.inet.ntl.com [62.253.185.86]

7 14 ms 15 ms 14 ms cr02.ldn01.pccwbtn.net [195.66.224.167]
8 193 ms 190 ms 189 ms 63.218.51.46
9 190 ms 191 ms 191 ms unknown.xeex.net [216.152.255.74]
10 191 ms 219 ms 190 ms 10.10.255.3

pccwtbn also has a problem which affected the rest of the route.

IanUK · 05-08-2005, 07:28

Its ok this morning, I wonder if its load related, we'll see tonight I guess ?

Thanks guys.

Ignition · 05-08-2005, 07:47

Quote:

Originally Posted by Chrysalis

It is true icmp is normally low priority but it should only show maybe the odd jump not persistant high ms, that usually indicates some sort of problem on the hop.

Wrong dude it depends on the model, make, and placement of the router, along with its' config. Foundry boxes especially are bad for it.

Anyway this has been done to death, the tele-ic routers have been upgraded and are now extremely meaty boxes indeed, and are running well within capacity.

So long as what's at the end of the trace is alright then all is fine, there was an issue but it appears to be outside the ntl network, no issues reported with the ntl core or edge last night.

IanUK · 05-08-2005, 20:51

Still bad tonight, seems NTL currently has a very bad connection to Cogent

Target Name: www.filefront.com
IP: 66.117.8.41
Date/Time: 05/08/2005 21:49:52
2 6 ms 7 ms wapk-t2cam1-b-ge-wan41-112.inet.ntl.com [80.1.170.185]
3 7 ms 8 ms popl-t2core-b-ge-wan64.inet.ntl.com [62.255.81.193]
4 7 ms 7 ms pop-bb-b-so-700-0.inet.ntl.com [213.105.174.117]
5 8 ms 11 ms tele-ic-1-so-010-0.inet.ntl.com [62.253.185.82]
6 121 ms 117 ms [212.250.14.62]
7 193 ms 188 ms p10-0.core01.bos01.atlas.cogentco.com [130.117.0.46]
8 209 ms 210 ms p5-0.core01.ord01.atlas.cogentco.com [66.28.4.110]
9 208 ms 210 ms p15-0.core02.ord01.atlas.cogentco.com [66.28.4.62]
10 222 ms 222 ms p12-0.core01.mci01.atlas.cogentco.com [66.28.4.33]
11 237 ms 245 ms p12-0.core01.iah01.atlas.cogentco.com [66.28.4.105]
12 268 ms 267 ms p14-0.core01.san01.atlas.cogentco.com [66.28.4.6]
13 269 ms 270 ms p4-0.core01.lax01.atlas.cogentco.com [66.28.4.77]
14 374 ms 373 ms filefront-allyance.dmarc.cogentco.com [38.112.29.134]
15 374 ms 373 ms node.www.la.ioerror.com [66.117.8.41]

Less than 20k speed yet again

Chrysalis · 05-08-2005, 21:26

Quote:

Originally Posted by Ignition

Wrong dude it depends on the model, make, and placement of the router, along with its' config. Foundry boxes especially are bad for it.

Anyway this has been done to death, the tele-ic routers have been upgraded and are now extremely meaty boxes indeed, and are running well within capacity.

So long as what's at the end of the trace is alright then all is fine, there was an issue but it appears to be outside the ntl network, no issues reported with the ntl core or edge last night.

Fair enough, as you said it doesnt cause any affect so all seems well. the other problem I mentioned wasnt on the trace I posted and was fixed by my contact and was affecting the rest of the trace.

jtwn · 06-08-2005, 00:19

Quote:

Originally Posted by IanUK

Still bad tonight, seems NTL currently has a very bad connection to Cogent

Target Name: www.filefront.com
IP: 66.117.8.41
Date/Time: 05/08/2005 21:49:52
2 6 ms 7 ms wapk-t2cam1-b-ge-wan41-112.inet.ntl.com [80.1.170.185]
3 7 ms 8 ms popl-t2core-b-ge-wan64.inet.ntl.com [62.255.81.193]
4 7 ms 7 ms pop-bb-b-so-700-0.inet.ntl.com [213.105.174.117]
5 8 ms 11 ms tele-ic-1-so-010-0.inet.ntl.com [62.253.185.82]
6 121 ms 117 ms [212.250.14.62]
7 193 ms 188 ms p10-0.core01.bos01.atlas.cogentco.com [130.117.0.46]
8 209 ms 210 ms p5-0.core01.ord01.atlas.cogentco.com [66.28.4.110]
9 208 ms 210 ms p15-0.core02.ord01.atlas.cogentco.com [66.28.4.62]
10 222 ms 222 ms p12-0.core01.mci01.atlas.cogentco.com [66.28.4.33]
11 237 ms 245 ms p12-0.core01.iah01.atlas.cogentco.com [66.28.4.105]
12 268 ms 267 ms p14-0.core01.san01.atlas.cogentco.com [66.28.4.6]
13 269 ms 270 ms p4-0.core01.lax01.atlas.cogentco.com [66.28.4.77]
14 374 ms 373 ms filefront-allyance.dmarc.cogentco.com [38.112.29.134]
15 374 ms 373 ms node.www.la.ioerror.com [66.117.8.41]

Less than 20k speed yet again

No, filefront has been slow as of late and as of I just tried to dl something. Just because you are downloading slow/er then you want, doesn't always mean its in the ntl part of the chain.

Chrysalis · 06-08-2005, 02:40

I believe the cogent problem is linked to linx itself in london, unfortently all these us providers who use cogent in a supposedbly bgp mix tend to carry on using cogent even when routing problems persist, I would guess to save money, but in this case I think this isnt ntl's problem.

IanUK · 06-08-2005, 21:28

The first large hop (212.250.14.62) appears to belong to NTL (according to Neotrace).
Looks like a problem they need to deal with.

I've contacted EV1 servers and they are looking into it from their end as well, but I think the problem is with NTL and not them.

Tons of sites are affected - Newshosting/Filefront/Newsreader and anything hosted at EV1

Chrysalis · 06-08-2005, 23:45

Well I am only forwarding what I know. There is a confirmed problem that the cogent link to linx is fubared possible at max load but nevertherless performing badly, since a lot of traffic goes through linx I put 2 and 2 together.

EV1 are the only ones who can solve this, they have a magnitude of providers they can use, should they wish to do so they can reroute uk traffic through a different provider but for cost reasons they are probably sticking to cogent. Cogent is america's fastest growing backbone and is by far the cheapest and more and more of their datacentre's are hooking up to it lured by the prices ($10 per mbit) but the cogent transatlantic link is congested to the hilt.

IanUK · 07-08-2005, 07:58

Hopefully you are right, this is the trace now:

Target Name: www.filefront.com
IP: 66.117.8.41
Date/Time: 07/08/2005 08:56:50
2 6 ms 6 ms [10.82.176.1]
3 8 ms 8 ms wapk-t2cam1-b-ge-wan41-112.inet.ntl.com [80.1.170.185]
4 7 ms 7 ms popl-t2core-b-ge-wan64.inet.ntl.com [62.255.81.193]
5 7 ms 7 ms pop-bb-b-so-700-0.inet.ntl.com [213.105.174.117]
6 9 ms 9 ms tele-ic-1-so-010-0.inet.ntl.com [62.253.185.82]
7 9 ms 9 ms [212.250.14.62]
8 79 ms 79 ms p10-0.core01.bos01.atlas.cogentco.com [130.117.0.46]
9 100 ms 100 ms p5-0.core01.ord01.atlas.cogentco.com [66.28.4.110]
10 100 ms 101 ms p15-0.core02.ord01.atlas.cogentco.com [66.28.4.62]
11 111 ms 111 ms p12-0.core01.mci01.atlas.cogentco.com [66.28.4.33]
12 125 ms 127 ms p12-0.core01.iah01.atlas.cogentco.com [66.28.4.105]
13 156 ms 157 ms p14-0.core01.san01.atlas.cogentco.com [66.28.4.6]
14 159 ms 159 ms p4-0.core01.lax01.atlas.cogentco.com [66.28.4.77]
15 159 ms 160 ms filefront-allyance.dmarc.cogentco.com [38.112.29.134]
16 160 ms 160 ms node.www.la.ioerror.com [66.117.8.41]

As you can see that hop is now low (9) and all is well.
I guess tonight it will be bad again

cheers

Ian