Anyone else getting really borked speeds this afternoon?

[qasdfdsaq]

Certain destinations seem to have a lag spike of ~100ms and pretty bad speeds as a result:

BBC - Fine:
C:\Users\Administrator>tracert www.bbc.co.uk (http://www.bbc.co.uk)

Tracing route to www.bbc.net.uk (http://www.bbc.net.uk) [212.58.246.93]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms n1 [10.0.0.254]
2 17 ms 12 ms 17 ms cpc5-sgyl34-2-0-gw.sgyl.cable.virginmedia.com [92.235.188.1]
3 10 ms 13 ms 32 ms sgyl-core-1b-ge243.network.virginmedia.net [81.97.48.45]
4 18 ms 7 ms 9 ms sgyl-core-2b-ae2-0.network.virginmedia.net [195.182.178.94]
5 13 ms 15 ms 12 ms leed-bb-1b-ae5-0.network.virginmedia.net [81.97.48.81]
6 19 ms 17 ms 21 ms leed-bb-1a-ae0-0.network.virginmedia.net [62.253.187.185]
7 29 ms 20 ms 20 ms nrth-bb-1b-as2-0.network.virginmedia.net [62.253.185.101]
8 19 ms 31 ms 16 ms nrth-tmr-2-ae6-0.network.virginmedia.net [213.105.159.34]
9 24 ms 38 ms 21 ms tele-ic-1-as0-0.network.virginmedia.net [62.253.184.2]
10 18 ms 24 ms 18 ms pos6-1.rt0.thdo.bbc.co.uk [212.58.239.237]
11 25 ms 19 ms 21 ms 212.58.238.153
12 67 ms 30 ms 19 ms te12-1.hsw1.cwwtf.bbc.co.uk [212.58.239.234]
13 29 ms 19 ms 23 ms 212.58.255.12
14 30 ms 29 ms 24 ms bbc-vip014.cwwtf.bbc.co.uk [212.58.246.93]
OVH - Not fine:
C:\Users\Administrator>tracert s2.qasdfdsaq.com

Tracing route to s2.qasdfdsaq.com [178.32.52.252]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms n1 [10.0.0.254]
2 5 ms 7 ms 12 ms cpc5-sgyl34-2-0-gw.sgyl.cable.virginmedia.com [92.235.188.1]
3 16 ms 7 ms 7 ms sgyl-core-1a-ge133.network.virginmedia.net [81.97.48.37]
4 11 ms 7 ms 13 ms sgyl-core-2a-ae2-0.network.virginmedia.net [195.182.178.90]
5 15 ms 15 ms 30 ms manc-bb-1c-ae6-0.network.virginmedia.net [81.97.48.77]
6 16 ms 15 ms 21 ms manc-bb-1a-ae11-0.network.virginmedia.net [213.105.159.85]
7 24 ms 23 ms 24 ms glfd-bb-1b-ae1-0.network.virginmedia.net [213.105.175.146]
8 63 ms 20 ms 18 ms glfd-tmr-1-ae5-0.network.virginmedia.net [213.105.159.46]
9 24 ms 28 ms 23 ms tcl5-ic-1-as0-0.network.virginmedia.net [62.253.185.78]
10 26 ms * 20 ms sw1.tc.lon.ovh.net [195.66.224.220]
11 117 ms 121 ms 115 ms rbx-g2-a9.fr.eu [91.121.131.181]
12 * 117 ms 121 ms vss-5a-6k.fr.eu [178.33.100.69]
13 118 ms 114 ms 134 ms s2.qasdfdsaq.com [178.32.52.252]

JANET - Also not fine, so not OVH's fault:
C:\Users\Administrator>tracert www.ja.net (http://www.ja.net)

Tracing route to www.ja.net (http://www.ja.net) [212.219.98.101]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms n1 [10.0.0.254]
2 27 ms 12 ms 12 ms cpc5-sgyl34-2-0-gw.sgyl.cable.virginmedia.com [92.235.188.1]
3 10 ms 23 ms 8 ms sgyl-core-1b-ge243.network.virginmedia.net [81.97.48.45]
4 9 ms 7 ms 7 ms sgyl-core-2b-ae2-0.network.virginmedia.net [195.182.178.94]
5 16 ms 14 ms 16 ms leed-bb-1b-ae5-0.network.virginmedia.net [81.97.48.81]
6 11 ms 15 ms 17 ms leed-bb-1a-ae0-0.network.virginmedia.net [62.253.187.185]
7 17 ms 24 ms 27 ms popl-bb-1b-as1-0.network.virginmedia.net [62.253.185.238]
8 18 ms 29 ms 28 ms tele-ic-5-ae0-0.network.virginmedia.net [213.105.159.117]
9 22 ms 28 ms 26 ms linx-gw2.ja.net [195.66.226.15]
10 22 ms 25 ms 20 ms ae1.lond-sbr3.ja.net [146.97.35.173]
11 107 ms 123 ms 107 ms ae12.lond-sbr1.ja.net [146.97.33.137]
12 114 ms 107 ms 114 ms be1.londsh-rbr1.ja.net [146.97.35.146]
13 21 ms 22 ms 45 ms janet-web-services.ja.net [146.97.137.170]
14 * * * Request timed out.
15 * * * Request timed out.
16 * * * Request timed out.
17 ^C

Inbound trace:
traceroute to d.qasdfdsaq.com (92.235.190.31), 30 hops max, 60 byte packets
1 core0.f.net.inf.ed.ac.uk (129.215.33.250) 16.471 ms 16.477 ms 16.481 ms
2 harnoncourt.inf.ed.ac.uk (129.215.33.246) 0.262 ms 0.257 ms 0.246 ms
3 vlan160.kb7-msfc.net.ed.ac.uk (129.215.160.254) 0.746 ms 0.794 ms 0.921 ms
4 vlan686.s-pop.eastman.net.uk (194.81.56.57) 0.532 ms 0.605 ms 0.656 ms
5 so-2-1-0.leed-sbr1.ja.net (146.97.42.45) 6.191 ms 6.267 ms 6.308 ms
6 as3.lond-sbr1.ja.net (146.97.33.98) 10.451 ms 10.479 ms 10.470 ms
7 ae13.lond-sbr4.ja.net (146.97.33.134) 10.846 ms 10.836 ms 10.871 ms
8 ae0.lond-gw-ixp4.ja.net (146.97.35.182) 10.936 ms 10.953 ms 10.934 ms
9 195.66.224.23 (195.66.224.23) 100.574 ms 96.998 ms 96.992 ms
10 glfd-tmr-1-as0-0.network.virginmedia.net (62.253.185.77) 98.357 ms 98.370 ms 98.410 ms
11 glfd-bb-1b-ae5-0.network.virginmedia.net (213.105.159.45) 102.483 ms 98.387 ms 101.945 ms
12 manc-bb-1a-ae3-0.network.virginmedia.net (213.105.175.145) 109.499 ms 105.057 ms 105.038 ms
13 manc-bb-1c-ae0-0.network.virginmedia.net (213.105.159.86) 108.150 ms 108.685 ms 108.705 ms
14 sgyl-core-2a-ae0-0.network.virginmedia.net (81.97.48.78) 142.906 ms 123.465 ms 123.588 ms
15 sgyl-core-1a-pc200.network.virginmedia.net (195.182.178.89) 121.618 ms 119.853 ms 123.527 ms
16 sgyl-cmts-34-gigaether-140.network.virginmedia.net (81.97.49.150) 116.428 ms 118.243 ms 125.483 ms
17 cpc5-sgyl34-2-0-cust542.sgyl.cable.virginmedia.com (92.235.190.31) 127.333 ms 124.061 ms 127.672 ms

Anyone else seeing this or have any ideas? Nothing's showing up on TBB charts, yet they go through LINX too. And it's not LINX vs TC or TH either.

[Ignitionnet]

Seeing it, yes, any idea, I blame the return path as the forward looks fine. I reckon 195.66.224.23 is maxed out inbound or is in strife. Wouldn't be the first time VM have allowed a LINX port to saturate and won't be the last so that's why the first thought is overloaded peering.

traceroute to www.ja.net (212.219.98.101), 30 hops max, 40 byte packets

1 cpc8-mort6-2-0-gw.croy.cable.virginmedia.com (82.43.108.1) 10.971 ms 33.176 ms 12.508 ms
2 mort-geam-1a-ge115.network.virginmedia.net (81.96.225.25) 42.964 ms 9.697 ms 7.909 ms
3 croy-core-1a-tenge81-490.network.virginmedia.net (62.30.242.41) 13.019 ms 11.734 ms 7.439 ms
4 croy-core-2a-ae2-0.network.virginmedia.net (195.182.178.82) 12.398 ms 55.686 ms 18.481 ms
5 brnt-bb-1a-ae9-0.network.virginmedia.net (81.96.226.1) 9.937 ms 8.553 ms 12.377 ms
6 brnt-bb-1b-ae0-0.network.virginmedia.net (213.105.174.226) 11.571 ms 13.038 ms 23.108 ms
7 brnt-tmr-1-ae5-0.network.virginmedia.net (213.105.159.50) 9.480 ms 15.460 ms 18.334 ms
8 telc-ic-1-as0-0.network.virginmedia.net (62.253.185.74) 40.359 ms 13.004 ms 12.809 ms
9 linx-gw2.ja.net (195.66.226.15) 9.848 ms 11.109 ms 13.205 ms
10 ae1.lond-sbr3.ja.net (146.97.35.173) 9.426 ms 9.019 ms 14.776 ms
11 ae12.lond-sbr1.ja.net (146.97.33.137) 166.128 ms 108.305 ms 111.326 ms
12 be1.londsh-rbr1.ja.net (146.97.35.146) 112.759 ms 96.114 ms 109.629 ms
13 janet-web-services.ja.net (146.97.137.170) 10.466 ms 180.958 ms 13.771 ms

traceroute to s2.qasdfdsaq.com (178.32.52.252), 30 hops max, 40 byte packets

1 cpc8-mort6-2-0-gw.croy.cable.virginmedia.com (82.43.108.1) 20.740 ms 10.303 ms 23.766 ms
2 mort-geam-1a-ge215.network.virginmedia.net (81.96.225.153) 7.307 ms 9.918 ms 7.536 ms
3 croy-core-1a-tenge81-490.network.virginmedia.net (62.30.242.41) 10.311 ms 9.132 ms 11.651 ms
4 croy-core-2a-ae2-0.network.virginmedia.net (195.182.178.82) 12.905 ms 9.554 ms 8.106 ms
5 brnt-bb-1a-ae9-0.network.virginmedia.net (81.96.226.1) 11.114 ms 17.304 ms 23.627 ms
6 brnt-bb-1b-ae0-0.network.virginmedia.net (213.105.174.226) 11.403 ms 15.318 ms 8.493 ms
7 brnt-tmr-1-ae5-0.network.virginmedia.net (213.105.159.50) 11.085 ms 8.900 ms 7.723 ms
8 telc-ic-1-as0-0.network.virginmedia.net (62.253.185.74) 74.747 ms 10.049 ms 12.689 ms
9 * * * Request timed out.
10 rbx-g1-a9.fr.eu (94.23.122.66) 115.414 ms 106.765 ms 121.598 ms
11 * vss-5a-6k.fr.eu (178.33.100.26) 101.760 ms *
12 s2.qasdfdsaq.com (178.32.52.252) 105.792 ms 140.557 ms 115.563 ms

traceroute to 195.66.224.23 (195.66.224.23), 30 hops max, 40 byte packets

1 cpc8-mort6-2-0-gw.croy.cable.virginmedia.com (82.43.108.1) 9.035 ms 26.917 ms 11.243 ms
2 mort-geam-1a-ge115.network.virginmedia.net (81.96.225.25) 14.453 ms 8.607 ms 25.676 ms
3 croy-core-1a-tenge81-490.network.virginmedia.net (62.30.242.41) 9.722 ms 10.982 ms 12.246 ms
4 croy-core-2a-ae2-0.network.virginmedia.net (195.182.178.82) 32.250 ms 9.269 ms 10.525 ms
5 brnt-bb-1a-ae9-0.network.virginmedia.net (81.96.226.1) 25.168 ms 13.017 ms 12.506 ms
6 brnt-bb-1b-ae0-0.network.virginmedia.net (213.105.174.226) 9.950 ms 22.397 ms 9.741 ms
7 brnt-tmr-1-ae5-0.network.virginmedia.net (213.105.159.50) 20.487 ms 24.456 ms 15.523 ms
8 telc-ic-1-as0-0.network.virginmedia.net (62.253.185.74) 18.656 ms 9.408 ms 13.513 ms
9 ge0-0.pr2.linx.net (195.66.227.254) 65.178 ms 151.480 ms 27.726 ms
10 ge0-0.tr3.linx.net (195.66.248.126) 42.124 ms 23.369 ms 11.906 ms
11 fe2-1.tr1.linx.net (195.66.248.179) 14.833 ms 20.741 ms 11.336 ms
12 fe1-0.pr1.linx.net (195.66.248.229) 13.844 ms 28.742 ms 20.141 ms
13 195.66.224.23 (195.66.224.23) 255.599 ms 98.604 ms 115.449 ms

TBB doesn't use the bad route so is fine.

# Host Sent Recvd Best Avg Worst
1 thinkbroadband-gw2.core-rs2.thdo.ncuk.net 15 15 0ms 3ms 11ms
2 gi4-3-10-star1.bdr-rt1.thn.ncuk.net 15 15 0ms 8ms 125ms
3 gi4-47-10-star1.bdr-rt3.thdo.ncuk.net 15 15 0ms 0ms 0ms
4 linx-gw2.router.ntli.net 15 15 0ms 0ms 0ms
5 brnt-tmr-1-as0-0.network.virginmedia.net 15 15 1ms 1ms 3ms
6 brnt-bb-1b-ae5-0.network.virginmedia.net 15 15 1ms 13ms 90ms
7 brnt-bb-1a-ae0-0.network.virginmedia.net 15 15 1ms 2ms 11ms
8 croy-core-2a-ae0-0.network.virginmedia.net 15 15 1ms 1ms 2ms
10 mort-geam-1a-tenge71.network.virginmedia.net 15 15 2ms 2ms 3ms
11 mort-cmts-06-gigaether-151.network.virginmedia.net 15 15 2ms 2ms 3ms

[qasdfdsaq]

Seeing it, yes, any idea, I blame the return path as the forward looks fine. I reckon 195.66.224.23 is maxed out inbound or is in strife. Wouldn't be the first time VM have allowed a LINX port to saturate and won't be the last so that's why the first thought is overloaded peering.

Interesting, return path only congestion. I hadn't thought of that. I forget these things are bidirectional sometimes. Was curious why the JANET trace only spiked on the second hop outside VM.

I wonder why today though... Locally my TB graph looks fairly tame overall.

Also I'd assumed OVH's sw1.tc.lon.ovh.net meant telecity but the IP is a LINX peer too, so yes it looks like a VM <=> LINX problem.

[thenry]

a few heads eating away a bit of paid bandwidth! pings are shot as a result too.. aahh the joys of paying for services!

[Ignitionnet]

Interesting, return path only congestion. I hadn't thought of that. I forget these things are bidirectional sometimes. Was curious why the JANET trace only spiked on the second hop outside VM.

I wonder why today though... Locally my TB graph looks fairly tame overall.

Also I'd assumed OVH's sw1.tc.lon.ovh.net meant telecity but the IP is a LINX peer too, so yes it looks like a VM <=> LINX problem.

There's a LINX PoP in several Telecity facilities.

https://www.linx.net/about/partnerpages/telecity.html