[SOLVED] Slow connectivity (TCP stuck to 1 Mbit/s)

mdop 2018-03-05 13:55:18 UTC #1

Since February 2018 the speed of some TCP connections is capped at 1 Mbit/s (around 125 kB/s). This affects in average every third to fifth connection (20-33%) while other run at full speed.

We started experiencing this issue when sending data from our ISP (T-Mobile Czech Republic) to our hosts at Online.net. The issue is in one way and only with TCP (UDP runs at full speed). From the side of Online.net the issue is verifiable by running “wget -O /dev/null https://www.t-mobile.cz/dcpublic/Navod_Huawei_LTE_TDD_B2338.pdf” couple of times.

Today we were able to find host/VPS (in Africa) that behaves just like our ISP when connecting to the Online.net network. This further confirms our suspicion that the issue is more close to France than Czech republic.

While we are paid customers of both T-Mobile CZ and Online.net (over 10 servers), we are unable to get any other help except for answers “we don’t see any problems/there is no packet loss/ping goes through”. No one is willing to pass our issue further.

Is there any body else who is experiencing something similar?

# iperf3 --client ping.online.net --port 5200 --time 5 --parallel 3
Connecting to host ping.online.net, port 5200
[  4] local 192.168.20.3 port 56816 connected to 62.210.18.40 port 5200
[  6] local 192.168.20.3 port 56818 connected to 62.210.18.40 port 5200
[  8] local 192.168.20.3 port 56820 connected to 62.210.18.40 port 5200
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  6.13 MBytes  51.4 Mbits/sec  269    163 KBytes
[  6]   0.00-1.00   sec   231 KBytes  1.89 Mbits/sec   12   4.01 KBytes
[  8]   0.00-1.00   sec  6.17 MBytes  51.8 Mbits/sec  309    164 KBytes
[SUM]   0.00-1.00   sec  12.5 MBytes   105 Mbits/sec  590
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   1.00-2.00   sec  5.15 MBytes  43.2 Mbits/sec    0    183 KBytes
[  6]   1.00-2.00   sec  96.2 KBytes   788 Kbits/sec    8   6.68 KBytes
[  8]   1.00-2.00   sec  5.15 MBytes  43.2 Mbits/sec    0    187 KBytes
[SUM]   1.00-2.00   sec  10.4 MBytes  87.2 Mbits/sec    8
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   2.00-3.00   sec  5.21 MBytes  43.7 Mbits/sec    2    148 KBytes
[  6]   2.00-3.00   sec   152 KBytes  1.25 Mbits/sec   11   5.34 KBytes
[  8]   2.00-3.00   sec  6.62 MBytes  55.6 Mbits/sec    0    208 KBytes
[SUM]   2.00-3.00   sec  12.0 MBytes   101 Mbits/sec   13
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   3.00-4.00   sec  4.41 MBytes  37.0 Mbits/sec    0    175 KBytes
[  6]   3.00-4.00   sec   102 KBytes   832 Kbits/sec    8   4.01 KBytes
[  8]   3.00-4.00   sec  6.62 MBytes  55.6 Mbits/sec    0    231 KBytes
[SUM]   3.00-4.00   sec  11.1 MBytes  93.4 Mbits/sec    8
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   4.00-5.00   sec  3.68 MBytes  30.9 Mbits/sec    6   72.1 KBytes
[  6]   4.00-5.00   sec   106 KBytes   865 Kbits/sec    7   2.67 KBytes
[  8]   4.00-5.00   sec  6.62 MBytes  55.5 Mbits/sec    0    252 KBytes
[SUM]   4.00-5.00   sec  10.4 MBytes  87.3 Mbits/sec   13
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-5.00   sec  24.6 MBytes  41.2 Mbits/sec  277             sender
[  4]   0.00-5.00   sec  23.0 MBytes  38.5 Mbits/sec                  receiver
[  6]   0.00-5.00   sec   687 KBytes  1.13 Mbits/sec   46             sender
[  6]   0.00-5.00   sec   640 KBytes  1.05 Mbits/sec                  receiver
[  8]   0.00-5.00   sec  31.2 MBytes  52.3 Mbits/sec  309             sender
[  8]   0.00-5.00   sec  30.1 MBytes  50.4 Mbits/sec                  receiver
[SUM]   0.00-5.00   sec  56.4 MBytes  94.7 Mbits/sec  632             sender
[SUM]   0.00-5.00   sec  53.7 MBytes  90.0 Mbits/sec                  receiver

iperf Done.

[ANSWERED] - Slow speed

gentlemon 2018-03-05 14:08:03 UTC #2

have you tried internally too ? Scaleway / online.net Amsterdam to Paris?

mdop 2018-03-05 14:34:29 UTC #3

I have extra VPS at Scaleway in Paris for testing and there doesn’t appear to be any issue (connecting to/from Online.net or T-Mobile). However there’s one similarity of Scaleway with Online.net. Following runs with the same speed limits from both Scaleway and Online.net (1 Mbit/s):

wget -O /dev/null http://fr2.rpmfind.net/linux/centos/7/os/x86_64/Packages/ImageMagick-doc-6.7.8.9-15.el7_2.x86_64.rpm

While even from Czech republic it never runs slower than 1 MB/s.

I could get another VPS in Amsterdam but we already tried other locations (including Czech Republic, Germany, Netherlands etc.) and they all work fine.

gentlemon 2018-03-05 14:59:34 UTC #4

I assume the error is somewhere on online.net’s networking gear or very near to online.net’s network.
Try a VPS in AMS and see If iperf results are the same, if so it is online.nets issue on their network basically. If not, this is going to be a long journey, since somewhere inbetween something is going very wrong.

Please post the results here.

Imen 2018-03-05 16:46:37 UTC #5

Hi there! Thank you very much for bringing this to our attention! Can you provide us your ticket ID so our network team can check what’s going on with your VPS? Thanks.

mdop 2018-03-05 17:22:23 UTC #6

Hi. Related ticket at Online.net is #619130.

Imen 2018-03-06 10:27:57 UTC #7

Thanks! Our network team are currently investigating this! You should see an update to your ticket shortly! Don’t hesitate to contact us if you are still facing this issue.

mikmak 2018-03-09 21:53:48 UTC #8

hi @mdop, please check again,
we have done some optimisation for your ISP which seems to have a laggy transit

mdop 2018-03-10 07:17:42 UTC #9

Thanks @mikmak I’m however still experiencing the same issues to both our servers and the public iperf server.

# iperf3 --client ping.online.net --port 5200 --time 5 --parallel 3
Connecting to host ping.online.net, port 5200
[  4] local 192.168.20.22 port 51890 connected to 62.210.18.40 port 5200
[  6] local 192.168.20.22 port 51892 connected to 62.210.18.40 port 5200
[  8] local 192.168.20.22 port 51894 connected to 62.210.18.40 port 5200
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  6.04 MBytes  50.7 Mbits/sec   82    226 KBytes
[  6]   0.00-1.00   sec   191 KBytes  1.56 Mbits/sec    9   4.01 KBytes
[  8]   0.00-1.00   sec  4.96 MBytes  41.6 Mbits/sec   21    183 KBytes
[SUM]   0.00-1.00   sec  11.2 MBytes  93.9 Mbits/sec  112
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   1.00-2.00   sec  5.89 MBytes  49.4 Mbits/sec    0    243 KBytes
[  6]   1.00-2.00   sec   135 KBytes  1.11 Mbits/sec    5   6.68 KBytes
[  8]   1.00-2.00   sec  5.15 MBytes  43.2 Mbits/sec    0    199 KBytes
[SUM]   1.00-2.00   sec  11.2 MBytes  93.7 Mbits/sec    5
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   2.00-3.00   sec  5.95 MBytes  49.9 Mbits/sec    1    187 KBytes
[  6]   2.00-3.00   sec   107 KBytes   876 Kbits/sec    7   4.01 KBytes
[  8]   2.00-3.00   sec  5.58 MBytes  46.8 Mbits/sec    0    219 KBytes
[SUM]   2.00-3.00   sec  11.6 MBytes  97.6 Mbits/sec    8
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   3.00-4.00   sec  4.41 MBytes  37.0 Mbits/sec    1    148 KBytes
[  6]   3.00-4.00   sec  80.2 KBytes   657 Kbits/sec    8   5.34 KBytes
[  8]   3.00-4.00   sec  6.44 MBytes  54.0 Mbits/sec    0    239 KBytes
[SUM]   3.00-4.00   sec  10.9 MBytes  91.7 Mbits/sec    9
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   4.00-5.00   sec  4.41 MBytes  37.0 Mbits/sec    0    167 KBytes
[  6]   4.00-5.00   sec   134 KBytes  1.09 Mbits/sec    6   4.01 KBytes
[  8]   4.00-5.00   sec  6.87 MBytes  57.6 Mbits/sec    0    259 KBytes
[SUM]   4.00-5.00   sec  11.4 MBytes  95.7 Mbits/sec    6
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-5.00   sec  26.7 MBytes  44.8 Mbits/sec   84             sender
[  4]   0.00-5.00   sec  25.7 MBytes  43.1 Mbits/sec                  receiver
[  6]   0.00-5.00   sec   647 KBytes  1.06 Mbits/sec   35             sender
[  6]   0.00-5.00   sec   590 KBytes   967 Kbits/sec                  receiver
[  8]   0.00-5.00   sec  29.0 MBytes  48.7 Mbits/sec   21             sender
[  8]   0.00-5.00   sec  28.3 MBytes  47.5 Mbits/sec                  receiver
[SUM]   0.00-5.00   sec  56.3 MBytes  94.5 Mbits/sec  140             sender
[SUM]   0.00-5.00   sec  54.6 MBytes  91.6 Mbits/sec                  receiver

iperf Done.

Misteur 2018-03-10 09:07:03 UTC #10

Hello @mdop,

Thank you for letting us know !

As your issue does not seem to be fixed, I will escalate it to our network team through your previously created ticket.
We will let you know through your ticket as soon as we have more information on our end !

Cheers,
Mehdi Mebrouk
Customer Success Specialist at Online / Scaleway

Max_Millow 2018-03-12 13:28:50 UTC #11

Here is my test with the same problem:

[details=Summary] root@:~# iperf3 --client ping.online.net --port 5200 --time 5 --parallel 3
Connecting to host ping.online.net, port 5200
[ 4] local 10.8.93. port 39738 connected to 62.210.18.40 port 5200
[ 6] local 10.8.93. port 39740 connected to 62.210.18.40 port 5200
[ 8] local 10.8.93.** port 39742 connected to 62.210.18.40 port 5200
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 93.0 MBytes 780 Mbits/sec 158 1.47 MBytes
[ 6] 0.00-1.00 sec 50.3 MBytes 422 Mbits/sec 8 724 KBytes
[ 8] 0.00-1.00 sec 352 KBytes 2.88 Mbits/sec 25 7.07 KBytes
[SUM] 0.00-1.00 sec 144 MBytes 1.20 Gbits/sec 191
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 1.00-2.00 sec 86.2 MBytes 723 Mbits/sec 12 1.11 MBytes
[ 6] 1.00-2.00 sec 33.8 MBytes 283 Mbits/sec 9 556 KBytes
[ 8] 1.00-2.00 sec 201 KBytes 1.64 Mbits/sec 12 7.07 KBytes
[SUM] 1.00-2.00 sec 120 MBytes 1.01 Gbits/sec 33
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 2.00-3.00 sec 68.8 MBytes 577 Mbits/sec 11 851 KBytes
[ 6] 2.00-3.00 sec 27.5 MBytes 231 Mbits/sec 4 433 KBytes
[ 8] 2.00-3.00 sec 204 KBytes 1.67 Mbits/sec 17 4.24 KBytes
[SUM] 2.00-3.00 sec 96.4 MBytes 810 Mbits/sec 32
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 3.00-4.00 sec 38.8 MBytes 325 Mbits/sec 6 650 KBytes
[ 6] 3.00-4.00 sec 17.5 MBytes 147 Mbits/sec 35 247 KBytes
[ 8] 3.00-4.00 sec 84.8 KBytes 695 Kbits/sec 7 8.48 KBytes
[SUM] 3.00-4.00 sec 56.3 MBytes 473 Mbits/sec 48
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 4.00-5.00 sec 38.8 MBytes 325 Mbits/sec 15 492 KBytes
[ 6] 4.00-5.00 sec 16.2 MBytes 136 Mbits/sec 7 208 KBytes
[ 8] 4.00-5.00 sec 308 KBytes 2.53 Mbits/sec 20 5.66 KBytes
[SUM] 4.00-5.00 sec 55.3 MBytes 464 Mbits/sec 42
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-5.00 sec 325 MBytes 546 Mbits/sec 202 sender
[ 4] 0.00-5.00 sec 322 MBytes 541 Mbits/sec receiver
[ 6] 0.00-5.00 sec 145 MBytes 244 Mbits/sec 63 sender
[ 6] 0.00-5.00 sec 143 MBytes 240 Mbits/sec receiver
[ 8] 0.00-5.00 sec 1.12 MBytes 1.88 Mbits/sec 81 sender
[ 8] 0.00-5.00 sec 1.11 MBytes 1.86 Mbits/sec receiver
[SUM] 0.00-5.00 sec 472 MBytes 792 Mbits/sec 346 sender
[SUM] 0.00-5.00 sec 466 MBytes 782 Mbits/sec receiver
***********************
root@*******:~# curl -s https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py | python -
Retrieving speedtest.net configuration…
Testing from ONLINE SAS (51.15.)…
Retrieving speedtest.net server list…
Selecting best server based on ping…
Hosted by fdcservers.net (Paris) [1.88 km]: 14.505 ms
Testing download speed…
Download: 36.87 Mbit/s
Testing upload speed…
Upload: 386.81 Mbit/s
*****************
[/details]
My download speed is dropped below 50mbits on my VC server instances and also I noticed performance drop. probably this internet speed problem is related to performance drop couse servers are virtual working over ethernet…

zcalusic 2018-04-10 15:36:52 UTC #12

Having the same problem, been investigating it for several days only to find this topic. Suddenly the speed is ~ 2Mbps, and it happens often (every several connections, as already reported). Also, SNMP monitoring constantly spams with alerts that it can’t connect to remote host, then immediately recovers. Meaning, small SNMP (UDP) packets get lost often, too.

Seeing that this issue is open for so long without resolution doesn’t inspire confidence that it will be resolved soon, so i’ll probably jump ship, because I’ve already spent enough time debugging this quite unacceptable networking behaviour. It has been a lot of pain…

bene 2018-04-10 15:55:16 UTC #13

Hello,

There is currently no known network issue from our side. Have you opened a ticket so we can have a look into your case?

Benedikt

zcalusic 2018-04-10 15:57:49 UTC #14

@bene I can’t add anything to what was already reported in this topic. All the same symptoms. 2Mbps on a 1Gbps server? And often… Check the above iperf3 outputs, mine is same. Fast, fast, then suddenly very slow connection… rinse and repeat.

mdop 2018-04-10 16:37:19 UTC #15

Just an update - we’re still affected by the reported issue. In the last communication (few weeks ago) support again asked for access to server. Even though we already give out (deleted) one before (probably miscommunication on both sides but anyway…).

I’m tired communicating the same (things like test cases no one actually runs) over and over.

As this issue can also affect some of our customers and we are running options, we are looking for other hosting providers.

zcalusic 2018-04-11 08:13:18 UTC #17

You’re completely right @gentlemon. After relocating services to Hetzner alerting is now completely silent and networking is plenty fast, no packet loss anymore.

This was an expensive lesson, tried to save a few € going with Online, but in the end lost much more, money AND time. Oh well…

gentlemon 2018-04-11 10:23:34 UTC #18

glad you are happy

unfortunately I had to learn the hard way, too. Now I use my time to develop new feature instead of debugging scaleways issues and doing their job

Slow connection speed (150kb/s) to hetzner

johnprinceton 2018-04-14 07:37:44 UTC #19

I can confirm that the same network issue is happening to me at dedicated server at Online.net, not just scaleway alone.

prefekt 2018-04-14 07:47:51 UTC #20

@johnprinceton yup me too, and probably a lot other users! and all that tell you in ticket is some stupid iperf3 test lol, THEY HAVE NETWORK PROBLEMS AND WE WANT FIXED THEM NOW !!!

gentlemon 2018-04-14 07:55:13 UTC #21

my experience from the last couple of bigger network issues has shown, that you probably need to annoy them on twitter or any other public social place to have some sort of reaction and action taken. Which I do not like to do, but this seems to be the only way to get them to do anything at all in time. Sadly.

https://twitter.com/scaleway
https://twitter.com/online_fr
https://twitter.com/online_en