[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]
Re: [tor-relays] inet_csk_bind_conflict
- To: Christopher Sheats <yawnbox@xxxxxxxxxxxxxxxx>
- Subject: Re: [tor-relays] inet_csk_bind_conflict
- From: Chris <tor@xxxxxxxxxxxxxxx>
- Date: Mon, 5 Dec 2022 01:08:10 -0500
- Autocrypt: addr=tor@xxxxxxxxxxxxxxx; keydata= mQINBFWMtxIBEADWtP+m+KK37yXz5i/w5nCjxpjwS6H2QTNIpHTx+444DxlX90L8GOLEOwS7 LRx/OJ/Vo+aqmAOL49Slejj6U29r8qJz7Nq7g+HfE9AilJMTvBWU5W21PF2wuKuQJLboPR4s iCnoHBg2ylds1aIweW126t01GBy+lvFJ/c6mqeHn69kq0VzJkMWv1YcIPaPMWj/foGf+yPp9 3hB2r8fIrY6q6jkQdP+mvBpn8rNteqgtGviMqCeQIwtaA3VcwQqcomSOjY4zBA7Uhse1iMF0 tjK3N0Bar/rV2zMu+jqppoat6E9y0AjshFCSgUMiZKIkMQ3oWEalkwyNXHpgcGb69WBKD+kl 5kJtOCP7aSs3VAR2xA86pX6lwfYlzNL2CsNzHGmkSm6mZygLpkQOOn9I3zDjhzGQ6j00I0P9 pi3a39BbgBrVYUa0O6NAIs816+vuHacR91jFvYQzMTQJ1cesRXnxvmDpaCuTLxFNWM2sORh3 BdBx3H0Xf5JNF11dJZ1b7XGxuPDA+hbSZwlU68EqptBocE9jlfTZ/ja0wx6Pgn137EKJ4ZUy d5PMp7eEUUfZFtEretfJ+IKJ+/UFjKRfgPiP6OY536TNzramtMsHN2uMsW18DY7oYaDZ2Nm/ EOg36Bjp8551gWnstogdjmoJAbmsN/WqZKuOYtZwrbEb485+8wARAQABtBpXQ0IhIDx3Y2JA d2Nic2VjdXJpdHkuY29tPokCoQQQAQgAizAUgAAAAAAgAAdwcmVmZXJyZWQtZW1haWwtZW5j b2RpbmdAcGdwLmNvbXBncG1pbWUICwkIBwMCAQoCGQEaGGxkYXA6Ly9rZXlzZXJ2ZXIyLnBn cC5jb20FFgADAgEFHgEAAAAGFQoICQMCFiEE0cq5h5p8USgUPeenb/FtB2wEq2cFAmITVg0C GyMACgkQb/FtB2wEq2fPvBAAkJ3Z9lxFPVvGT5j40GnhsDkF7F0gJ2O1SNo+gTgWKs9Bn8MT 5UuShLhdXazFNfMLezjNXDWlRtEIIpoHkJ5MN/IPQmZ6/Pc/zLKfEIZVdD1HD6MImNVT8GyW vXuSrmQEqne7vV3CeoABXZSPyl7wiCibahlHyjPNmFcIpBY+EqSVKhTXwNgtquL5dJUUSwdz rwSoI9sNOQqcaqpyzZZOEUAEbqDImMMC3FsYf7p3VBmq0+fddxCVzoC0ibEf1YDPwZuVulbn URPv+O5AD+hQkAmYMXiI7As3FLnHY0XoimVhxnLKHPj+LZvYsrFczC808rnWuuW3ri05euLd P98/O4Y3Cpe88hFhaeG37Bp/wiwIzKXrYb0+pPf40GGUl1dI0694TgtV5Am1ithBiOJlL7IA I+/npSRitS751CpT8s9not5kQJVa7J9AIE0+7T20bR7dYXgLWyRank3vExOzy4GdeCeoFct9 M9WYEaxsp7dQdz4LNs0rndQeHkVr0nkwiQszeDrZntWGknMlXYx0wzJPF+RB1um1+YFs8bOA +N4qJ7MGejG4ucNC1DBkeZoAYYQJZfHWiyz1l3ncN9wOv9qNH7+zhSnyu3OwyjOa4GaIFtqj L2cjQI/LRU6J4hW+cN/G2weK5KPxntEyI5rPEqIR5AcmOvTqQrcSWLtnfdu5Ag0EVYy3FQEQ AM+zo7GBG+zkHJv+rK1RrwSCGeFdrHL1M6qHvvekc+29aId1i5V99C1fJo4a+Yl9LXFvS+p7 43rN/35FjR09FQx8wYUTRaYDYC287/Rwl1QI0AVzu7X4qnlnFvbij+BQyUXQyxoILQEoMjBA WnlQRe6OCMsi4AyozIvYCWZ2QG/03sMiQq9KCZM1UrTvBHGdtMaaq7b90VUZGCk+ME8Sz1b6 uSKjUttGqz+14U4c5lrMY5Ao4hZObsYwGq3JCfDfb8Ibiaj2qSRMC1lWBO6d/Cd3HD5jC1pE FFQHwZZmI26YIQMsrJ2+V7vWue/X+PUHMur2x6laMNE9ds06jipVV+ZQzzUp/V2Ledok2+4m f85sEOAC9mwloI2vjNqnyVM8k5VtnOAD86Y1I1wO87pthZSX5ZHmxkxxl+Vw5cx+siPo+etU FC+hHk7/zInv1lKnuVHrX2IHcW714l3I7RDaNWvLXuiAMY9M6loFLuG6VTXcrf2FI7BMgqXD dStznJNdBfslEbq5cOLXxfYKYbuwipJ/2LuaE3KVXcdebmBx12oMl8T/F70C1A+ynrfKecND eyar7OakYvxk6+lT7hdzkKzs9DDI75BU8SNKUKdWT/wl0d/6tTIQRhqySrcICGn6O+FU1ODh ccVrgPJiNBhRfXt4WG4P5yhSMnGXlywjzKoFABEBAAGJBEEEGAECAisFAlWMtxYFGwwAAADB XSAEGQEIAAYFAlWMtxUACgkQ//PgyfkqRnuEHQ//S626WYYerZMsjglzG9X+gRy9X15iYxm5 sfw2KMbVRXSkt++beT9R+4eefasdzOKzo37Hehnv8EpToKwk7nz5CFmPvicN/wZ4h4/UM1Jn wNMJH5QaLWzhNf+bD9a/8l+TUIMAfIQx+Ub82xgLFf1dD7JkIddH3WfEIOmDaoR9MlyLzglF +WKGaXZXYPkko6h7lZuq7rxkabtVohvQhk2UypwwfU25wqrI1i8RsivB5kFn7+UrgzulNCFM UDzvld/Ym6crqxGmYa0ayxnyTRzFdFGCA/A7fhuJo3WcJq8OuimV22BcXeK8t8cvijOmGFMR QZ74dh5yU/axg+EnWNsVQCwU8qjNgsm7vX2dfovE+8/tMFGh+NibfpXtfBtpQYpvYMsuQ4Z+ hxaozESGxWBmNXzZ9kPKrIQqTuut/tIe5IylISkfz0oTEskJp2E10zZq1mKIwtghlhmHXuA2 XW9/y7ivgoylHyDj+vopCeQo73UTNA5dOAfuzN8AyAaTo4cgki4KYq1mnKfMHp2yYku1RJEF 4BxqTeKfE2WdTYhsusqn6ZNT375hMYeA1pAbYfb3Ybh9NN4jxmXV5hMd15jQUdwwCZecK/34 zmgY9z2kV+9tbkofyc/8SJcYypqOPKIU4b6iPQdNyX2DoWuygEMtK+wLRxH/kTAQ9LYs7tM2 15MACgkQb/FtB2wEq2dU+Q/+NvoqGo1gDu+yvep5YhQLVyJZLvvYlMhQuGhYD02I22EeNZSh HPT2++9U2hbTAZuME+AZZfNVZUuc51BUZLXLYAwUE5+eJCyDq4amLiQaYYVJxK6q2XGcaSpJ K5gbR9U9BclXTLdcrpFUXLs2BTQvKRji8c6MjxTrmLgj5fKh4UAx4Aul0DkLH3lP+DgHmiLM JCPdTnyUpM5+TYe0spqybeEySWI5+e0BJHJkAYigdq97JWEvwnavvQo2mc9PRDyviVx/B21f jnqAmP9ihnWlBUc++jGLRAXfexpDvoWRaWCrNCh6SlNVfADgsnPmZeNx6IFVWSePVpn88Q1N RaNsAhEeFH7RxAHLzsxksbfHq18dUvv727E/JPtHF8gy6ymIdNgG9T0i24IxKhjZeDI+gHlp 6c85+R9P72tuf6K9RJ8qeM4EApwGADI5Y4hrlfpsoQhoxNe9AAuS1NJG+K43TxR1qGPirPq5 g7ZpbHrF3F4ZniPpAHYose6Dwy5iMIJcDITuZI9ek47UYtOkgdqOhcZVnKFXG7B1hWHX8G8L O3dJkuLU1JpZCg7BUXD4iQmlHd+niYwglexRtJBTdmzrbXHDSMVR1BaZuTyDY1pJTsJ9RkuP kMlJQPU9sRLT8Va+4ld4XXzJL9yjyP7WafiXaXrL+mew7zTcUNTtNFbwvS+5Ag0EVYy3GAEQ AL2lIVP8MSA4pviJ8D83MrJL5IEeR7CQwjWHFxh92T+M7JZVQ0pHkIMBHit1jRDo5MARqnR2 W14jZ6Nt7lVHQ8P7puOpYlvokDbv3+ln5JvgFvc/7P06VWesi24Ft5V8LBXGS9QZ8Jn031DE eGV+BVhX9UlF8U1WqWONRR1Sb3Nda+rg8n21fGGidUm+1gmjm50u9VYHb0ZK226OLfMASPYQ 8wt8ZSjkERK2fJwofGLZR1X67K3L4VuR7IQvS/xrY906oqZtl3/tjB2A7yFMWfxisv31kb/s mv3EWJspLs4eK+a3PZhHTsG4X1MQxP1UskM/9rx82aiMKENv8DsXxpvb+UPIX/AR4y2rwZwU KLYNVpGED9f/FcTxwozdl48GMsF/IQHQzYEu1LpAiznPj4icEGnjvdtABWk2ZMjhKYEXfTaj iEQxO3yaazkL9IuRssPIS87EwEKcIqIpxwXUBbEYKLe7XFijl+XkXKWl+35IYUzV/M0mLigE Xc4WReudjzVWF0jA/dP7Rsc1GEUq4vOxuiOOBiXxVKZPPWxU3MunGhhc8Cx8dEXf3FcDML2M px9WU5QxxuPon830KB4yz37tmgui5qkDaHuQkhTEwlvnP+ELo0yrw2CQXDqBVf77dmnUw9c9 jUPZmqsbrl4/nrbhQ89P8EhL2dKYfIH0iG6dABEBAAGJBEEEGAECAisFAlWMtxkFGwIAAADB XSAEGQEIAAYFAlWMtxgACgkQgIt/XQhd0uslbhAAruDf2E38K45HWP3qV0oDYnMNRIWq/dNZ qXHkmkSjmlKcbCHrklCb155Z9lLU75Yqtjax2KhiNCiNRHFhaSMuappO2pBhnHZqnlLbB250 FPWdu+mSzi4yi0pUrtEJ+Sksb4HyKZzfHIqaDV8XsCU1vPYu1b7rHkXVuFXP1HDKTGp7dimt VQ5vDSx0hikU6kFwYT685AC6VIFymGL6VxjwKwsZ1uR0/xAktryFAzVIRtnXV6jjQ7NoCkVV BrtQUAiVSxO0V7FeOmNhkoSAQewJKIhg0KjJ+le8+899ypw5+/JnnlvFbvv/ZKTNOMXi9oNO w8k2y+HyPx0PQjemb37JPTF7DgSkFqHpSPQ3UISMpceKByhOnnNhq92JsMp/Q3zVR+zvipj9 eOFNHxpoc0AFelcU1hnH61qUEutIsveGUvhw0IegWN0xxBD5KpGM3nwEqdJC6OnzqKwRD3TO tVGGHUA1lVkJu6AwwxPSyRDVGFIN8RBAMB4/ysuAKIraNWsIf0Y6fgXWLeaE2iuG43MWiYqR wtwrZ/R4nSBe+1EIUV4vIZXjpP6vLSfzeLnM8OHUw41TebDhfMwyMvIDaT3pnwpizTdoZ89g vJJJiDZmCWeWOV6HjMYhH5cg5VWor76uinG1fDRWJCbVUf5ZMYq+If5uLO893qMtA1oXcs5R /tsACgkQb/FtB2wEq2dMuw/+JQ6zm9C+tqReGGw1KM/39wVXTwnchJQKtSJYmcAsAd3tR7C5 e/xlX/ANLcjt/pZD9CKFdp9lBhT/yMzFFX7+k3UFspb+wPR83qjsZueEvf0VBC/FcW48vCJE BikGIGHpJhL3czSsLr6Tmtdy6D8iQ1HvWcZxAoZXa5h8J5T6QWD+u7TH7am+DZPg7ZqGA7zl Derbiiv9Xo8z4kRBO4eRin4funW7zPlpp9wOrsD8peTaw6QJcgZ/y60Ef+6Uz0FCngz8vq3O weXZFp87olPMsCoTZ/2gd9Q1346ozXGNfKIIrFeGnZhWSZIsbFrH1Eqm5bR/CwKFnPeQfogo JPKeTWX0john8EJAjkHwQOsypqzXGtFvL7k6tANL6V3A0lyhyC+GvfmSEUO+ey7s+VLK48QX 1JSr0kXenElTgzj3mKnCgRqf60oBTwepVXzwKahNhC5CJVQLfM3b1IS9y46sybyQQshup9YE sMaMBSyrSyiO6vaCwNhP6jWKwCK8FDyO78vOxfrUtcPahHb1YstnFLegLnrdl/OzanvfNB2s 54N5NCsONeKy3Iok1CTQbUb+ECIp16AN22lNICZoMj6csSR1S6Ah8Y3FVLSThaUpUMLMR9EE 4xEnRmqUaCvSvafjazfkbAUgxnN9V+CCVI68904gcm0Wkxk+u5KsBVFtVfY=
- Cc: "tor-relays@xxxxxxxxxxxxxxxxxxxx" <tor-relays@xxxxxxxxxxxxxxxxxxxx>
- Delivered-to: archiver@xxxxxxxx
- Delivery-date: Mon, 05 Dec 2022 03:00:22 -0500
- Dkim-signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=wcbsecurity.com; s=default; h=Content-Transfer-Encoding:Content-Type: In-Reply-To:MIME-Version:Date:Message-ID:From:References:Cc:To:Subject:Sender :Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=t4BeLPQpPX9PBABmruyFztDUTpjdRAIrtkRNI5SefVA=; b=fKSMbvQeHxGamZE5IC4f+9Jsmu XXhtgKywKp7R7/yWe7ZltkFoA+cz1KZs6TiGcH6LNkUt2vgh4/7QHuCl4R8mggF9RMWgCerznGI3u iCZG3BP1XfqKxdXyoc34keJiGh4v1qmYc9VQD0gQR863/OXmHsfsx3Gig3gGNkWxHJQs4USlnu/Fl HtrYA8jSykZu0n3UOGQRovEZIE7PsauYD5srdPQpYocl8Qj+oCKVlW4V81UKzpo4sPMXtNQddkIj9 D23LbPD+1Ue/v1NUZ0xCnkQEgUwihkbx32LaeYcnQEFTlsEaz4KHuD4qWxe+u1gCyTRj2lO3bSKih kbupYptA==;
- In-reply-to: <CD0F3641-2457-4884-B3A2-4906BFFCCE85@emeraldonion.org>
- List-archive: <http://lists.torproject.org/pipermail/tor-relays/>
- List-help: <mailto:tor-relays-request@lists.torproject.org?subject=help>
- List-id: "support and questions about running Tor relays \(exit, non-exit, bridge\)" <tor-relays.lists.torproject.org>
- List-post: <mailto:tor-relays@lists.torproject.org>
- List-subscribe: <https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays>, <mailto:tor-relays-request@lists.torproject.org?subject=subscribe>
- List-unsubscribe: <https://lists.torproject.org/cgi-bin/mailman/options/tor-relays>, <mailto:tor-relays-request@lists.torproject.org?subject=unsubscribe>
- References: <B11BC7C2-89A1-47AC-B640-E601AD3E496F@emeraldonion.org> <92e81ff4-fef3-db57-0467-9f6eb4bc1e0b@wcbsecurity.com> <CD0F3641-2457-4884-B3A2-4906BFFCCE85@emeraldonion.org>
- Reply-to: tor-relays@xxxxxxxxxxxxxxxxxxxx
- Sender: "tor-relays" <tor-relays-bounces@xxxxxxxxxxxxxxxxxxxx>
- User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Thunderbird/68.12.0
Sorry to hear it wasn't much help. Even though the additions I suggested
didn't help they certainly couldn't cause any harm and can't be
responsible for the drops in traffic.
As for the torutils scripts, I'm sure toralf would be able to better
investigate that but I have a feeling you have a certain set up that
might not have worked with the script. May I ask what your set up is?
Are you running your relays on separate VMs on the main system or are
you using a different set up like having all IP addresses on the same OS
and using OutboundBindAddress , routing, etc... to separate them? If I
know more, I might be able to make a script specific to your set up.
On 12/3/2022 2:07 PM, Christopher Sheats wrote:
> Hello,
>
> Thank you for this information. After 24-hours of testing, these
> configurations brought Tor to a halt.
>
> At first I started with the sysctl modifications. After a few hours
> with just that, there was no improvement in ~75%
> inet_csk_bind_conflict utilization. I then installed Torutils for both
> IPv4 and IPv6. After only a couple of hours, Tor dropped to below 15
> Mbps across both servers (40 relays). 16 hours later, Tor dropped
> below 2 Mbps.
>
> I've removed all of these new settings and restarted.
>
> --
> Christopher Sheats (yawnbox)
> Executive Director
> Emerald Onion
> Signal: +1 206.739.3390
> Website: https://emeraldonion.org/
> Mastodon: https://digitalcourage.social/@EmeraldOnion/
>
>
>
>
>> On Dec 2, 2022, at 7:30 AM, Chris <tor@xxxxxxxxxxxxxxx> wrote:
>>
>> Hi,
>>
>> As I'm sure you've already gathered, your system is maxing out trying to
>> deal with all the connection requests. When inet_csk_get_port is called
>> and the port is found to be occupied then inet_csk_bind_conflict is
>> called to resolve the conflict. So in normal circumstances you shouldn't
>> see it in perf top much less at 79%. There are two ways to deal with it,
>> and each method should be complimented by the other. One way is to try
>> to increase the number of ports and reduce the wait time which you have
>> somehow tried. I would add the following:
>>
>> net.ipv4.tcp_fin_timeout = 20
>>
>> net.ipv4.tcp_max_tw_buckets = 1200
>>
>> net.ipv4.tcp_keepalive_time = 1200
>>
>> net.ipv4.tcp_syncookies = 1
>>
>> net.ipv4.tcp_max_syn_backlog = 8192
>>
>> The complimentary method to the above is to lower the number of
>> connection requests by removing the frivolous connection requests out of
>> the equation using a few iptables rules.
>>
>> I'm assuming the increased load you're experiencing is due to the
>> current DDos attacks and I'm not sure if you're using anything to
>> mitigate that but you should consider it.
>>
>> You may find something useful at the following links
>>
>> [1](https://github.com/Enkidu-6/tor-ddos)
>>
>> [2](https://github.com/toralf/torutils)
>>
>> [background](https://gitlab.torproject.org/tpo/community/support/-/issues/40093)
>>
>> Cheers.
>>
>> On 12/1/2022 3:35 PM, Christopher Sheats wrote:
>>> Hello tor-relays,
>>>
>>> We are using Ubuntu server currently for our exit relays.
>>> Occasionally, exit throughput will drop from ~4Gbps down to ~200Mbps
>>> and the only observable data point that we have is a significant
>>> increase in inet_csk_bind_conflict, as seen via 'perf top', where it
>>> will hit 85% [kernel] utilization.
>>>
>>> A while back we thought we solved with with two /etc/sysctl.conf
>>> settings:
>>> net.ipv4.ip_local_port_range = 1024 65535
>>> net.ipv4.tcp_tw_reuse = 1
>>>
>>> However we are still experiencing this problem.
>>>
>>> Both of our (currently, two) relay servers suffer from the same
>>> problem, at the same time. They are AMD Epyc 7402P bare-metal servers
>>> each with 96GB RAM, each has 20 exit relays on them. This issue
>>> persists after upgrading to 0.4.7.11.
>>>
>>> Screenshots of perf top are shared
>>> here: https://digitalcourage.social/@EmeraldOnion/109440197076214023
>>>
>>> Does anyone have experience troubleshooting and/or fixing this problem?
>>>
>>> Cheers,
>>>
>>> --
>>> Christopher Sheats (yawnbox)
>>> Executive Director
>>> Emerald Onion
>>> Signal: +1 206.739.3390
>>> Website: https://emeraldonion.org/
>>> Mastodon: https://digitalcourage.social/@EmeraldOnion/
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> tor-relays mailing list
>>> tor-relays@xxxxxxxxxxxxxxxxxxxx
>>> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
>
_______________________________________________
tor-relays mailing list
tor-relays@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays