I actually have 95.9121031670259% FREE which means I’m only using around 4% of my ram!MemTotal: 65777296 kBMemFree: 63088388 kB
Bleedangel,You definitely have a nice rig with 12 CPUs and 64GB of RAM. I have a bunch of 2 CPU and 256MB rigs that I loadbalance as a single relay.Following David's explanation as to how the overloaded state is metered, I believe your issue is with RAM availability being over the 3/4 rule.MemTotal: 65777296 kBMemFree: 63088388 kBYou might try freeing up RAM resources (below the 3/4 limit) and confirm whether that resolves the issue, after 72 hours.Personally, I think a better approach would be to have the upstream relay monitor and report overloaded states, but I only know so much about Tor.Hope this helps.Respectfully,GaryOn Tuesday, September 28, 2021, 10:28:42 AM PDT, Bleedangel Tor Admin <tor@xxxxxxxxxxxxxx> wrote:-----BEGIN PGP SIGNED MESSAGE-----Hash: SHA512I am having a lot of trouble figuring out why my relay keeps showing as overloaded on the search page. I believe I have more than enough memory and cpu power to not be overloaded on hardware. My server internet connection is 10GB up/down unmetered.I cannot for the life of me figure out why the relay search page continuously tells me i am overloaded.Can someone assist me in troubleshooting this?Thank you. Pertinent hardware information is pasted below:output of /proc/meminfo:--[ BEGIN PASTE ]--MemTotal: 65777296 kBMemFree: 63088388 kBMemAvailable: 63415088 kBBuffers: 180096 kBCached: 736396 kBSwapCached: 0 kBActive: 449428 kBInactive: 1729304 kBActive(anon): 14552 kBInactive(anon): 1292048 kBActive(file): 434876 kBInactive(file): 437256 kBUnevictable: 0 kBMlocked: 0 kBSwapTotal: 33520636 kBSwapFree: 33520636 kBDirty: 236 kBWriteback: 0 kBAnonPages: 1262300 kBMapped: 273164 kBShmem: 48592 kBKReclaimable: 94828 kBSlab: 204624 kBSReclaimable: 94828 kBSUnreclaim: 109796 kBKernelStack: 5040 kBPageTables: 9308 kBNFS_Unstable: 0 kBBounce: 0 kBWritebackTmp: 0 kBCommitLimit: 66409284 kBCommitted_AS: 2374432 kBVmallocTotal: 34359738367 kBVmallocUsed: 34348 kBVmallocChunk: 0 kBPercpu: 16384 kBHardwareCorrupted: 0 kBAnonHugePages: 0 kBShmemHugePages: 0 kBShmemPmdMapped: 0 kBFileHugePages: 0 kBFilePmdMapped: 0 kBCmaTotal: 0 kBCmaFree: 0 kBHugePages_Total: 0HugePages_Free: 0HugePages_Rsvd: 0HugePages_Surp: 0Hugepagesize: 2048 kBHugetlb: 0 kBDirectMap4k: 332988 kBDirectMap2M: 7981056 kBDirectMap1G: 58720256 kB--[ END PASTE ]--output of /proc/cpuinfo:--[ BEGIN PASTE ]--processor : 0vendor_id : AuthenticAMDcpu family : 23model : 113model name : AMD Ryzen 5 3600 6-Core Processorstepping : 0microcode : 0x8701021cpu MHz : 2200.000cache size : 512 KBphysical id : 0siblings : 12core id : 0cpu cores : 6apicid : 0initial apicid : 0fpu : yesfpu_exception : yescpuid level : 16wp : yesflags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_esbugs : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypassbogomips : 7202.22TLB size : 3072 4K pagesclflush size : 64cache_alignment : 64address sizes : 43 bits physical, 48 bits virtualpower management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]processor : 1vendor_id : AuthenticAMDcpu family : 23model : 113model name : AMD Ryzen 5 3600 6-Core Processorstepping : 0microcode : 0x8701021cpu MHz : 2200.000cache size : 512 KBphysical id : 0siblings : 12core id : 1cpu cores : 6apicid : 2initial apicid : 2fpu : yesfpu_exception : yescpuid level : 16wp : yesflags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_esbugs : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypassbogomips : 7202.22TLB size : 3072 4K pagesclflush size : 64cache_alignment : 64address sizes : 43 bits physical, 48 bits virtualpower management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]processor : 2vendor_id : AuthenticAMDcpu family : 23model : 113model name : AMD Ryzen 5 3600 6-Core Processorstepping : 0microcode : 0x8701021cpu MHz : 2200.000cache size : 512 KBphysical id : 0siblings : 12core id : 2cpu cores : 6apicid : 4initial apicid : 4fpu : yesfpu_exception : yescpuid level : 16wp : yesflags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuidextd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilterpfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_esbugs : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypassbogomips : 7202.22TLB size : 3072 4K pagesclflush size : 64cache_alignment : 64address sizes : 43 bits physical, 48 bits virtualpower management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]processor : 3vendor_id : AuthenticAMDcpu family : 23model : 113model name : AMD Ryzen 5 3600 6-Core Processorstepping : 0microcode : 0x8701021cpu MHz : 2199.014cache size : 512 KBphysical id : 0siblings : 12core id : 4cpu cores : 6apicid : 8initial apicid : 8fpu : yesfpu_exception : yescpuid level : 16wp : yesflags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuidextd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilterpfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_esbugs : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypassbogomips : 7202.22TLB size : 3072 4K pagesclflush size : 64cache_alignment : 64address sizes : 43 bits physical, 48 bits virtualpower management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]processor : 4vendor_id : AuthenticAMDcpu family : 23model : 113model name : AMD Ryzen 5 3600 6-Core Processorstepping : 0microcode : 0x8701021cpu MHz : 2200.000cache size : 512 KBphysical id : 0siblings : 12core id : 5cpu cores : 6apicid : 10initial apicid : 10fpu : yesfpu_exception : yescpuid level : 16wp : yesflags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuidextd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilterpfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_esbugs : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypassbogomips : 7202.22TLB size : 3072 4K pagesclflush size : 64cache_alignment : 64address sizes : 43 bits physical, 48 bits virtualpower management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]processor : 5vendor_id : AuthenticAMDcpu family : 23model : 113model name : AMD Ryzen 5 3600 6-Core Processorstepping : 0microcode : 0x8701021cpu MHz : 2200.000cache size : 512 KBphysical id : 0siblings : 12core id : 6cpu cores : 6apicid : 12initial apicid : 12fpu : yesfpu_exception : yescpuid level : 16wp : yesflags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuidextd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilterpfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_esbugs : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypassbogomips : 7202.22TLB size : 3072 4K pagesclflush size : 64cache_alignment : 64address sizes : 43 bits physical, 48 bits virtualpower management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]processor : 6vendor_id : AuthenticAMDcpu family : 23model : 113model name : AMD Ryzen 5 3600 6-Core Processorstepping : 0microcode : 0x8701021cpu MHz : 2200.000cache size : 512 KBphysical id : 0siblings : 12core id : 0cpu cores : 6apicid : 1initial apicid : 1fpu : yesfpu_exception : yescpuid level : 16wp : yesflags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuidextd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilterpfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_esbugs : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypassbogomips : 7202.22TLB size : 3072 4K pagesclflush size : 64clflush size : 64cache_alignment : 64address sizes : 43 bits physical, 48 bits virtualpower management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]processor : 7vendor_id : AuthenticAMDcpu family : 23model : 113model name : AMD Ryzen 5 3600 6-Core Processorstepping : 0microcode : 0x8701021cpu MHz : 2199.809cache size : 512 KBphysical id : 0siblings : 12core id : 1cpu cores : 6apicid : 3initial apicid : 3fpu : yesfpu_exception : yescpuid level : 16wp : yesflags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuidextd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilterpfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_esbugs : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypassbogomips : 7202.22TLB size : 3072 4K pagesclflush size : 64cache_alignment : 64address sizes : 43 bits physical, 48 bits virtualpower management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]processor : 8vendor_id : AuthenticAMDcpu family : 23model : 113model name : AMD Ryzen 5 3600 6-Core Processorstepping : 0microcode : 0x8701021cpu MHz : 2200.000cache size : 512 KBphysical id : 0siblings : 12core id : 2cpu cores : 6apicid : 5initial apicid : 5fpu : yesfpu_exception : yescpuid level : 16wp : yesflags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuidextd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilterpfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_esbugs : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypassbogomips : 7202.22TLB size : 3072 4K pagesclflush size : 64cache_alignment : 64address sizes : 43 bits physical, 48 bits virtualpower management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]processor : 9vendor_id : AuthenticAMDcpu family : 23model : 113model name : AMD Ryzen 5 3600 6-Core Processorstepping : 0microcode : 0x8701021cpu MHz : 2199.644cache size : 512 KBphysical id : 0siblings : 12core id : 4cpu cores : 6apicid : 9initial apicid : 9fpu : yesfpu_exception : yescpuid level : 16wp : yesflags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuidextd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilterpfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_esbugs : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypassbogomips : 7202.22TLB size : 3072 4K pagesclflush size : 64cache_alignment : 64address sizes : 43 bits physical, 48 bits virtualpower management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]processor : 10vendor_id : AuthenticAMDcpu family : 23model : 113model name : AMD Ryzen 5 3600 6-Core Processorstepping : 0microcode : 0x8701021cpu MHz : 2200.000cache size : 512 KBphysical id : 0siblings : 12core id : 5cpu cores : 6apicid : 11initial apicid : 11fpu : yesfpu_exception : yescpuid level : 16wp : yesflags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuidextd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilterpfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_esbugs : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypassbogomips : 7202.22TLB size : 3072 4K pagesclflush size : 64cache_alignment : 64address sizes : 43 bits physical, 48 bits virtualpower management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]processor : 11vendor_id : AuthenticAMDcpu family : 23model : 113model name : AMD Ryzen 5 3600 6-Core Processorstepping : 0microcode : 0x8701021cpu MHz : 2200.000cache size : 512 KBphysical id : 0siblings : 12core id : 6cpu cores : 6apicid : 13initial apicid : 13fpu : yesfpu_exception : yescpuid level : 16wp : yesflags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilterpfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_esbugs : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypassbogomips : 7202.22TLB size : 3072 4K pagesclflush size : 64cache_alignment : 64address sizes : 43 bits physical, 48 bits virtualpower management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]--[END PASTE]--‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐On Tuesday, September 28th, 2021 at 11:31 AM, Gary C. New via tor-relays <tor-relays@xxxxxxxxxxxxxxxxxxxx> wrote:> David,>> This is exactly the type of information I was hoping for. You should make this an article and link it to the overloaded support page.>> I guess I assumed that Tor preformed external timeout monitoring apposed to relay reported resource monitoring.>> It's interesting that you mention loadbalancing Tor as that is precisely what my recent efforts have been geared toward.>> I'm fairly confident that my last overloaded state was due to migrating one of my Tor relay nodes onto a previously provisioned BotFarm node and forgetting to kill the existing bot processes; thus, having competing resources. I can confirm that when loadbalancing Tor relay nodes that the whole is only as good as the weakest link; thus, it's important to have identical Tor relay nodes to evenly distribute circuits and maintain consensus.>> In this paradigm, I was hoping to be able to define a timeout value associated with the overloaded state and tune the loadbalancer to redistribute to different upstream nodes should a Tor relay node reach such a value. However, it seems this is a moot point, after reading your summary of the reporting process.>> At present, I have the upstream, loadbalancing timeout values disabled and let the Tor nodes build or teardown circuits based on available resources per node. I do see spikes alternate through various nodes throughout the day. It would be nice to find an upstream timeout value to better manage those spikes.>> Any recommendations would be greatly appreciated.>> Respectfully,>> Gary>> P.S. This is all being done on ASUSWRT-Merlin using AiMesh nodes, but isn't limited to that architecture. I hope to publish a tutorial, after ironing out all the kinks.>> On Tuesday, September 28, 2021, 7:01:04 AM MDT, David Goulet <dgoulet@xxxxxxxxxxxxxx> wrote:>> On 27 Sep (14:23:34), Gary C. New via tor-relays wrote:>> > George,>> > The referenced support article provides recommendations as to what might be>> > causing the overloaded state, but it doesn't provide the metric(s) for how>> > Tor decides whether a relay is overloaded. I'm trying to ascertain the>> > later. I would assume the overloaded state metric(S) is/are a maximum>> > timeout value and/or reoccurrence value, etc. By knowing what the>> > overloaded state metric is, I can tune my Tor relay just short of it. Thank>> > you for your reply. Respectfully,>> Hi Gary!>> I'll try to answer the best I can from what we've have worked on for these>> overload metrics.>> Essentially, there few places within a Tor relay that we can easily notice an>> "overloaded" state. I'll list them and tell you how we decide:>> 1. Out-Of-Memory invocation>> Tor has its own OOM and it is invoked when 75% of the total memory tor>> thinks it can use is reached. Thus, let say tor thinks it can use 2GB in>> total then at 1.5GB of memory usage, it will start freeing memory. That, is>> considered an overload state.>> Now the real question here is what is the memory "tor thinks" it has.>> Unfortunately, it is not the greatest estimation but that is what it is.>> When tor starts, it will use MaxMemInQueues for that value or will look at>> the total RAM available on the system and apply this algorithm:>> if RAM >= 8GB {>> memory = RAM * 40%>> } else {>> memory = RAM * 75%>> }>> /* Capped. */>> memory = min(memory, 8GB) -> [8GB on 64bit and 2GB on 32bit)>> /* Minimum value. */>> memory = max(250MB, memory)>> Why we picked those numbers, I can't tell you that, these come from the very>> early days of the tor software and I can't tell you why.>> And so to avoid such overload state, clearly run a relay above 2GB of RAM on>> 64bit should be the bare minimum in my opinion. 4GB would be much much>> better. In DDoS circumstances, there is a whole lot of memory pressure.>> One keen observer can notice that this approach also has the problem that it>> doesn't shield tor from being called by the OS OOM itself. Reason is that>> because we take the total memory on the system when tor starts, if the>> overall system has many other applications running using RAM, we end up>> eating too much memory and the OS could OOM tor without tor even noticing>> memory pressure. Fortunately, this is not a problem affecting the overload>> status situation.>> 2. Onionskins processing>> Tor is sadly single threaded _except_ for when the "onion skins" are>> processed that is the cryptographic work that needs to be done on the famous>> "onion layers" in every circuits.>> For that we have a thread pool and outsource all of that work to that pool.>> It can happen that this pool starts dropping work due to back pressure and>> that in turn is an overload state.>> Why this can happen, essentially CPU pressure. If your server is running at>> capacity and it is not only your tor, then this is likely to trigger.>> 3. DNS Timeout>> This applies only to Exits. If tor starts noticing DNS timeouts, you'll get>> the overload flag. This might not be because your relay is overloaded in>> terms of resources but it signals a problem on the network.>> And DNS timeouts at the Exits are a _huge_ UX problem for tor users and so>> Exit operators really need to be on top of those to help. It is not clear>> with the overload line but at least if an operator notices the overload>> line, it can then investigate DNS timeouts in case there is no resources>> pressure.>> 4. TCP port exhaustion>> This should be extremely rare though. The idea about this one is that you>> ran out of TCP ports which is a range that is usually, on Linux, 32768-60999>> and so having that many connections would lead to the overload state.>> However, I think (I might be wrong though) that nowadays this range is per>> source IP and not process wide so it would likely have to be deliberate from>> someone to put your relay in that state.>> There are two other overload lines that tor relay report:>> "overload-ratelimits" and "overload-fd-exhausted" but they are not used yet>> for the overload status on Metrics. But you can find them in your relay>> descriptor[0] if you are curious.>> They are about when your relay reaches its connection global limit too often>> and when your relay runs out of file descriptors.>> Hope this helps but overall as you can see, a lot of factor can influence>> these metrics and so the ideal ideal ideal situation for a tor relay is that>> it runs alone on a fairly good machine. Any kinds of pullback from a tor relay>> like being overloaded has cascading effects on the network both in terms of UX>> but also in terms of load balancing which tor is not yet very good at (but we>> are working on hard on making it much better!!).>> Cheers!>> David>> [0]>>> -->> +7Xz1XCshqTyudrO7K4kGBEl+NghDNbqiTGYZpSOw4U=>> _______________________________________________>> tor-relays mailing list>>-----BEGIN PGP SIGNATURE-----Version: ProtonMailwnUEARYKAAYFAmFTUJgAIQkQ07hp34FMyg4WIQSgEMifmnN5ohFWvfnTuGnfgUzKDjtGAP0VaRiV4ZWyG5h1cjxAKHq4pV5j33cXbD92wHt9hSjIwgEAjotywI1YdgZe3frT8wRj43Tm8MF/f2p/mytGzJHySAc==aXqg-----END PGP SIGNATURE-----
Attachment:
publicKey - tor@bleedangel.com - a010c89f.asc
Description: application/pgp-keys
Attachment:
signature.asc
Description: OpenPGP digital signature
_______________________________________________ tor-relays mailing list tor-relays@xxxxxxxxxxxxxxxxxxxx https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays