[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]

Re: [tor-relays] diff-cache OOM

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Sunday, January 27, 2019 5:37 PM, teor <teor@xxxxxxxxxx> wrote:

> On January 27, 2019 1:07:12 PM UTC, notatorservernotatorserver@xxxxxxxxxxxxxx wrote:
>
> > I am running tor 4.0.1-alpha inside a docker container with a memory
> > limit (6GB). Tor runs out of memory and aborts or crashes periodically.
>
> Tor assumes that 64-bit machines have 8 GB of RAM, unless the relevant APIs return a lower value.
>
> How does docker implement memory limits?
> Does it modify the values returned by the Linux RAM APIs?
>

Docker sets up a cgroup with a memory limit. I guess you could check the value: `cat /sys/fs/cgroup/memory/memory.limit_in_bytes`.
The value reported in `cat /proc/meminfo` is the same for every container, i.e. host system value. So no, it does not modify the values.


> > Usually I see an error message like this:
> > Could not mmap file "/var/lib/tor/data/diff-cache/1149": Out of memory
> > ...(repeated some number of times)
>
> Please paste your log onhttps://paste.debian.net

here's a sample of logs: http://paste.debian.net/1062943/
I can capture some more if these are helpful. Retention period is short


>
> How many times are these messages repeated?
Depends but it can be anywhere from 1 to ~20.

> Is the diff number the same, or different?
diff number is increasing

> How many files are in that directory?
>
currently 259 files (~123MB)

> > followed by segfault
> > Sometimes I see a message:
> > Out of memory on malloc(). Dying
> > followed by abort.
> > Am I correct to assume the diff-cache is the issue here? Looking at the
> > files it seems they are all pretty small (~500K). Is some badly behaved
> > client requesting 12,000 of these diffs causing my relay to mmap them
> > all at once?
>
> How many times are these messages repeated?
>
> > or is it just expensive to generate and generating 30-60
> > at once is enough to use all the memory?
>
> The diffs are generated once an hour, then cached for requests.
> Do the errors happen at generation time, or request time?

I guess at generation time if there is no way for a client to request to generate many diffs.

Looking at mtime of these files it seems they are created in bursts:
/ # stat /var/lib/tor/data/diff-cache/* | grep Modify | sort | uniq -c
...
     24 Modify: 2019-01-28 17:15:30.000000000
      6 Modify: 2019-01-28 17:15:31.000000000
     15 Modify: 2019-01-28 17:15:32.000000000
      6 Modify: 2019-01-28 17:15:33.000000000
     12 Modify: 2019-01-28 17:15:34.000000000
     21 Modify: 2019-01-28 17:15:35.000000000
      1 Modify: 2019-01-28 17:20:27.000000000
      2 Modify: 2019-01-28 18:03:27.000000000
     21 Modify: 2019-01-28 18:03:28.000000000
      6 Modify: 2019-01-28 18:03:29.000000000
      9 Modify: 2019-01-28 18:03:30.000000000
     18 Modify: 2019-01-28 18:03:31.000000000
      3 Modify: 2019-01-28 18:03:32.000000000
      6 Modify: 2019-01-28 18:03:33.000000000
     18 Modify: 2019-01-28 18:03:34.000000000
     18 Modify: 2019-01-28 18:03:35.000000000




>
> > Are there any config options to reject expensive queries or otherwise
> > limit concurrency?
>
> Try
> MaxMemInQueues 4 GB
>

I already have max mem set to 5GB but I will set to 4GB.

> If that doesn't work, try
> NumCPUs 2
> and please let us know, because we'd like to fix these kinds of bugs.
>

I will try this also.

Thanks

> T
>
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> teor
>
> -----
>
> tor-relays mailing list
> tor-relays@xxxxxxxxxxxxxxxxxxxx
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays


_______________________________________________
tor-relays mailing list
tor-relays@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays