[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]
Re: [tor-bugs] #24368 [Core Tor/Tor]: Tune zstd parameters to decrease memory usage during streaming
#24368: Tune zstd parameters to decrease memory usage during streaming
-------------------------------------------------+-------------------------
Reporter: teor | Owner: ahf
Type: defect | Status:
| assigned
Priority: Medium | Milestone: Tor:
| 0.3.4.x-final
Component: Core Tor/Tor | Version: Tor:
| 0.3.1.1-alpha
Severity: Normal | Resolution:
Keywords: regression, compression, zstd, tor- | Actual Points:
dir, 032-backport |
Parent ID: | Points: 1
Reviewer: | Sponsor:
-------------------------------------------------+-------------------------
Changes (by nickm):
* milestone: Tor: 0.3.3.x-final => Tor: 0.3.4.x-final
Comment:
I've run some initial experimentation, and here's what I found:
Adjusting the pre-set values shouldn't be necessary if instead we tell
zstd to build its own parameters (using `ZSTD_getCParams()` or
`ZSTD_getParams`), with the `estimatedSrcSize` argument to tell zstd how
big we expect the input to be.
I also think that our current estimates are higher than zstd actually
uses, which is a good thing. I'm attaching a python script that I used
for these tests; it requires the "zstandard" package.
Now the catch here is that we can't actually adjust the parameters to
anything besides the presets unless we use the "advanced" (a.k.a "static-
only" zstd APIs). I've opened ticket #25162 about doing that safely. But
the complexity is enough that I think we should call this an 0.3.4.x
ticket: it is more than simply tweaking a couple of numbers.
--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/24368#comment:6>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
_______________________________________________
tor-bugs mailing list
tor-bugs@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs