[tor-bugs] #15719 [Ooni]: Handle scaling issues with OONI pipeline

["Tor Bug Tracker & Wiki" <blackhole@xxxxxxxxxxxxxx>]

#15719: Handle scaling issues with OONI pipeline
-------------------------+-------------------------
 Reporter:  hellais      |          Owner:  hellais
     Type:  enhancement  |         Status:  new
 Priority:  normal       |      Milestone:
Component:  Ooni         |        Version:
 Keywords:               |  Actual Points:
Parent ID:               |         Points:
-------------------------+-------------------------
 It looks like the new machine I acquired for hosting the OONI pipeline
 just 6 months ago has now run out of disk space.

 {{{
 585G    mongodb
 68G     pipeline
 }}}

 It appears that since we have started compressing the reports now the
 majority of the disk usage is sucked up by mongodb.

 I think this is a good opportunity to consider the migration over to
 elasticsearch as was suggested by @robertkeizer:
 https://github.com/TheTorProject/ooni-
 pipeline/commit/c20ad24e96736eb8dbebda6b23fe6ccde944b066#commitcomment-10450852

 Support for elastic search in ooni-pipeline has been implemented inside of
 https://github.com/TheTorProject/ooni-
 pipeline/compare/feature/elasticsearch.

 Once that is merged we need to consider the possibility of starting
 building an elastic search cluster for all the data.

 I propose we use this venue to discuss.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/15719>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
_______________________________________________
tor-bugs mailing list
tor-bugs@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs