Re: [tor-bugs] #2947 [Torflow]: BwScanner does not clear stream data between slices

["Tor Bug Tracker & Wiki" <torproject-admin@xxxxxxxxxxxxxx>]

#2947: BwScanner does not clear stream data between slices
---------------------+------------------------------------------------------
 Reporter:  aagbsn   |          Owner:  mikeperry
     Type:  defect   |         Status:  new      
 Priority:  normal   |      Milestone:           
Component:  Torflow  |        Version:           
 Keywords:           |         Parent:           
   Points:           |   Actualpoints:           
---------------------+------------------------------------------------------

Comment(by aagbsn):

 Update:

 1. We suspected that this bug was the cause of BwAuthority's memory leak
 problems as items were not being cleared after each slice completed.
 However, after fixing this issue the memory leaks persisted.

 2. mikeperry and I decided that refactoring BwAuthority as a parent-child
 pair of processes would ensure that memory leaks would not persist past
 each run. Basically, rather than 1 long-running process, we split
 bwauthority.py into a pair of processes: one parent process that was
 responsible for passing slice parameters to a child process that would
 actually scan that slice.

 3. After refactoring BwAuthority, we discovered more issues -- in some
 cases circuit_status_events referenced Router objects that were not stored
 in the database (SQLAlchemy raised sqlalchemy.orm.exc.NoResultFound) even
 though update_consensus() and _update_db() had supposedly inserted the
 referenced Router objects. To restate: objects that were supposedly stored
 in the database were failing in queries only seconds later.

 4. This issue happens pretty rarely; my best estimate is about 2 weeks or
 so...

 5. This is likely caused by a race between the sessions bound to Elixir
 models and another shared session (tc_session).

 6. I believe this race occurs because of misuse of SQLAlchemy
 scoped_sessions. I refactored BwAuthority to use local sessions for each
 function in torctl/SQLSupport.py that accesses the database, and call
 tc_session.remove() prior to returning, as is recommended here:
 http://www.sqlalchemy.org/docs/orm/session.html#contextual-thread-local-
 sessions . The effect should be to flush mapped objects from the local
 session to the database so that queries from the Elixir bound sessions
 will succeed.

 7. Now we wait and see if this race condition persists :-(.

-- 
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/2947#comment:3>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
_______________________________________________
tor-bugs mailing list
tor-bugs@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs