Hello guys,
I had a problem and currently I'm not able to solve it. So, here I am ;) I have a python script that uses python-stem to create and handle a tor instance (on a defined port). What it does is retrieving (using a HTTP GET) a web page and submitting information (using HTTP POST messages).
Basically i use tor because I need to test this server from different IP addresses with more requests in parallel. What I also do is keeping trace of Cookies. Here's a sample of the code I use, based on the example on stem website
https://stem.torproject.org/tutorials/to_russia_with_love.html (to have more parallel requests, i launch the script many times with different socks_port value):
----------------------------
import socket, socks, stem.process
import mechanize, cookielib
SOCKS_PORT = 9000
DATA_DIRECTORY = "TOR_%s" % SOCKS_PORT
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, '127.0.0.1', SOCKS_PORT)
socket.socket = socks.socksocket
tor_process = stem.process.launch_tor_with_config(
config = {
'SocksPort': str(SOCKS_PORT),
'ControlPort': str(SOCKS_PORT+1),
'DataDirectory': DATA_DIRECTORY,
'ExitNodes': '{it}',
},
)
# initialize python mechanize, with cookies (it works exactly like urllib2, urllib3, etc. already tried...)
br = mechanize.Browser()
cj = cookielib.LWPCookieJar()
br.set_cookiejar(cj)
...
for number in num_list:
req = br.open_novisit("
http://example.com") #_1_
res = req.read()
print res
req.close()
req2 = br.open("
http://example.com/post_to_me", data_to_post) #_2_
res2 = req2.read()
req2.close()
--------------------------------
And that's it. The problem occurs on the lines i marked as _1_ and _2_: basically when it reaches around 200 requests, it seems to block undefinitely, waiting for a response that never comes. Of course, wiresharking doesn't work because it's encrypted. The same stuff, without TOR, works perfectly. So, why does it stuck at about 200 requests!? I tried to:
1. Telnet on control port, forcing to renew circuits with SIGNAL NEWNYM
2. instantiating mechanize (urllib2, 3, whatever) in the loop
3. ...i don't remember what else
I thought it could be a local socket connection limit: actually without TOR, i see in wireshark the source port changes every time a request is performed. But actually i don't know if the problem is in using the same source port every time (but i don't think so) and if so, should I close the current socket and open a new one? Should I kill the tor process? I can't exaplain myself why...
What I only know is: *when the script stucks, if i kill the python process (ctrl+c) and then re-launch, it starts working again.*. I've seen that it's possible to set the value of TrackHostExitsExpire, is it useful in my case?
Thanks in advance to whoever can help me!!
Ed