http - How to Speed Up Python's urllib2 when doing multiple requests

Question

I am making several http requests to a particular host using python's urllib2 library. Each time a request is made a new tcp and http connection is created which takes a noticeable amount of time. Is there any way to keep the tcp/http connection alive using urllib2?

Answer 1

If you switch to httplib, you will have finer control over the underlying connection.

For example:

import httplib

conn = httplib.HTTPConnection(url)

conn.request('GET', '/foo')
r1 = conn.getresponse()
r1.read()

conn.request('GET', '/bar')
r2 = conn.getresponse()
r2.read()

conn.close()

This would send 2 HTTP GETs on the same underlying TCP connection.