Python Grequests + multiprocessing
I'm trying to make async http calls using grequests from within a Pool
from the multiprocessing library. I'm encountering a problem that suggests
Grequests and multiprocessing are possibly incompatible with each other;
specifically, calling monkey.patch_all() messes with Pool creation.
Initially, without calling monkey.patch_all() in my code:
from gevent import monkey
monkey.patch_all()
I get these two errors:
NotImplementedError: gevent is only usable from a single thread
and
requests.exceptions.ConnectionError: None: Max retries exceeded with url:
SOME_URL (Caused by redirect)
Calling monkey.patch_all() fixes the above errors, but causes my code to
hang at:
p = Pool(THREAD_POOL_SIZE)
Not calling monkey.patch_all() leads to my Pool created successfully.
Calling monkey.patch_all(thread=False, socket=False) leads to my Pool
created successfully, too, but doesn't solve the initial two errors.
No comments:
Post a Comment