Threading in python?

Up until now, all of my networking projects have been in C/C++, and I’d
like
to think that I’m pretty good with threading in those languages. As with
the
previous projects, I feel like threading will be a must. It’s true that
with
wireless nodes, I won’t be sending/receiving simultaneously with the
same
node. I figure that I’ll have the node receiving unless the user inputs
a
file to be sent, so I’ll need a thread that does nothing but receive
(and
that sendswhen it needs to), and a thread that constantly looks for user
input. Should I be implementing the threading on the python level, or
can I
get away with doing this as a C++ file?

-Michael F.-

On Wed, Jun 14, 2006 at 02:59:24PM -0500, Michael F. wrote:

-Michael F.-
Michael,

You may want to look at the BBN m-block proposal (check the archives
for a link to the latest pdf). Basically everything becomes event
driven, and the need for explict thread management goes away.

Eric

Quoting M. Ford [email protected]:

-Michael F.-
A somewhat more generic answer than Eric’s:

Python and threads don’t mix easily. There’s one python
interpreter/state, and
if it’s interpreting in one thread, and you try to get it to do
something else
from another thread, things will go bad (trying to access state, running
functions, etc may all do Bad Things ™). Python 2.4 and up has the
concept
of a GIL (Global Interpreter Lock), which you must acquire before doing
anything else with the interpreter in a thread context. Alternatively
you can
dedicate one thread to synchronize access to the python interpreter. To
my
knowledge neither SWIG nor Boost.Python are aware of the GIL, so you
must
acquire/release it for them.

Additionally, python does user threads (cooperative), though they are
probably
not what you’re looking for.

-Ilia

On Thu, Jun 15, 2006 at 12:35:50AM -0400, Ilia Mirkin wrote:

Quoting M. Ford [email protected]:

A somewhat more generic answer than Eric’s:

Thanks Ilia.

Python and threads don’t mix easily. There’s one python interpreter/state,
and if it’s interpreting in one thread, and you try to get it to do something
else from another thread, things will go bad (trying to access state, running
functions, etc may all do Bad Things ™). Python 2.4 and up has the
concept of a GIL (Global Interpreter Lock), which you must acquire before doing
anything else with the interpreter in a thread context. Alternatively you
can dedicate one thread to synchronize access to the python interpreter. To my
knowledge neither SWIG nor Boost.Python are aware of the GIL, so you must
acquire/release it for them.

Actually it’s not too bad. The GIL has been around for quite a
while. Pretty sure it was before 2.4. Anyhow, if your threads are
all in python everything “just works”.

Additionally, python does user threads (cooperative), though they are
probably not what you’re looking for.

Python threads are mapped to pthreads. Unless you take action
(releasing the GIL), only one of them will run at a time. They do
“time slice” between themselves in the python interpreter.

SWIG 1.3.28 and later has an option that will have the wrapper code
drop and then reacquire the GIL. For our scheduler code, we manually
drop
the GIL headed into C++ land, and then reaquire on the way out.
That’s how we run the signal processing in one or more threads, and
the GUI in another.

If you want to see how we handle it, take a look at

gnuradio-core/src/python/gnuradio/gr/scheduler.py
gnuradio-core/src/lib/runtime/gr_single_threaded_scheduler.{h,cc,i}

Eric

On Wed, 2006-06-14 at 22:09 -0700, Eric B. wrote:

else from another thread, things will go bad (trying to access state, running
functions, etc may all do Bad Things ™). Python 2.4 and up has the
concept of a GIL (Global Interpreter Lock), which you must acquire before doing
anything else with the interpreter in a thread context. Alternatively you
can dedicate one thread to synchronize access to the python interpreter. To my
knowledge neither SWIG nor Boost.Python are aware of the GIL, so you must
acquire/release it for them.

Actually it’s not too bad. The GIL has been around for quite a
while. Pretty sure it was before 2.4. Anyhow, if your threads are
all in python everything “just works”.

You’re right, the concept of the GIL has been around for a while. But
Python 2.4 added a couple of crucial convenience functions:

PyGILState_Ensure() and PyGILState_Release()

Which allow a thread which doesn’t have (direct) access to the
interpreter pointer to still be able to run python code. This is
particularly useful if instead of spawning an interpreter the
interpreter spawns you (e.g. swig/boost.python wrapper dlopen’d by
python) and you spawn some threads that still need to deal with the
interpter.

-Ilia