Making gnuradio blocks entirely in python

So there is a useful feature I think gnuradio is missing, and thats the
ability to rapidly prototype signal processing in python. It would be
great if a python block would have access to all the facilities
available to a block written in a c++ environment.

So I put together pyblock (not to be confused with those other project
by the same name). Its a stand-alone build that links against gnuradio.
https://github.com/guruofquality/pyblock

Theres a few demos: an adder, using tags, interp and decim blocks.
https://github.com/guruofquality/pyblock/tree/master/examples

The interesting thing is that the work functions just call into numpy
routines, so there is a chance that the implementations can be
reasonably fast.

I would like to be able to support history so i can implement a filter
using numpy but I am a little lacking in the basic understanding so fill
me in if you know. Suppose I have this work function:

int work(int noutput_items, gr_vector_const_void_star
&input_items,gr_vector_void_star &output_items){
const float *in = reinterpret_cast<const float *>(input_items[0]);

I am assuming history is on the input buffer.
Is the vector “in” ninput_items + history() items long? Where
ninput_items = noutput_items*/some_factor.
Whats the first index of the first item? Is in[0] or in[0-history()]?

-Josh

The “in” buffer should start at in[0] the history comes into play at
allowing enough valid indices to perform the filtering operation over
the entire buffer which allows N valid output data versus N-nTAPS. When
I looked at filtering in gnuradio the way I thought about it as and at
the end I’ve included some ASCII art to demonstrate it, from the
illustration you can see how the history comes into play. I’m no DSP
engineer but this seems to work for me someone can feel free to correct
me if I’m wrong.

< adapted from gr_fir_fff_generic.cc>

for(i=0; i<n; i++)
for(j=0; j< ntaps; j++)
out[i] += taps[j] * input[i+j]

0 1 2 3 4 5 6 7 8 9
hist0 hist1 hist2 hist3 in0 in1 in2 in3 in4 in5
tap3 tap2 tap1 tap0

0 1 2 3 4 5 6 7 8 9
hist0 hist1 hist2 hist3 in0 in1 in2 in3 in4 in5
tap3 tap2 tap1 tap0

0 1 2 3 4 5 6 7 8 9
hist0 hist1 hist2 hist3 in0 in1 in2 in3 in4 in5
tap3 tap2 tap1 tap0

0 1 2 3 4 5 6 7 8 9
hist0 hist1 hist2 hist3 in0 in1 in2 in3 in4 in5
tap3 tap2 tap1 tap0

0 1 2 3 4 5 6 7 8 9
hist0 hist1 hist2 hist3 in0 in1 in2 in3 in4 in5
tap3 tap2 tap1 tap0

0 1 2 3 4 5 6 7 8 9
hist0 hist1 hist2 hist3 in0 in1 in2 in3 in4 in5
tap3 tap2 tap1 tap0

0 1 2 3 4 5 6 7 8 9
hist0 hist1 hist2 hist3 in0 in1 in2 in3 in4 in5
tap3 tap2 tap1
tap0

al fayez

On Sun, Sep 25, 2011 at 10:58 PM, Josh B. [email protected] wrote:

https://github.com/guruofquality/pyblock/tree/master/examples
&input_items,gr_vector_void_star &output_items){
const float *in = reinterpret_cast<const float *>(input_items[0]);

I am assuming history is on the input buffer.
Is the vector “in” ninput_items + history() items long? Where
ninput_items = noutput_items*/some_factor.
Whats the first index of the first item? Is in[0] or in[0-history()]?

-Josh

Josh,
This is REALLY cool, thanks!

Now, here’s the question. Do we keep this as a separate project or put
it
into GNU Radio proper?

If you want to keep it separate and on github, we could at least clone
it on
gnuradio.org and have a redmine projects page for it. Or even just a
link if
you don’t want to worry about updating and supporting the redmine
interface
and feel that github gives you everything you need.

Part of me wants to put this into GNU Radio, just because it’s a nice
feature for developers, especially those not comfortable developing code
in
C++ going back and forth between GNU Radio and Python (which I’ve often
done
in the past).

On the other hand, part of me doesn’t want it in GNU Radio because I
don’t
want people to start using it as the default way of doing things,
specifically, that they don’t start with a pyblock and never move it
into a
proper C++ block. In my experience with numpy (and scipy), they are
great
interfaces and they are faster than using Python, but they are still
much
slower than they could/should be. So we shouldn’t be relying on
Python-level
stuff for real implementations.

On the other hand (I think I’ve run out of hands…), this also goes a
long
way to solving the problem of people wanting a Matlab interface system.
If
you could work in using Matplotlib into your examples, people might see
how
much they can do in Python that replicates/replaces/supersedes Matlab
for
these purposes.

Again, great stuff. Let us know how you want to continue to support it.

Tom

Now, here’s the question. Do we keep this as a separate project or put it
into GNU Radio proper?

Id like it too see more testing. But I hope to get it in gnuradio.

The implementation is relying on a hairy thing like swig directors.
Shiver…

If you want to keep it separate and on github, we could at least clone it on
gnuradio.org and have a redmine projects page for it. Or even just a link if
you don’t want to worry about updating and supporting the redmine interface
and feel that github gives you everything you need.

I thought we were transitioning to github so I dont even know anymore.

On the other hand, part of me doesn’t want it in GNU Radio because I don’t
want people to start using it as the default way of doing things,
specifically, that they don’t start with a pyblock and never move it into a
proper C++ block. In my experience with numpy (and scipy), they are great
interfaces and they are faster than using Python, but they are still much
slower than they could/should be. So we shouldn’t be relying on Python-level
stuff for real implementations.

All the tag qa code could be in python. That could make a developers
life easier.

Id like to point out that the adder implemented w/ numpy could be
performing arithmetic faster because thats vectorized and until we use
volk, the adder in gnuradio C++ is not.

The real overhead probably comes from the memory allocations that happen
going between python and C++ and making python objects. I havent
benchmarked it, but I am guessing its non-trivial.

On the other hand (I think I’ve run out of hands…), this also goes a long
way to solving the problem of people wanting a Matlab interface system. If
you could work in using Matplotlib into your examples, people might see how
much they can do in Python that replicates/replaces/supersedes Matlab for
these purposes.

Again, great stuff. Let us know how you want to continue to support it.

Yes:

I want to support proper message passing. I noticed that I can register
a message handler to receive messages. Can you point me to how messages
are produced?

I really hate gr_tag_info.h. I would rather get_tags_in_range filled a
vector of type tag_type, where tag_type has methods like .get_key(),
.get_value(), etc… Basically im complaining that we didnt use object
orientedness here when its realy well suited for it. You will see that I
did this wrap-around in my pyblock_gateway so I didnt have to swig
tag_info. See the tags_demo.py to see how I used it.

_josh

On Mon, Sep 26, 2011 at 05:24, Tom R. [email protected]
wrote:

On the other hand (I think I’ve run out of hands…)

…that would be “on the gripping hand.”

(obscure SF reference of the week.)

Johnathan

On Mon, Sep 26, 2011 at 09:05, Josh B. [email protected] wrote:

The implementation is relying on a hairy thing like swig directors.
Shiver…

We’ve had mixed results with swig directors in the past. The bigger
issue here, though is the Python global interpreter lock. This is
released by the GNU Radio runtime in top_block.start() and
top_block.run() so that the Python calling thread can proceed after
GNU Radio kicks off the flowgraph threads.

When calling back up to Python, that lock has to be re-acquired, and
all other Python threads in operation (including GUI threads for
wxPython and QTGui) will block waiting for it to be released.

So while this would work in theory, it could create a nasty coupling
in application performance between signal processing code and GUI
code, and possibly even deadlocks if the non-GNU Radio Python code
needed to interact with the Python block.

Fortunately, you’ve done this in a way that can be tested without any
modifications to GNU Radio, so we can get some empirical results.

I think a more useful area for C+±>Python communication is to make it
easy for a GNU Radio block to notify a Python object that something
has happened, possibly with data attached.

For example, in the uhd_fft.py and other GUI programs that use the fft
sink, we have to end the GNU Radio flowgraph proper in a message
sink/message queue, and then have a Python thread run this queue for
entries, process them, then post the processed FFT frames to
wxPython’s GUI thread. This is not only inefficient, but hard to
generalize from or even just learn from by inspection. I created the
gru.msgq_runner class to help this, but it’s still a hack.

So a mechanism that takes an arbitrary pmt on the GNU Radio C++ end
and results in a function call to a Python bare function or class
method with that pmt as the argument would create a generic method to
pass events of all sorts up from the “data plane” portion of the app
to the “control plane”, non-GNU Radio code.

Johnathan

On 09/26/2011 09:22 AM, Johnathan C. wrote:

modifications to GNU Radio, so we can get some empirical results.
generalize from or even just learn from by inspection. I created the
gru.msgq_runner class to help this, but it’s still a hack.

So a mechanism that takes an arbitrary pmt on the GNU Radio C++ end
and results in a function call to a Python bare function or class
method with that pmt as the argument would create a generic method to
pass events of all sorts up from the “data plane” portion of the app
to the “control plane”, non-GNU Radio code.

So a little bit about the implementation I used: The communication
between pyblock_gateway.cc and python is actually message based.

When I made the announcement, the implementation used a python thread to
block on a call, get a message, and dispatch work. This worked, but I
was kind of annoyed dealing with the python thread and getting it to
exit. And its definitely more ideal to have the scheduler call directly
into this python work(). So last night I changed the implementation to
use a SWIG director. So the python handler is called to read the message
and dispatch work.

  • In the case of the thread implementation, you need to take the GIL
    locking into account. Basically, I did the same thing as the message
    queue blocking calls.

  • In the case of the director, the GIL locking also has to be taking
    into account. So, I grabbed the GIL locker implementation from gr_feval.

So basically, any implementation has to deal with unlocking the GIL
properly. I would like to see how well this current implementation works
for people. I think its the right way to do it, but SWIG directors still
seem like black magic so I am less trustworthy of them.

I will add a throw exception when the default handler gets called (means
something about directors didnt build right on your machine).

_josh

On Mon, Sep 26, 2011 at 12:05 PM, Josh B. [email protected] wrote:

If you want to keep it separate and on github, we could at least clone it
on
gnuradio.org and have a redmine projects page for it. Or even just a
link if
you don’t want to worry about updating and supporting the redmine
interface
and feel that github gives you everything you need.

I thought we were transitioning to github so I dont even know anymore.

Sort of, yes, and no… we’ll still have everything on gnuradio.org as a
permanent repo that’s cloned on github. For one thing, I was looking
into
Redmine, and I’m not sure it can handle a project that’s not local to
the
file system; so we have to have a local clone.

In this case, if it’s kept as a separate project, we would clone it on
gnuradio.org and have a project page set up for it and periodically
refresh
it from github.

All the tag qa code could be in python. That could make a developers
life easier.

Id like to point out that the adder implemented w/ numpy could be
performing arithmetic faster because thats vectorized and until we use
volk, the adder in gnuradio C++ is not.

Maybe… as I’ve said, I haven’t been hugely impressed with the speed of
numpy/scipy in many cases. For the adder, you might be right due to the
vectorization.

The real overhead probably comes from the memory allocations that happen
going between python and C++ and making python objects. I havent
benchmarked it, but I am guessing its non-trivial.

True.

Yes:

I want to support proper message passing. I noticed that I can register
a message handler to receive messages. Can you point me to how messages
are produced?

So you basically need to create a callback function and set it as the
message handler. So you call:

 template <typename T> void set_msg_handler(T msg_handler){
  d_msg_handler = msg_handler_t(msg_handler);
}

Where d_msg_handler is of type:

typedef boost::function<void(pmt::pmt_t)> msg_handler_t;

You then use gruel::send (in msg_passing.h) to a block that has a
message
acceptor handler defined (or not; if there is no handler, nothing
happens).
You can see gnuradio-core/src/lib/runtime/qa_set_msg_handler.cc for an
example of this.

We have wrapped most of the PMT stuff into Python, but I don’t think
that
the send method has been made available, yet. So there’s work to be done
there.

I really hate gr_tag_info.h. I would rather get_tags_in_range filled a

vector of type tag_type, where tag_type has methods like .get_key(),
.get_value(), etc… Basically im complaining that we didnt use object
orientedness here when its realy well suited for it. You will see that I
did this wrap-around in my pyblock_gateway so I didnt have to swig
tag_info. See the tags_demo.py to see how I used it.

_josh

Ok, that’s fixable. We probably want to redo much of the tags interface
to
make it easier, and those are good suggestions.

Tom

  d_msg_handler = msg_handler_t(msg_handler);

I did my own digging last night, and found that gr_block inherits
gr_basic_block inherits gr_msg_accepter which has post(). So the
gruel::send is just a function to call this post() method.

So, there is nothing here that really deals with message propagation.
Message sending exists at the block level but not at the flow graph
level. Or am I am misunderstanding.

-Josh

On Mon, Sep 26, 2011 at 12:17 PM, Ben H. [email protected]
wrote:

“Devil’s avocado, here, Dave” - For what it’s worth, I think you could put
prototype code that people all over the world are writing from desks covered
Cheers,
Ben

Yes, it’s just my concern that people will use it for development but
just
get comfortable in that mode and not move it into c++ for production.

As always, the way we intend for things to be used is not always the way
they are used.

That’s just a concern, though, and not a show-stopper. If that’s how
people
want to work and they aren’t having performance issues, then why not let
them work away?

Tom

On Tue, Sep 27, 2011 at 11:14 AM, Josh B. [email protected] wrote:

 template <typename T> void set_msg_handler(T msg_handler){

You can see gnuradio-core/src/lib/runtime/qa_set_msg_handler.cc for an

-Josh

You are correct, sir.

The messages are handled, on the other hand, in the flowgraph. You can
find
it in gr_block_executor, if you are so inclined. But that’s purposefully
behind the scenes, so you shouldn’t have to worry about that at all.

Tom