Recent development activity, upcoming plans

GNU Radio
1

I just wanted to give a heads up on several things that are in flight
right now, and what Tom and I are planning soon.

  • Removal of dependency on gsl for gnuradio-core. The wavelet blocks
    that were in gnuradio-core, which depend on the GNU Scientific Library
    for some of their functions, have been moved into their own top-level
    component, gr-wavelet. This means that for the majority of users, the
    need for installing libgsl and libgsl-dev will go away. Python users
    wishing to use these blocks will find them in the ‘wavelet’ namespace
    under gnuradio, and C++ users will find them renamed to wavelet_*.

  • Clean up and preparation for releasing 3.5.3. There are still a few
    more development items remaining to merge in, but we expect to tag and
    release this next weekend. Numerous bug fixes, the addition of
    support for the FUNcube dongle SDR, enhancements to support the Ettus
    Research UHD 3.4.0 release, and a few smaller items will make this
    release.

  • Preparation to to merge the ‘next’ branch back in to ‘master’, after
    the 3.5.3 release. The ‘next’ branch is the long-running development
    branch where we implement things that are either somewhat unstable, or
    more often, make changes to the API such that user code might break.
    After the 3.5.3 stable release, we’ll switch the master branch to the
    3.6 API by merging ‘next’ back into master. The largest change
    involved here is the official switch from GNU autotools to CMake as
    the build system for GNU Radio. Our plan is to focus on testing and
    prepping for a 3.6.0 release, then start a new long-term ‘next’ branch
    for the 3.7 API.

If you normally use GNU Radio by compiling from a tarball, these
changes won’t affect to you until you start using the 3.5.3 or 3.6.0
release tarballs. If you are tracking our master branch and compiling
from that, then once 3.5.3 releases, you need to be prepared for the
change to the 3.6 API.

More details on the changes will be provided in release notes and on
the list here.

Johnathan

2
  • Removal of dependency on gsl for gnuradio-core. The wavelet blocks
    that were in gnuradio-core, which depend on the GNU Scientific Library

I would make a similar argument for the FFTW dependency on
gnuradio-core. I’m not volunteering, but that would bring gnuradio-core
down to gruel, boost, and cppunit as dependencies. Even cppunit could be
replaced with boost unit test.

  • Preparation to to merge the ‘next’ branch back in to ‘master’, after
    the 3.5.3 release. The ‘next’ branch is the long-running development
    branch where we implement things that are either somewhat unstable, or
    more often, make changes to the API such that user code might break.

There was this idea of loadable modules for gr-audio. That way one could
build gr-audio for all the audio development environments, but when
deploying, you only install the modules that the OS has runtime support
for. Think deb packaging, where each module becomes a package with
specific requirements like libjack, libalsa… Would this merge event be
a good time to implement something like that?

Also, for those of us who are interested in gnuradio’s PMT library:

  1. Would this be a good time to fix the tag/PMT mutability issue? This
    would prevent multiple downstream blocks from being able to manipulate a
    shared resource, but it is a slight API change for the tags:
    http://gnuradio.org/redmine/issues/490

  2. I would love to see the concept of tag/PMT based message passing make
    it into the scheduler. The basic idea is to pass tags (without samples
    associated) between blocks as a form of out-of-band
    communication/control-plane. Here is here squashed into a single
    changeset: GitHub - guruofquality/gnuradio at msg_passing

  3. I would also like to purpose some extensions to the PMT library to
    make PMTs more intelligent about memory management and allocation
    overhead. Also, the work is squashed into a single changeset:
    GitHub - guruofquality/gnuradio at pmt_extensions

-josh

3

On Mon, Apr 2, 2012 at 2:17 AM, Josh B. [email protected] wrote:

  • Removal of dependency on gsl for gnuradio-core. The wavelet blocks
    that were in gnuradio-core, which depend on the GNU Scientific Library

I would make a similar argument for the FFTW dependency on
gnuradio-core. I’m not volunteering, but that would bring gnuradio-core
down to gruel, boost, and cppunit as dependencies. Even cppunit could be
replaced with boost unit test.

FFTs are pretty vital to signal processing and are used in multiple
places. Many of these are in the filters, and we have a plan (for 3.7)
to create a gr-filter category. You aren’t going to get a whole lot
accomplished in signal processing if you don’t have filters and the
components that use FFTW. So even if we made a separate component for
FFTs, it’s still and effective requirement to install it even if
it’s not a required requirement.

When we get gr-filter, then we could have a discussion of whether its
appropriate to have a gr-fft component. I feel that’s a bit too much
granularity, but we can discuss it at that point. I think as we are
moving towards more top-level components and less in gnuradio-core
(mainly just runtime), we will be working on these issues in general.
We don’t want too much granularity, and I still see gr_fft_vcc being a
part of a larger component structure and not just on its own.

Wavelets are their own class of algorithms that are really useful for
lots of things, but aren’t fundamental to most processing we do. So on
one level, separating it out means most people won’t notice that it’s
missing, and on another, there are lots of new things that can be
added that fall under the gr-wavelet component in the future. That’s
not really true for FFTs. I had suggested we have a gr-scientific or
maybe a gr-transforms, but I don’t think that plays too well, really,
even if it is technically appropriate (and if we did FFTs and wavelets
in one component, we’d force people to install GSL just to get FFTs;
not where we want to be going with this).

Basically, I’m saying this is a discussion for a little while in the
future. Definitely at least after we’ve move gr-filter out of
gnuradio-core.

As for the unit testing, frankly, I don’t care what tool is used so
long as the behavior is the same as we have now. One thing that’s
important is that the QA results are stored in the XML files that
Jenkins reads. I’m pretty sure Jenkins can handle Boost output files,
too, but we’d want to make sure that was working cleanly if we made
the move. What I don’t want is to have two such systems running (we
already have a Python and C++ unit test, which some find confusing
already). So an switch would have to be made all at once. I personally
don’t see that it’s such a big deal to go through that effort to make
the change.

a good time to implement something like that?

Also, for those of us who are interested in gnuradio’s PMT library:

  1. Would this be a good time to fix the tag/PMT mutability issue? This
    would prevent multiple downstream blocks from being able to manipulate a
    shared resource, but it is a slight API change for the tags:
    http://gnuradio.org/redmine/issues/490

Johnathan and I have discussed this, but I don’t think we’ve come up
with a plan. The idea is that we fix the issue of being able to
manipulate pmt_vectors (and any other containers) after their creation
such that all PMTs are now read-only. The PMT vectors are a good idea
in that you want to be able to have a vector of items, but the
implementation has allowed them to be changed, which produces thread
safety issues.

  1. I would love to see the concept of tag/PMT based message passing make
    it into the scheduler. The basic idea is to pass tags (without samples
    associated) between blocks as a form of out-of-band
    communication/control-plane. Here is here squashed into a single
    changeset: GitHub - guruofquality/gnuradio at msg_passing

I’m of the mind that yes, this is a good thing. The squashed commit it
nice, so I finally went through to look at the code. This can probably
be merged in soon. Johnathan and I are going to take a closer look at
it, so we might have a few requests for it soon.

One request. Could you write either some documentation for how to use
the message passing system or, probably better, a simple example that
shows connecting blocks together and passing messages between them?
That would help clarify their use. I’m not saying that this is a
requirement before the merge, but it would be nice to see how it’s
used.

  1. I would also like to purpose some extensions to the PMT library to
    make PMTs more intelligent about memory management and allocation
    overhead. Also, the work is squashed into a single changeset:
    GitHub - guruofquality/gnuradio at pmt_extensions

-josh

There are a couple of problems with the work here. First, there is the
issue of adding a read/write buffer to the pmt_blobs. That produces
thread safety issues with PMTs. The idea of a PMT is that if you need
to change the contents, you create a new PMT with the new contents.
Yes, this generates overhead in creating the PMT and doing any memory
copying, but we don’t know that this is actually a problem right now.
Also, while this might cause some overhead to be incurred when working
with PMTs, the alternative is to introduce the possibility of
segfaults. PMTs are meant to be used in different threads and even
processes. It becomes a nightmare to think about mutex locking them
all over the place for concurrency. Having them read-only keeps us
from having to worry about this. I think the current PMTs that can be
manipulated after their creation are bugs that we need to fix. Not the
other way around.

This unfortunately makes the PMT pool idea unworkable. If I understand
it correctly, the idea is to create a set of PMTs that can then be
grabbed and modified instead of always creating new ones. This can’t
work since PMTs are immutable. Unless you are doing it some other way
that doesn’t require this. But again, I’m not sure where PMT creation
is the bottleneck anywhere, and Boost intrusive pointers work really
fast already.

I hope this clears up why the PMT read/write issue is a non-starter.
I’d rather see a bit of slowdown then segfaults :slight_smile:

Tom (with input from Johnathan)

4

On Sun, Apr 1, 2012 at 23:17, Josh B. [email protected] wrote:

There was this idea of loadable modules for gr-audio. That way one could
build gr-audio for all the audio development environments, but when
deploying, you only install the modules that the OS has runtime support
for. Think deb packaging, where each module becomes a package with
specific requirements like libjack, libalsa…

When gr-audio was individual top-level directories, doing the above
was trivial, and in fact, was how it was done for the old debian
packages in 3.2.

How do you see making this work now?

Would this merge event be
a good time to implement something like that?

No, I think it would be development work that could happen after the
3.6.0 release, either on master as part of 3.6.1, or on the new 3.7
next branch, depending on the intrusiveness of the solution.

Johnathan

5

On 04/02/2012 03:20 PM, Johnathan C. wrote:

packages in 3.2.

How do you see making this work now?

Well, so I would like to restore that install capability that was lost
when we combined gr-audio components into one.

The basic idea is, rather than all implementations of audio source/sink
being built into the gnuradio-audio.so, we build the implementations
into separate modules. Whatever modules are installed at runtime are
loaded, and available for selection via the audio.conf settings.

-josh

6

the change.

Looks like Boost unit test can output XML FWIW. Dont know if thats a
format jenkins can parse:
http://www.alittlemadness.com/2009/04/07/boosttest-xml-reports-for-continuous-integration/

Johnathan and I have discussed this, but I don’t think we’ve come up
with a plan. The idea is that we fix the issue of being able to
manipulate pmt_vectors (and any other containers) after their creation
such that all PMTs are now read-only. The PMT vectors are a good idea
in that you want to be able to have a vector of items, but the
implementation has allowed them to be changed, which produces thread
safety issues.

In the plan outlined in that ticket, gr_tag_t would be filled with
pmt_const_t and not pmt_t. This would prevent the downstream dudes from
violating the immutability contract with the upstream. The
implementation of creating and filling a PMT would not have to change.

One request. Could you write either some documentation for how to use
the message passing system or, probably better, a simple example that
shows connecting blocks together and passing messages between them?
That would help clarify their use. I’m not saying that this is a
requirement before the merge, but it would be nice to see how it’s
used.

Absolutely.

Here is a QA test:

And usage is fully documented on the blocks coding guide for c++/python:
http://gnuradio.org/redmine/projects/gnuradio/wiki/BlocksCodingGuide

that doesn’t require this. But again, I’m not sure where PMT creation
is the bottleneck anywhere, and Boost intrusive pointers work really
fast already.

Boost intrusive pointers are great. But they are actually just
stack-allocated objects. Its the pointer you fill them with, which is
the allocation in question.

So, In small quantities, when latency isnt a concern, of course its not
a big deal. However, for large quantities, I learned this the hard way
with UHD and oprofile. If you have a malloc on every packet, the
overhead is on the same order of magnitude as converting a packet worth
of samples. So, it can be serious overhead, and its a good thing to give
users a way to avoid allocating heavily in the work function (should
they be using PMTs that heavily).

I hope this clears up why the PMT read/write issue is a non-starter.
I’d rather see a bit of slowdown then segfaults :slight_smile:

I’m not sure what segfaults you are referring to. But the thread safety
issue is already a problem. Technically, tags/PMTs are one giant race
condition if your downstream blocks do not follow the implicit contract.
We should fix this.

-Josh

7

On Mon, Apr 2, 2012 at 11:10 PM, Josh B. [email protected] wrote:

-Josh
Tags are PMTs and write-once, read-only objects. To change them, you
have to create a new tag. This is what’s done for the sample number
value when going through rate change blocks. We instantiate a new PMT
with the new value. Since they are read-only and this is handled in
the block executor, there is no problem. Even if we change the PMT of
a tag for the number of samples, anyone holding a reference to the PMT
at the time will still have that PMT until it goes out of scope.

The only issue with thread safety are those PMTs that are read/write
(like vectors, and we don’t use them in the tags), and we are going to
fix them. There’s no need for a pmt_const class because all PMTs are
constant. We’re not going to introduce more writable shared memory
objects between blocks to exacerbate the problem.

Tom

8

On 04/05/2012 07:08 AM, Tom R. wrote:

The only issue with thread safety are those PMTs that are read/write
(like vectors, and we don’t use them in the tags), and we are going to
fix them.

That sounds like a lot more of a complicated change than the
pmt_const_t. What did you have in mind for the “fix”?

There’s no need for a pmt_const class because all PMTs are

constant. We’re not going to introduce more writable shared memory
objects between blocks to exacerbate the problem.

This is email is totally bonkers. I am going to have to issue “the PMT
Challenge”. Please wait for another thread and challenge details.

-josh