Gr_modtool cmake/make problems on OSX

WarrenS_Kevin_M · February 25, 2013, 10:12pm

Greetings all, this is my first post here so I apologize ahead of time
for any clumsiness in the order and detail of my question. I’m using a
Macport install of Gnuradio 3.6.3 (system particulars below). So far
I’ve had a good deal of success with Gnuradio but I’m at the point where
I need to start constructing custom modules. Unfortunately, I’ve
encountered some difficulties with the out-of-tree build tutorial using
gr_modtool.

Here is a summary of the problems, the first two being general FYI, the
last being my question to the group.

A minor difficulty is that gr_modtool.py does not appear to be
placing the class definitions and the instantiation of the class
object(s) in the proper order. This was easy enough to remedy by hand.
A second minor difficulty is that makexml does not appear to work at
all. Again, this is easy enough to remedy by hand.

My main problem that I’m 8+ hours into is that with my installation I
cannot get cmake to set an explicit path to my dylib file and changing
DYLD_LIBRARY_PATH produces other conflicts with gnuradio-companion.
Unfortunately my competency with cmake/make and modifying related files
is minimal. Can someone provide some insight into how I can explicitly
link my _howto_swig.so to the full path name of the dylib
(libgnuradio-howto.dylib) associated with it during the build process?
My suspicion is that I need to modify some aspect of the template in
gr_modtool.py but any ideas would be appreciated. For clarity, the
details of my system and build problems are provided below sequentially.

System: Mac OSX 10.6.8
%uname -a:
Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011;
root:xnu-1504.15.3~1/RELEASE_I386 i386

Gnuradio 3.6.3 from Macports:
gnuradio
@3.6.3_1+docs+full+grc+jack+orc+portaudio+python27+qtgui+sdl+swig+uhd+wavelet+wxgui

Using:
gr_modtool.py from github

Following tutorial:
http://gnuradio.org/redmine/projects/gnuradio/wiki/OutOfTreeModules

I followed the tutorial to the letter up to the cmake configuration
where I called cmake as follows (macport install):
%cmake -DCMAKE_INSTALL_PREFIX=/opt/local …/

This process proceeds and completes unremarkably. At this point
%make test
Fails.
%ctest -V
reports:

fails with:
2: File
“/Users/xxxxxx/Documents/Work/Gnuradio/modules/gr-howto/build/swig/howto_swig.py”,
line 314, in
2: square_ff = square_ff.make;
2: NameError: name ‘square_ff’ is not defined
2/2 Test #2: qa_square_ff …***Failed 0.32 sec

An inspection of howto_swig.py reveals that square_ff = square_ff.make
is called before the class square_ff is defined. This appears to be an
error in order with how gr_modtool constructs the swig file.
Rearranging the order within howto_swig.py eliminates the “name
‘square_ff’ is not defined error.” At this point there is a thread join
error in the test which is remedied by rolling back boost 1.52 to boost
1.51 (known issue).

An attempt to make the xml file fails entirely:

%gr_modtool.py makexml square_ff
Operating in directory .
GNU Radio module name identified: howto
Warning: This is an experimental feature. Don’t expect any magic.
Searching for matching files in lib/:
Making GRC bindings for lib/square_ff_impl.cc…
Can’t parse the argument list: Found closing parentheses before
finishing last argument (this is how far I got: [])

Fortunately this is no problem. Easy enough to edit by hand.

Execution of
%sudo make install

completes successfully in the proper directories, or at least the
directories I think I want the install to be located.

When I obtain a Python command prompt and attempt to import howto I get
the following error message:

Traceback (most recent call last):
File “”, line 1, in
File “/opt/local/lib/python2.7/site-packages/howto/init.py”, line
45, in
from howto_swig import *
File “/opt/local/lib/python2.7/site-packages/howto/howto_swig.py”,
line 26, in
_howto_swig = swig_import_helper()
File “/opt/local/lib/python2.7/site-packages/howto/howto_swig.py”,
line 22, in swig_import_helper
_mod = imp.load_module(‘_howto_swig’, fp, pathname, description)
ImportError:
dlopen(/opt/local/lib/python2.7/site-packages/howto/_howto_swig.so, 2):
Library not loaded: libgnuradio-howto.dylib
Referenced from:
/opt/local/lib/python2.7/site-packages/howto/_howto_swig.so
Reason: image not found

The dylib is properly installed:

%ls /opt/local/lib|grep howto
libgnuradio-howto.dylib

The files appear to be in the desired location:

%ls /opt/local/lib/python2.7/site-packages/howto
init.py init.pyc init.pyo _howto_swig.so howto_swig.py
howto_swig.pyc howto_swig.pyo

otool indicates that there is no explicit path to ibgnuradio-howto.dylib

%otool -L
/opt/local/lib/python2.7/site-packages/howto/_howto_swig.so/opt/local/lib/python2.7/site-packages/howto/_howto_swig.so:
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/Python
(compatibility version 2.7.0, current version 2.7.0)
libgnuradio-howto.dylib (compatibility version 0.0.0, current version
0.0.0)
/opt/local/lib/libboost_filesystem-mt.dylib (compatibility version
0.0.0, current version 0.0.0)
/opt/local/lib/libboost_system-mt.dylib (compatibility version 0.0.0,
current version 0.0.0)
/opt/local/lib/libgruel.3.6.4git.dylib (compatibility version 3.6.4,
current version 0.0.0)
/opt/local/lib/libgnuradio-core.3.6.4git.dylib (compatibility version
3.6.4, current version 0.0.0)
/usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version
7.9.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version
125.2.11)

It appears as if I need to set target_link_libraries somewhere in the
middle of all of this build process.
Unfortunately I cannot simply append /opt/local/lib to DYLD_LIBRARY_PATH
as this causes catastrophic errors with the gtk system framework when I
attempt to launch gnuradio-companion. Again, I cannot modify
DYLD_LIBRARY_PATH. I need to have it linked with its full path during
the build process.

WarrenS_Kevin_M · February 25, 2013, 10:39pm

On Mon, Feb 25, 2013 at 12:40 PM, Warren, Kevin M <
[email protected]> wrote:

Greetings all, this is my first post here so I apologize ahead of time for
any clumsiness in the order and detail of my question. I’m using a Macport
install of Gnuradio 3.6.3 (system particulars below). So far I’ve had a
good deal of success with Gnuradio but I’m at the point where I need to
start constructing custom modules. Unfortunately, I’ve encountered some
difficulties with the out-of-tree build tutorial using gr_modtool.

The gr_modtool has been integrated into the GNU Radio master branch
post-3.6.3, and will be part of the 3.6.4 release. At least one of the
issues below has already been fixed.

A minor difficulty is that gr_modtool.py does not appear to be
placing the class definitions and the instantiation of the class object(s)
in the proper order. This was easy enough to remedy by hand.

This has been fixed.

A second minor difficulty is that makexml does not appear to work at

all. Again, this is easy enough to remedy by hand.

I’ll let Martin B. address this.

My main problem that I’m 8+ hours into is that with my installation I
cannot get cmake to set an explicit path to my dylib file and
changing DYLD_LIBRARY_PATH produces other conflicts with gnuradio-companion.

Michael D. is our go-to guy for MacOSX issues; hopefully he can
respond to this.

Using:

gr_modtool.py from github

As I mentioned, this has now become a standard feature of GNU Radio. We
should be releasing 3.6.4 very shortly; this will (eventually) make it
into
MacPorts.

Finally–I wanted to specifically thank you for the detailed bug report
and
a list of all the steps you have already taken to try to fix the issue.
It
really helps us to 1) understand your problem, 2) rule out things you’ve
already tried, and 3) helps others searching the mailing list archives
trying to debug similar issues.

Johnathan

WarrenS_Kevin_M · February 26, 2013, 12:38am

The gr_modtool has been integrated into the GNU Radio master branch post-3.6.3,
and will be part of the 3.6.4 release. At least one of the issues below has
already been fixed.

Okay, great. I saw that comment on the github README file but assumed it
wasn’t in the latest release since I couldn’t find it.

Finally–I wanted to specifically thank you for the detailed bug report and a
list of all the steps you have already taken to try to fix the issue. It really
helps us to 1) understand your problem, 2) rule out things you’ve already tried,
and 3) helps others
searching the mailing list archives trying to debug similar issues.

And thank you, this is a great resource. Hopefully I can get the OSX
particulars sorted out. Again, I imagine it’s strictly due to a lack of
understanding on my part about how the linking process works.

Kevin

WarrenS_Kevin_M · February 26, 2013, 3:48am

Michael, this worked perfectly, thank you so much. I’m glad there was a
direct way to do it rather than poking around into the make side of
things.

sudo install_name_tool -id /opt/local/lib/libgnuradio-howto.dylib
/opt/local/lib/libgnuradio-howto.dylib

sudo install_name_tool -change libgnuradio-howto.dylib
/opt/local/lib/libgnuradio-howto.dylib
/opt/local/lib/python2.7/site-packages/howto/_howto_swig.so

I -highly- recommend not installing non-MacPorts stuff into the MacPorts area;
it’s easy to overwrite something and not know it. Most of the time what you’re
doing works OK, but when it does not it’s really difficult to debug. It’s
generally simple enough to set PATH and PYTHONPATH to include /usr/local/bin and
/usr/local/lib/python2.7, which should take care of all of your GNU Radio needs if
you install into /usr/local .

Interesting, I thought I could likely do more damage by jamming work
into /usr/local, at least if I needed to untangle something gone amiss.
This is partially based on some bad experiences with openCV and
advice/experiences related to me by a colleague who keeps regular
system ‘stuff’ separate from his various projects. Obviously I’m not too
far into building custom modules so if going the /usr/local route is
better I’ll migrate that way.

Hope this help! - MLD
Tremendously, thanks again.

Kevin

WarrenS_Kevin_M · February 26, 2013, 3:20am

Hi Kevin - You’ll want to use “install_name_tool” to fix the DLYD name
id for both _howto_swig.so and libgnuradio-howto.dylib. Here’s what I
would do (watch wrap):

{{{

sudo install_name_tool
-id /opt/local/lib/libgnuradio-howto.dylib /opt/local/lib/libgnuradio-h
owto.dylib

sudo install_name_tool -change libgnuradio-howto.dylib
/opt/local/lib/libgnuradio-howto.dylib
/opt/local/lib/python2.7/site-packages/howto/_howto_swig.so

}}}

and, then you can verify that these are correct via:

{{{

otool -L /opt/local/lib/python2.7/site-packages/howto/_howto_swig.so

otool -L /opt/local/lib/libgnuradio-howto.dylib

}}}

as well as by loading in your howto module into Python. There’s a way
to coerce CMake into doing this auto-magically, but I’d have to look it
up and test it out. Which won’t be easy just right now because my
primary computer is temporarily down for the count.

BTW> I -highly- recommend not installing non-MacPorts stuff into the
MacPorts area; it’s easy to overwrite something and not know it. Most
of the time what you’re doing works OK, but when it does not it’s
really difficult to debug. It’s generally simple enough to set PATH
and PYTHONPATH to include /usr/local/bin and /usr/local/lib/python2.7,
which should take care of all of your GNU Radio needs if you install
into /usr/local .

Hope this help! - MLD

On Mon, Feb 25, 2013, at 03:40 PM, Warren, Kevin M wrote:

%otool -L
/opt/local/lib/python2.7/site-packages/howto/_howto_swig.so/opt/local/l
ib/python2.7/site-packages/howto/_howto_swig.so:
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/Python
(compatibility version 2.7.0, current version 2.7.0)
libgnuradio-howto.dylib (compatibility version 0.0.0, current version
0.0.0)
/opt/local/lib/libboost_filesystem-mt.dylib (compatibility version
0.0.0, current version 0.0.0)
/opt/local/lib/libboost_system-mt.dylib (compatibility version 0.0.0,
current version 0.0.0)
/opt/local/lib/libgruel.3.6.4git.dylib (compatibility version 3.6.4,
current version 0.0.0)
/opt/local/lib/libgnuradio-core.3.6.4git.dylib (compatibility version
3.6.4, current version 0.0.0)
/usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current
version 7.9.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 125.2.11)

WarrenS_Kevin_M · February 26, 2013, 3:47pm

I assume you mean in the SWIG files? Or are you talking about something else?
The SWIG order bug was fixed in both the standalone and the master branch version
of gr-modtool.

Martin,
It’s “howto_swig.py” that is generated following the out of tree module
tutorial (link in original post) downloaded from github. I see there’s
been a new version posted within the past few hours.

Also, are you using the pre-3.7 or the post-3.7 module structure (gr_modtool
info tells you this if you don’t know what this means).

Unfortunately I do not see any switches in the parser or hard coded
information containing the version information, though there is a
reference to self.info[‘_version’] I don’t see the actual assignment.
Either way, it was the top version on github as of two days ago.

In general, stuff that’s “easy to remedy by hand” should be easy to get right in
modtool. If you’re not talking about the swig files, perhaps you can post the
faulty code?

I hope this is sufficient. I cut out some of the details of the code to
keep it short.

square_ff_sptr_swigregister = _howto_swig.square_ff_sptr_swigregister
square_ff_sptr_swigregister(square_ff_sptr)

square_ff_sptr.repr = lambda self: “<gr_block %s (%d)>” %
(self.name(), self.unique_id())
square_ff = square_ff.make;

class square_ff(object):
“”“<+description of block+>”“”
…
return _howto_swig.square_ff_make()

def make():
....
make = staticmethod(make)
__swig_destroy__ = _howto_swig.delete_square_ff
__del__ = lambda self : None;

def square_ff_make():
“”"
square_ff_make() → square_ff_sptr

Return a shared_ptr to a new instance of howto::square_ff.
 ..."""
return _howto_swig.square_ff_make()

square_ff_swigregister = _howto_swig.square_ff_swigregister
square_ff_swigregister(square_ff)

makexml makes a lot of assumptions, and right now there’s no code parser for
Python blocks. Message ports aren’t supported at all right now. Do you have a C++
block? If so, I’d be interested to see the code chunks that makexml stumbles over.

It is identical to that posted in the out of tree module tutorial.
http://gnuradio.org/redmine/projects/gnuradio/wiki/OutOfTreeModules

However, there’s only so much I can do in makexml, and it will never be able to
fully understand all blocks (although it should fail gracefully).

Understood. As far as I can tell gr_modtool didn’t do anything to the
original template when I ran it on the example. When I have some time I
will poke into the details of the script but someone else’s parsing and
code generation tools can be challenging to unravel. I can barely follow
my own a few days after I’ve written them.

Thanks,
Kevin

WarrenS_Kevin_M · February 26, 2013, 9:19am

On Mon, Feb 25, 2013 at 08:40:31PM +0000, Warren, Kevin M wrote:

A minor difficulty is that gr_modtool.py does not appear to be placing the
class definitions and the instantiation of the class object(s) in the proper
order. This was easy enough to remedy by hand.

Hi Kevin,

I assume you mean in the SWIG files? Or are you talking about something
else? The SWIG order bug was fixed in both the standalone and the master
branch version of gr-modtool. Also, are you using the pre-3.7 or the
post-3.7 module structure (gr_modtool info tells you this if you don’t
know what this means).

In general, stuff that’s “easy to remedy by hand” should be easy to get
right in modtool. If you’re not talking about the swig files, perhaps
you can post the faulty code?

A second minor difficulty is that makexml does not appear to work at all.
Again, this is easy enough to remedy by hand.

makexml makes a lot of assumptions, and right now there’s no code parser
for Python blocks. Message ports aren’t supported at all right now.

Do you have a C++ block? If so, I’d be interested to see the code chunks
that makexml stumbles over.

However, there’s only so much I can do in makexml, and it will never be
able to fully understand all blocks (although it should fail
gracefully).

MB

–
Karlsruhe Institute of Technology (KIT)
Communications Engineering Lab (CEL)

Dipl.-Ing. Martin B.
Research Associate

Kaiserstraße 12
Building 05.01
76131 Karlsruhe

Phone: +49 721 608-43790
Fax: +49 721 608-46071
www.cel.kit.edu

KIT – University of the State of Baden-Württemberg and
National Laboratory of the Helmholtz Association

WarrenS_Kevin_M · February 26, 2013, 5:43pm

On Tue, Feb 26, 2013 at 02:45:52PM +0000, Warren, Kevin M wrote:

Martin,
It’s “howto_swig.py” that is generated following the out of tree module tutorial
(link in original post) downloaded from github. I see there’s been a new version
posted within the past few hours.

If the swig file was the problem, you should be fine.

Also, are you using the pre-3.7 or the post-3.7 module structure (gr_modtool
info tells you this if you don’t know what this means).

Unfortunately I do not see any switches in the parser or hard coded information
containing the version information, though there is a reference to
self.info[‘_version’] I don’t see the actual assignment. Either way, it was the
top version on github as of two days ago.

This doesn’t have anything to do with the modtool version, but the
version of your code structure. If you don’t know what this means,
you’re probably on post-3.7, but ‘gr_modtool info’ will tell you for
sure.

In general, stuff that’s “easy to remedy by hand” should be easy to get right
in modtool. If you’re not talking about the swig files, perhaps you can post the
faulty code?

I hope this is sufficient. I cut out some of the details of the code to keep it
short.
[…]

Yep, this is the bug that was fixed a while back.

makexml makes a lot of assumptions, and right now there’s no code parser for
Python blocks. Message ports aren’t supported at all right now. Do you have a C++
block? If so, I’d be interested to see the code chunks that makexml stumbles over.

It is identical to that posted in the out of tree module tutorial.
http://gnuradio.org/redmine/projects/gnuradio/wiki/OutOfTreeModules

However, there’s only so much I can do in makexml, and it will never be able
to fully understand all blocks (although it should fail gracefully).

Understood. As far as I can tell gr_modtool didn’t do anything to the original
template when I ran it on the example. When I have some time I will poke into the
details of the script but someone else’s parsing and code generation tools can be
challenging to unravel. I can barely follow my own a few days after I’ve written
them.

I just pushed a fix to the standalone, perhaps that help.

Note that I’ll only be doing easy bugfixes to the standalone version
since modtool was merged.

MB

–
Karlsruhe Institute of Technology (KIT)
Communications Engineering Lab (CEL)

Dipl.-Ing. Martin B.
Research Associate

Kaiserstraße 12
Building 05.01
76131 Karlsruhe

Phone: +49 721 608-43790
Fax: +49 721 608-46071
www.cel.kit.edu

KIT – University of the State of Baden-Württemberg and
National Laboratory of the Helmholtz Association