Forum: GNU Radio gr_modtool cmake/make problems on OSX

Greetings all, this is my first post here so I apologize ahead of time 
for any clumsiness in the order and detail of my question. I'm using a 
Macport install of Gnuradio 3.6.3 (system particulars below). So far 
I've had a good deal of success with Gnuradio but I'm at the point where 
I need to start constructing custom modules. Unfortunately, I've 
encountered some difficulties with the out-of-tree build tutorial using 
gr_modtool.

Here is a summary of the problems, the first two being general FYI, the 
last being my question to the group.

1) A minor difficulty is that gr_modtool.py does not appear to be 
placing the class definitions and the instantiation of the class 
object(s) in the proper order. This was easy enough to remedy by hand.

2) A second minor difficulty is that makexml does not appear to work at 
all. Again, this is easy enough to remedy by hand.

- My main problem that I'm 8+ hours into is that with my installation I 
cannot get cmake to set an explicit path to my dylib file and changing 
DYLD_LIBRARY_PATH produces other conflicts with gnuradio-companion. 
Unfortunately my competency with cmake/make and modifying related files 
is minimal. Can someone provide some insight into how I can explicitly 
link my _howto_swig.so to the full path name of the dylib 
(libgnuradio-howto.dylib) associated with it during the build process? 
My suspicion is that I need to modify some aspect of the template in 
gr_modtool.py but any ideas would be appreciated. For clarity, the 
details of my system and build problems are provided below sequentially.

System: Mac OSX 10.6.8
%uname -a:
Darwin Kernel Version 10.8.0: Tue Jun  7 16:33:36 PDT 2011; 
root:xnu-1504.15.3~1/RELEASE_I386 i386

Gnuradio 3.6.3 from Macports:
gnuradio 
@3.6.3_1+docs+full+grc+jack+orc+portaudio+python27+qtgui+sdl+swig+uhd+wavelet+wxgui

Using:
gr_modtool.py from github

Following tutorial:
http://gnuradio.org/redmine/projects/gnuradio/wiki...

I followed the tutorial to the letter up to the cmake configuration 
where I called cmake as follows (macport install):
%cmake -DCMAKE_INSTALL_PREFIX=/opt/local ../

This process proceeds and completes unremarkably. At this point
%make test
Fails.
%ctest -V
reports:

fails with:
2:   File 
"/Users/xxxxxx/Documents/Work/Gnuradio/modules/gr-howto/build/swig/howto_swig.py", 
line 314, in <module>
2:     square_ff = square_ff.make;
2: NameError: name 'square_ff' is not defined
2/2 Test #2: qa_square_ff .....................***Failed    0.32 sec

An inspection of howto_swig.py reveals that square_ff = square_ff.make 
is called before the class square_ff is defined. This appears to be an 
error in order with how gr_modtool constructs the swig file. 
Rearranging the order within howto_swig.py eliminates the "name 
'square_ff' is not defined error." At this point there is a thread join 
error in the test which is remedied by rolling back boost 1.52 to boost 
1.51 (known issue).

An attempt to make the xml file fails entirely:

%gr_modtool.py makexml square_ff
Operating in directory .
GNU Radio module name identified: howto
Warning: This is an experimental feature. Don't expect any magic.
Searching for matching files in lib/:
Making GRC bindings for lib/square_ff_impl.cc...
Can't parse the argument list:  Found closing parentheses before 
finishing last argument (this is how far I got: [])

Fortunately this is no problem. Easy enough to edit by hand.

Execution of
%sudo make install

completes successfully in the proper directories, or at least the 
directories I think I want the install to be located.

When I obtain a Python command prompt and attempt to import howto I get 
the following error message:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/lib/python2.7/site-packages/howto/__init__.py", line 
45, in <module>
    from howto_swig import *
  File "/opt/local/lib/python2.7/site-packages/howto/howto_swig.py", 
line 26, in <module>
    _howto_swig = swig_import_helper()
  File "/opt/local/lib/python2.7/site-packages/howto/howto_swig.py", 
line 22, in swig_import_helper
    _mod = imp.load_module('_howto_swig', fp, pathname, description)
ImportError: 
dlopen(/opt/local/lib/python2.7/site-packages/howto/_howto_swig.so, 2): 
Library not loaded: libgnuradio-howto.dylib
  Referenced from: 
/opt/local/lib/python2.7/site-packages/howto/_howto_swig.so
  Reason: image not found

The dylib is properly installed:

%ls /opt/local/lib|grep howto
libgnuradio-howto.dylib

The files appear to be in the desired location:

%ls /opt/local/lib/python2.7/site-packages/howto
__init__.py __init__.pyc __init__.pyo _howto_swig.so howto_swig.py 
howto_swig.pyc howto_swig.pyo

otool indicates that there is no explicit path to ibgnuradio-howto.dylib

%otool -L 
/opt/local/lib/python2.7/site-packages/howto/_howto_swig.so/opt/local/lib/python2.7/site-packages/howto/_howto_swig.so:
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/Python 
(compatibility version 2.7.0, current version 2.7.0)
libgnuradio-howto.dylib (compatibility version 0.0.0, current version 
0.0.0)
/opt/local/lib/libboost_filesystem-mt.dylib (compatibility version 
0.0.0, current version 0.0.0)
/opt/local/lib/libboost_system-mt.dylib (compatibility version 0.0.0, 
current version 0.0.0)
/opt/local/lib/libgruel.3.6.4git.dylib (compatibility version 3.6.4, 
current version 0.0.0)
/opt/local/lib/libgnuradio-core.3.6.4git.dylib (compatibility version 
3.6.4, current version 0.0.0)
/usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 
7.9.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 
125.2.11)

It appears as if I need to set target_link_libraries somewhere in the 
middle of all of this build process.
Unfortunately I cannot simply append /opt/local/lib to DYLD_LIBRARY_PATH 
as this causes catastrophic errors with the gtk system framework when I 
attempt to launch gnuradio-companion. Again, I cannot modify 
DYLD_LIBRARY_PATH. I need to have it linked with its full path during 
the build process.

On Mon, Feb 25, 2013 at 12:40 PM, Warren, Kevin M <
kevin.m.warren@vanderbilt.edu> wrote:


> Greetings all, this is my first post here so I apologize ahead of time for
> any clumsiness in the order and detail of my question. I'm using a Macport
> install of Gnuradio 3.6.3 (system particulars below). So far I've had a
> good deal of success with Gnuradio but I'm at the point where I need to
> start constructing custom modules. Unfortunately, I've encountered some
> difficulties with the out-of-tree build tutorial using gr_modtool.
>

The gr_modtool has been integrated into the GNU Radio master branch
post-3.6.3, and will be part of the 3.6.4 release.  At least one of the
issues below has already been fixed.


>  1) A minor difficulty is that gr_modtool.py does not appear to be
> placing the class definitions and the instantiation of the class object(s)
> in the proper order. This was easy enough to remedy by hand.
>

This has been fixed.

 2) A second minor difficulty is that makexml does not appear to work at
> all. Again, this is easy enough to remedy by hand.
>

I'll let Martin Braun address this.


> - My main problem that I'm 8+ hours into is that with my installation I
> cannot get cmake to set an explicit path to my dylib file and
> changing DYLD_LIBRARY_PATH produces other conflicts with gnuradio-companion.
>

Michael Dickens is our go-to guy for MacOSX issues; hopefully he can
respond to this.

Using:
> gr_modtool.py from github
>

As I mentioned, this has now become a standard feature of GNU Radio.  We
should be releasing 3.6.4 very shortly; this will (eventually) make it 
into
MacPorts.

Finally--I wanted to specifically thank you for the detailed bug report 
and
a list of all the steps you have already taken to try to fix the issue. 
It
really helps us to 1) understand your problem, 2) rule out things you've
already tried, and 3) helps others searching the mailing list archives
trying to debug similar issues.

Johnathan

>> The gr_modtool has been integrated into the GNU Radio master branch post-3.6.3, 
and will be part of the 3.6.4 release.  At least one of the issues below has 
already been fixed.

Okay, great. I saw that comment on the github README file but assumed it 
wasn't in the latest release since I couldn't find it.

>> Finally--I wanted to specifically thank you for the detailed bug report and a 
list of all the steps you have already taken to try to fix the issue.  It really 
helps us to 1) understand your problem, 2) rule out things you've already tried, 
and 3) helps others
 searching the mailing list archives trying to debug similar issues.

And thank you, this is a great resource. Hopefully I can get the OSX 
particulars sorted out. Again, I imagine it's strictly due to a lack of 
understanding on my part about how the linking process works.


Kevin

Hi Kevin - You'll want to use "install_name_tool" to fix the DLYD name
id for both _howto_swig.so and libgnuradio-howto.dylib.  Here's what I
would do (watch wrap):

{{{

sudo install_name_tool
-id /opt/local/lib/libgnuradio-howto.dylib /opt/local/lib/libgnuradio-h
owto.dylib

sudo install_name_tool -change libgnuradio-howto.dylib
/opt/local/lib/libgnuradio-howto.dylib
/opt/local/lib/python2.7/site-packages/howto/_howto_swig.so

}}}

and, then you can verify that these are correct via:

{{{

otool -L /opt/local/lib/python2.7/site-packages/howto/_howto_swig.so

otool -L /opt/local/lib/libgnuradio-howto.dylib

}}}

as well as by loading in your howto module into Python. There's a way
to coerce CMake into doing this auto-magically, but I'd have to look it
up and test it out. Which won't be easy just right now because my
primary computer is temporarily down for the count.



BTW> I -highly- recommend not installing non-MacPorts stuff into the
MacPorts area; it's easy to overwrite something and not know it.  Most
of the time what you're doing works OK, but when it does not it's
really difficult to debug.  It's generally simple enough to set PATH
and PYTHONPATH to include /usr/local/bin and /usr/local/lib/python2.7,
which should take care of all of your GNU Radio needs if you install
into /usr/local .



Hope this help! - MLD



On Mon, Feb 25, 2013, at 03:40 PM, Warren, Kevin M wrote:

%otool -L
/opt/local/lib/python2.7/site-packages/howto/_howto_swig.so/opt/local/l
ib/python2.7/site-packages/howto/_howto_swig.so:
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/Python
(compatibility version 2.7.0, current version 2.7.0)
libgnuradio-howto.dylib (compatibility version 0.0.0, current version
0.0.0)
/opt/local/lib/libboost_filesystem-mt.dylib (compatibility version
0.0.0, current version 0.0.0)
/opt/local/lib/libboost_system-mt.dylib (compatibility version 0.0.0,
current version 0.0.0)
/opt/local/lib/libgruel.3.6.4git.dylib (compatibility version 3.6.4,
current version 0.0.0)
/opt/local/lib/libgnuradio-core.3.6.4git.dylib (compatibility version
3.6.4, current version 0.0.0)
/usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current
version 7.9.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 125.2.11)

Michael, this worked perfectly, thank you so much. I'm glad there was a 
direct way to do it rather than poking around into the make side of 
things.

>> sudo install_name_tool -id /opt/local/lib/libgnuradio-howto.dylib 
/opt/local/lib/libgnuradio-howto.dylib
>> sudo install_name_tool -change libgnuradio-howto.dylib 
/opt/local/lib/libgnuradio-howto.dylib 
/opt/local/lib/python2.7/site-packages/howto/_howto_swig.so


>> I -highly- recommend not installing non-MacPorts stuff into the MacPorts area; 
it's easy to overwrite something and not know it.  Most of the time what you're 
doing works OK, but when it does not it's really difficult to debug.  It's 
generally simple enough to set PATH and PYTHONPATH to include /usr/local/bin and 
/usr/local/lib/python2.7, which should take care of all of your GNU Radio needs if 
you install into /usr/local .

Interesting, I thought I could likely do more damage by jamming work 
into /usr/local, at least if I needed to untangle something gone amiss. 
This is partially based on some bad experiences with openCV and 
advice/experiences related to me by a colleague  who keeps regular 
system 'stuff' separate from his various projects. Obviously I'm not too 
far into building custom modules so if going the /usr/local route is 
better I'll migrate that way.

>> Hope this help! - MLD
 Tremendously, thanks again.

Kevin

On Mon, Feb 25, 2013 at 08:40:31PM +0000, Warren, Kevin M wrote:
> 1) A minor difficulty is that gr_modtool.py does not appear to be placing the
> class definitions and the instantiation of the class object(s) in the proper
> order. This was easy enough to remedy by hand.

Hi Kevin,

I assume you mean in the SWIG files? Or are you talking about something
else? The SWIG order bug was fixed in both the standalone and the master
branch version of gr-modtool. Also, are you using the pre-3.7 or the
post-3.7 module structure (gr_modtool info tells you this if you don't
know what this means).

In general, stuff that's "easy to remedy by hand" should be easy to get
right in modtool. If you're not talking about the swig files, perhaps
you can post the faulty code?

> 2) A second minor difficulty is that makexml does not appear to work at all.
> Again, this is easy enough to remedy by hand.

makexml makes a lot of assumptions, and right now there's no code parser
for Python blocks. Message ports aren't supported at all right now.

Do you have a C++ block? If so, I'd be interested to see the code chunks
that makexml stumbles over.

However, there's only so much I can do in makexml, and it will never be
able to fully understand all blocks (although it should fail
gracefully).

MB

--
Karlsruhe Institute of Technology (KIT)
Communications Engineering Lab (CEL)

Dipl.-Ing. Martin Braun
Research Associate

Kaiserstraße 12
Building 05.01
76131 Karlsruhe

Phone: +49 721 608-43790
Fax: +49 721 608-46071
www.cel.kit.edu

KIT -- University of the State of Baden-Württemberg and
National Laboratory of the Helmholtz Association

>>I assume you mean in the SWIG files? Or are you talking about something else? 
The SWIG order bug was fixed in both the standalone and the master branch version 
of gr-modtool.

Martin,
It's "howto_swig.py" that is generated following the out of tree module 
tutorial (link in original post) downloaded from github. I see there's 
been a new version posted within the past few hours.

>>Also, are you using the pre-3.7 or the post-3.7 module structure (gr_modtool 
info tells you this if you don't know what this means).

Unfortunately I do not see any switches in the parser or hard coded 
information containing the version information, though there is a 
reference to self.info['_version'] I don't see the actual assignment. 
Either way, it was the top version on github as of two days ago.

>>In general, stuff that's "easy to remedy by hand" should be easy to get right in 
modtool. If you're not talking about the swig files, perhaps you can post the 
faulty code?

I hope this is sufficient. I cut out some of the details of the code to 
keep it short.

square_ff_sptr_swigregister = _howto_swig.square_ff_sptr_swigregister
square_ff_sptr_swigregister(square_ff_sptr)

square_ff_sptr.__repr__ = lambda self: "<gr_block %s (%d)>" % 
(self.name(), self.unique_id())
square_ff = square_ff.make;

class square_ff(object):
    """<+description of block+>"""
     ...
        return _howto_swig.square_ff_make()

    def make():
    ....
    make = staticmethod(make)
    __swig_destroy__ = _howto_swig.delete_square_ff
    __del__ = lambda self : None;

def square_ff_make():
  """
    square_ff_make() -> square_ff_sptr

    Return a shared_ptr to a new instance of howto::square_ff.
     ..."""
    return _howto_swig.square_ff_make()

square_ff_swigregister = _howto_swig.square_ff_swigregister
square_ff_swigregister(square_ff)

>>makexml makes a lot of assumptions, and right now there's no code parser for 
Python blocks. Message ports aren't supported at all right now. Do you have a C++ 
block? If so, I'd be interested to see the code chunks that makexml stumbles over.

It is identical to that posted in the out of tree module tutorial.
http://gnuradio.org/redmine/projects/gnuradio/wiki...

>>However, there's only so much I can do in makexml, and it will never be able to 
fully understand all blocks (although it should fail gracefully).

Understood. As far as I can tell gr_modtool didn't do anything to the 
original template when I ran it on the example. When I have some time I 
will poke into the details of the script but someone else's parsing and 
code generation tools can be challenging to unravel. I can barely follow 
my own a few days after I've written them.

Thanks,
Kevin

On Tue, Feb 26, 2013 at 02:45:52PM +0000, Warren, Kevin M wrote:
> Martin,
> It's "howto_swig.py" that is generated following the out of tree module tutorial 
(link in original post) downloaded from github. I see there's been a new version 
posted within the past few hours.

If the swig file was the problem, you should be fine.

> >>Also, are you using the pre-3.7 or the post-3.7 module structure (gr_modtool 
info tells you this if you don't know what this means).
>
> Unfortunately I do not see any switches in the parser or hard coded information 
containing the version information, though there is a reference to 
self.info['_version'] I don't see the actual assignment. Either way, it was the 
top version on github as of two days ago.

This doesn't have anything to do with the modtool version, but the
version of your code structure. If you don't know what this means,
you're probably on post-3.7, but 'gr_modtool info' will tell you for
sure.

> >>In general, stuff that's "easy to remedy by hand" should be easy to get right 
in modtool. If you're not talking about the swig files, perhaps you can post the 
faulty code?
>
> I hope this is sufficient. I cut out some of the details of the code to keep it 
short.
> [...]

Yep, this is the bug that was fixed a while back.

> >>makexml makes a lot of assumptions, and right now there's no code parser for 
Python blocks. Message ports aren't supported at all right now. Do you have a C++ 
block? If so, I'd be interested to see the code chunks that makexml stumbles over.
>
> It is identical to that posted in the out of tree module tutorial.
> http://gnuradio.org/redmine/projects/gnuradio/wiki...
>
> >>However, there's only so much I can do in makexml, and it will never be able 
to fully understand all blocks (although it should fail gracefully).
>
> Understood. As far as I can tell gr_modtool didn't do anything to the original 
template when I ran it on the example. When I have some time I will poke into the 
details of the script but someone else's parsing and code generation tools can be 
challenging to unravel. I can barely follow my own a few days after I've written 
them.

I just pushed a fix to the standalone, perhaps that help.

Note that I'll only be doing easy bugfixes to the standalone version
since modtool was merged.

MB

--
Karlsruhe Institute of Technology (KIT)
Communications Engineering Lab (CEL)

Dipl.-Ing. Martin Braun
Research Associate

Kaiserstraße 12
Building 05.01
76131 Karlsruhe

Phone: +49 721 608-43790
Fax: +49 721 608-46071
www.cel.kit.edu

KIT -- University of the State of Baden-Württemberg and
National Laboratory of the Helmholtz Association