I’ve been doing a lot of work lately on the in-code documentation, and
I’ve
finally gotten around to working with Ben R.'s docstring work to
insert
the Doxygen docstrings into Python. This results in us being able call
help(gr.something) and get the full documentation that’s available in
the
header file. I’ve made a few changes, such as moving the meat of the
code
into the docs directory and a few other minor things, and it’s now in a
branch ‘docstrings’ available at:
git://github.com/trondeau/gnuradio.git
This branch has also been updated to the latest in next.
The issue that I wanted to bring up here is that the process for doing
this
is not quite automatic, though it’s mostly automatic. The problem is
that
the docs directory is processed last (and has to be), so the output XML
files that are parsed are not available until after the build is done.
This
means that you have to build once, run the swig_doc.py file to generate
a
SWIG file from the XML files, then rebuild the entire project to insert
the
documentation into Python.
The good news is that the .i file is kept in git and is part of the
project,
so it’s as relevant as the last time someone checked it in. So for the
most
part, you don’t really have to worry about this process unless you are
changing documentation in the code. The bad part is that if you are
working
on the documentation, you have to remember to update this periodically.
I think that the benefits of this outweigh the negatives, and I’m about
ready to put this into our next branch so it will be a part of 3.5. But
I
wanted to give people some time to look over the process and suggest
possible modifications that might make things work more smoothly or even
integrated into the regular build process.
Here’s the information that I put into a README.doxyxml file for the
instructions to produce the docstrings:
===========================================================
The process of updating and exporting the Doxygen document strings
into Python consists of a few steps.
-
Make sure the ‘docs’ component will be built, which requires
Doxygen.
-
Build the project like normal, which will run Doxygen and store the
XML files into $(top_builddir).
-
In $(top_srcdir)/docs/doxygen, run the command:
$ python swig_doc.py
$(top_builddir)/docstrings/docs/doxygen/xml
$(top_srcdir)/gnuradio-core/src/lib/swig/swig_doc.i
This uses the XML output of Doxygen to to rebuild a SWIG file that
contains all of the current Doxygen markups.
-
Rebuild the GNU Radio libraries. Since gnuradio.i is included in
all of the GNU Radio components, and gnuradio.i includes
swig_doc.i, when the libraries are rebuilt, they will now include
the documentation strings in Python.
-
Install GNU Radio. Now, when you run help() in Python on a GNU
Radio block, you will get the full documentation.
===========================================================
Thanks,
Tom
On 10/14/2011 07:44 PM, Tom R. wrote:
This branch has also been updated to the latest in next.
The issue that I wanted to bring up here is that the process for doing this
is not quite automatic, though it’s mostly automatic. The problem is that
the docs directory is processed last (and has to be), so the output XML
files that are parsed are not available until after the build is done. This
means that you have to build once, run the swig_doc.py file to generate a
SWIG file from the XML files, then rebuild the entire project to insert the
documentation into Python.
So suppose it this way:
Currently the docs/doxygen component builds a monolithic html and xml
folder for the whole project. I suppose we may like the monolithic html
because it makes a nice index.html and other index-like stuff. But the
purpose of the xml has been primarily to extract documentation
information (like in grc).
So, perhaps we keep the docs/doxygen html generator and turn off the
xml. In place of the monolithic xml, we have a per-component set of
build rules that calls doxygen on the headers to generate xml, and a
build rule that calls swig_doc.py to generate swig_doc.i. Think of a
macro that takes a list of .h files and generates the appropriate build
rules for xml, and then xml -> swig_docs.i.
If the docs are accessible through python, we probably dont need grc to
parse the xml. Instead, it can find the component in python and query
its docs, hopefully by reading the module’s docstrings (doc) if it
really works like that.
-Josh
-
Build the project like normal, which will run Doxygen and store the
-
Rebuild the GNU Radio libraries. Since gnuradio.i is included in
all of the GNU Radio components, and gnuradio.i includes
swig_doc.i, when the libraries are rebuilt, they will now include
the documentation strings in Python.
-
Install GNU Radio. Now, when you run help() in Python on a GNU
Radio block, you will get the full documentation.
===========================================================
OK, so I ran swig_doc.py on gr-digital and got this swig_doc.i:
/* * This file was automatically generated using swig_doc.py. * * Any c - Pastebin.com And the docstrings are clearly in there. I
then added %include “swig_doc.i” to digital_swig.i. So now the module
should be generated w/ the docstrings.
So, how can I verify that this worked? I tried something like this:
python -c “from gnuradio import digital; print
help(digital.kurtotic_equalizer_cc)”
Basically, where do the docstrings actually go in the module?
Heres my work that does the build rules for swig docs in gr-digital:
http://gnuradio.org/cgit/jblum.git/log/?h=swig_docs
-Josh
On Sat, Oct 15, 2011 at 12:28 AM, Josh B. [email protected] wrote:
contains all of the current Doxygen markups.
Discuss-gnuradio Info Page
Hi Josh,
The docstrings should be accessible using the doc property in the
normal way.
If you look at the generated python file the docstrings should be
present in the function and class definitions as usual.
e.g.
def kurtolic_equalizer_cc(blah blah):
“docstring here”
blah blah blah
If that’s not happening, then it’s not working.
I cloned your repo and checked out your swig_docs branch but got a
compilation error.
[ 46%] Generating gengen/gr_peak_detector_fb.i,
gengen/gr_peak_detector_ib.i, gengen/gr_peak_detector_sb.i
[ 51%] Built target gengen_generated
[ 51%] Swig source
[ 52%] Swig source
/home/ben/gnuradio-jblum/jblum/gnuradio-core/src/lib/gengen/gengen.i:33:
Error: Unable to find ‘gengen_generated.i’
make[2]: ***
[gnuradio-core/src/lib/swig/gnuradio_core_gengenPYTHON_wrap.cxx]
Error 1
make[1]: ***
[gnuradio-core/src/lib/swig/CMakeFiles/_gnuradio_core_filter.dir/all]
Error 2
make: *** [all] Error 2
Cheers,
Ben
"docstring here"
blah blah blah
If that’s not happening, then it’s not working.
Looks like i did it wrong then. But I cant see anything wrong with the
process. 
python -c “from gnuradio import digital; print
digital.kurtotic_equalizer_cc.doc”
kurtotic_equalizer_cc(int num_taps, float mu) ->
digital_kurtotic_equalizer_cc_sptr
make[2]: *** [gnuradio-core/src/lib/swig/gnuradio_core_gengenPYTHON_wrap.cxx]
Error 1
make[1]: ***
[gnuradio-core/src/lib/swig/CMakeFiles/_gnuradio_core_filter.dir/all]
Error 2
make: *** [all] Error 2
Ah, yea, I fixed that this morning but forgot to push. Just re-checkout
the repo.
-Josh
On Sat, Oct 15, 2011 at 4:16 PM, Josh B. [email protected] wrote:
def kurtolic_equalizer_cc(blah blah):
digital.kurtotic_equalizer_cc.doc"
[ 51%] Swig source
Ah, yea, I fixed that this morning but forgot to push. Just re-checkout
the repo.
-Josh
So, it’s a bug in swig_doc.py, but I haven’t worked out where yet…
It should be generating a docstring for “kurtotic_equalizer_cc”, but
it is generating it for “digital_make_kurtotic_equalizer_cc” instead.
There’s some stuff in swig_doc.py that determines whether a class or
function is part of a “Block” or just a normal class or function.
That appears to be not working. I’ll see if I can work out why not.
On Sat, Oct 15, 2011 at 10:58 PM, Ben R. [email protected] wrote:
e.g.
python -c "from gnuradio import digital; print
[ 51%] Built target gengen_generated
function is part of a “Block” or just a normal class or function.
That appears to be not working. I’ll see if I can work out why not.
Nope, that wasn’t the problem. I have no idea why it’s not working.
I’ll have another look at it tomorrow night.
Ahhh much better. I added generation for a few of the components in core
(general, gengen, filter, io)
http://gnuradio.org/cgit/jblum.git/commit/?h=swig_docs&id=d65572359a74e97ee6a01d89dcc44fd77ce54fef
Basically, its pretty nice. The xml only regenerates when header files
change, and the swig docs.i only regenerate when the xml is changed, and
so on. No checked-in generated files. XML generation is very quick. Your
python script actually takes a bit longer (but im not complaining). 
I would like to get the grc docs parser looking for doc strings
soon.
A few comments on your doc generator. Im not sure we agree on a standard
here for documenting headers… but useful doc blocks may be in several
places in the header:
- docs for the factor/make function
- docs for class definition
- docs just in the header, think \file
It looks like the generated docstrings only gets one of these. I suggest
concatenating all forms of docs into the one class docstring so the full
amount of docs is available at runtime to python.
-Josh
On Mon, Oct 17, 2011 at 12:36 PM, Josh B. [email protected] wrote:
Ahhh much better. I added generation for a few of the components in core
(general, gengen, filter, io)
http://gnuradio.org/cgit/jblum.git/commit/?h=swig_docs&id=d65572359a74e97ee6a01d89dcc44fd77ce54fef
Basically, its pretty nice. The xml only regenerates when header files
change, and the swig docs.i only regenerate when the xml is changed, and
so on. No checked-in generated files. XML generation is very quick. Your
python script actually takes a bit longer (but im not complaining).
That’s great, Josh, thanks! I’ll look at it soon, but sounds like a
good,
sane way of doing this.
One question: what if the user’s don’t have doxygen installed? Right now
it’s only an optional dependency. My guess is that it’s not too hard to
turn
this on or off depending on if you have doxygen. I just want to make
sure
this is handled.
Since we’re all hoping that we get to move to cmake completely, I’m
going to
abandon any work on getting this to work in autotools.
It looks like the generated docstrings only gets one of these. I suggest
concatenating all forms of docs into the one class docstring so the full
amount of docs is available at runtime to python.
-Josh
I agree. It would be nice to have all the documentation for every
function,
not just the class docs.
Thanks!
Tom
On Sat, Oct 15, 2011 at 11:33 PM, Josh B. [email protected] wrote:
normal way.
compilation error.
make[1]: ***
[gnuradio-core/src/lib/swig/CMakeFiles/_gnuradio_core_filter.dir/all]
So, it’s a bug in swig_doc.py, but I haven’t worked out where yet…
Each invocation of the generation macro would just need to pass this
ok cool
im looking forward to generating docs for each package, and getting grc
to read the docstrings rather than the xml
It was the position of the %include “swig_doc.i” in the digital_swig.i
file.
It needs to be at the top rather than at the bottom.
Here’s a diff.
diff --git a/gr-digital/swig/digital_swig.i
b/gr-digital/swig/digital_swig.i
index c804b5c…f6372b1 100644
— a/gr-digital/swig/digital_swig.i
+++ b/gr-digital/swig/digital_swig.i
@@ -23,6 +23,8 @@
%include <gri_control_loop.i>
+%include “swig_doc.i”
+
%{
#include “digital_binary_slicer_fb.h”
#include “digital_clock_recovery_mm_cc.h”
@@ -59,8 +61,6 @@
%include “digital_cpmmod_bc.i”
%include “digital_gmskmod_bc.i”
-%include “swig_doc.i”
#if SWIGGUILE
%scheme %{
(load-extension-global “libguile-gnuradio-digital_swig”
“scm_init_gnuradio_digital_swig_module”)
On Mon, Oct 17, 2011 at 11:31 AM, Josh B. [email protected] wrote:
it’s only an optional dependency. My guess is that it’s not too hard to
turn
this on or off depending on if you have doxygen. I just want to make sure
this is handled.
Right now its generating an empty swig docs file if doxygen was not
found. In other words, everything will still build.
Perfect!
Also, as long as cmake can find the doxygen executable, it will perform
the generation. The way to turn this off right now is to tell cmake that
the doxygen executable is undefined/not-found. I would like to add a
more official option: either it gets its own option, or its part of the
existing ENABLE_DOXYGEN option.
I think we just wrap it into the ENABLE_DOXYGEN that’s also contingent
on
Python being enabled. Doesn’t really make sense to build the docs and
the
Python swig and not put the two together now that we can do it.
Now having said that, the time taken to compile the swig generated cc
file >> than the time taken to generate the docs. So its really just
fine to always build the docs as far as I am concerned.
Yeah, sounds fine to me, too.
Since we’re all hoping that we get to move to cmake completely, I’m
going to
abandon any work on getting this to work in autotools.
If we want to keep autotools working along-side cmake w/ this doc work,
autotools would need to generate an empty swig_docs.i file to keep the
compile happy.
-josh
That’s easily done. A tad bit hackish, but if we end up keeping the
autotools stuff, I can go in and do it right. This is just to keep us
going
in parallel until we can move over to cmake completely.
Thanks,
Tom
On 10/17/2011 10:46 AM, Tom R. wrote:
python script actually takes a bit longer (but im not complaining).
That’s great, Josh, thanks! I’ll look at it soon, but sounds like a good,
sane way of doing this.
One question: what if the user’s don’t have doxygen installed? Right now
it’s only an optional dependency. My guess is that it’s not too hard to turn
this on or off depending on if you have doxygen. I just want to make sure
this is handled.
Right now its generating an empty swig docs file if doxygen was not
found. In other words, everything will still build.
Also, as long as cmake can find the doxygen executable, it will perform
the generation. The way to turn this off right now is to tell cmake that
the doxygen executable is undefined/not-found. I would like to add a
more official option: either it gets its own option, or its part of the
existing ENABLE_DOXYGEN option.
Now having said that, the time taken to compile the swig generated cc
file >> than the time taken to generate the docs. So its really just
fine to always build the docs as far as I am concerned.
Since we’re all hoping that we get to move to cmake completely, I’m going to
abandon any work on getting this to work in autotools.
If we want to keep autotools working along-side cmake w/ this doc work,
autotools would need to generate an empty swig_docs.i file to keep the
compile happy.
-josh
Ben, Josh,
Where are we with this? We had good momentum and were like 95% there,
but
then we dropped it about a month ago. I believe that Josh had a branch
that, using cmake, took the docstrings code and would build a .i file
per
top-level component and install it with the Python code. I think the
issue
was that this only got the constructors, but Ben’s note here talks about
fixing that. Has this been ported to the cmake build process to
autogenerate the doc strings?
Thanks,
Tom
On 11/16/2011 02:57 PM, Tom R. wrote:
Ben, Josh,
Where are we with this? We had good momentum and were like 95% there, but
then we dropped it about a month ago. I believe that Josh had a branch
that, using cmake, took the docstrings code and would build a .i file per
top-level component and install it with the Python code. I think the issue
was that this only got the constructors, but Ben’s note here talks about
fixing that. Has this been ported to the cmake build process to
autogenerate the doc strings?
Sorry, Ive just been too busy doing too many awesome things. I have a
branch with ben’s latest python scripts for doc extractions, but its
gotten too stale, so heres my list to get this going again:
- Get ben’s python foo rebased off of latest master branch
- Get my cmake swig docs macro also rebased
- Implement in a few complicated places like gr-core, hopefully i can
get some help for the other components
- Extract docstrings in grc’s doc extractor/parser
Sounds like a good weekend project.
-Josh
On 11/16/2011 03:50 PM, Josh B. wrote:
autogenerate the doc strings?
You’re such slave driver Tom
http://gnuradio.org/cgit/jblum.git/log/?h=more_swigness_docs
I only did core and digital. GRC seems to extract the docs just fine.
The digital should stand as an example how to swig doc’ify the others
I’ve made a few changes to doxyxml so that the DoxyFile objects should
be exposing their descriptions and have changed swig_doc.py so that
it’s grabbing documentation from the file if it is present. It
doesn’t appear to be picking any file descriptions up and I’m not sure
if that’s because it’s not working or because there aren’t any. If
you know of some good blocks to test it on let me know.
commit is
https://github.com/benreynwar/gnuradio/commit/d6e36498174dcb20f92369cc6097514a5d102ded
At the moment the layout for a docstring for a python creator function:
“”"
brief description from class
detailed description from class
brief description from make func
detailed description from make func
brief description from file
detailed description from file
Params: (list of input parameters)
“”"
The docstring for the class itself is the same without the parameter
list at the end.
Cheers,
Ben