Documentation generation using Sphinx

GNU Radio
1

Hi all,

Every 6 months or so I have a crack at getting some python level
documentation working. In this attempt, I’ve generated documentation
for the gr and digital modules using sphinx.

The generated html is at:
http://www.reynwar.net/gnuradio/sphinx/

Source for the generated documentation is at:
https://github.com/benreynwar/gnuradio/tree/sphinx/docs/sphinx/source

The documentation generation is semi-automatic. I created files
containing lists of the objects we want to document, and organized
them into categories. The actual documentation itself was pulled from
the docstrings automatically. It would be necessary to manually edit
files when new blocks or other objects are added, so there is a danger
that this kind of documentation could become incomplete, however
because the content is taken from the docstrings it should remain
accurate, if not complete.

Are there any objections to me continuing down this path of
documentation generation? Any suggestions?

Cheers,
Ben

2

Hi Ben - I like the look of the generated HTML much more than that done
by Doxygen. That said, there is a serious deficiency … maybe this is
the “continuing down this path” of which you speak: Is there any way for
specific blocks to get sphinx to generate actual arguments?
“gnuradio.gr.fft_filter_ccc(*args, **kwargs)” isn’t very useful, but
“gnuradio.gr.fft_filter_ccc(vector < gr_complex > taps)” would be (if
that’s what the actual prototype is; I don’t remember it exactly right
now). Nice job; I say to keep it up … - MLD

3

On Wed, Feb 29, 2012 at 3:26 PM, Ben R. [email protected] wrote:

https://github.com/benreynwar/gnuradio/tree/sphinx/docs/sphinx/source
Are there any objections to me continuing down this path of
documentation generation? Any suggestions?

Cheers,
Ben

I’m a bit confused, Ben. What does this do that we aren’t doing in the
swig_docs work we put in last year?

Tom

4

On Wed, Feb 29, 2012 at 6:50 PM, Tom R. [email protected] wrote:

accurate, if not complete.

Tom

The swig_doc stuff got all the documentation from doxygen into the
python docstrings.

What I’m trying to do with this is to create some nice documentation
that we can put online that serves as a reference for someone
developing with gnuradio in python. I had a go at this last year, but
didn’t get very far, partly because the swig_doc stuff wasn’t fully
working. Now that the swig_docs is all sorted, it felt like a good
time to push with the documentation again.

Stuff that needs work is

  • (*args, **kwargs) appears for some blocks but for others it
    displays the parameters correctly.
  • sometimes it displays _dummy_0 for parameter types
  • there a bunch of subpackages which I haven’t touched yet

It’ll take a bit of work to get this done, and unless people are on
board with the general concept of using sphinx to generate this
documentation, it doesn’t make sense for me to spend time doing it,
which is why I’m bugging you all now with some half-finished
documentation.

Cheers,
Ben

5

On Wed, Feb 29, 2012 at 08:05:46PM -0700, Ben R. wrote:

  • sometimes it displays _dummy_0 for parameter types
  • there a bunch of subpackages which I haven’t touched yet

It’ll take a bit of work to get this done, and unless people are on
board with the general concept of using sphinx to generate this
documentation, it doesn’t make sense for me to spend time doing it,
which is why I’m bugging you all now with some half-finished
documentation.

I think it’s great! Something like this is really missing, especially if
you’re used to browsing the official Python docs; in that case Sphinx is
probably a sight you’re used to.

One thing I don’t love is this:

gnuradio.gr.glfsr_source_b Creates a glfsr_source_b blocksk.
gnuradio.gr.glfsr_source_f Creates a glfsr_source_f block.
gnuradio.gr.lfsr_32k_source_s Creates a lfsr_32k_source_s block.
gnuradio.gr.nowull_source Creates a null_source block.
gnuradio.gr.noise_source_c Createseates a noise_source_c block.
gnuradio.gr.noise_source_f Creates a noise_source_fse_source_f block.

This contains zero information. To get to the interesting information, I
have to first click the entry. If I try help(gr.glfsr_source_b) on my
machine, I get the interesting part of the docs straight away (“Galois
LFSR pseudo-random source generating float outputs -1.0 - 1.0.” instead
of “Creates a glfsr_source_b block”).

In the gr.digital package, this isn’t quite as bad. Perhaps a
documentation
“standard” wouldn’t be a bad idea (but none of this has anything to do
with Sphinx, I guess ;).

MB

–
Karlsruhe Institute of Technology (KIT)
Communications Engineering Lab (CEL)

Dipl.-Ing. Martin B.
Research Associate

Kaiserstraße 12
Building 05.01
76131 Karlsruhe

Phone: +49 721 608-43790
Fax: +49 721 608-46071
www.cel.kit.edu

KIT – University of the State of Baden-Württemberg and
National Laboratory of the Helmholtz Association

6

Just to clarify, I’m not suggesting we get rid of the Doxygen
documentation. I think it documents the C++ side of things really
well. What I’m suggesting is that we have two sets of documentation.
Doxygen-generated docs for the C++ stuff, and sphinx-generated docs
for the python interface.

At the moment the subheaders are coming from the \ingroup tag
indirectly, in that I use that them to generate the initial
documentation, but it won’t update automatically if they are changed.
I think automating the documentation generation to the level where
these would update automatically would complicate things more than it
would help.

I’ll continue extending and tidying up the documentation and let you
know when it’s at a more presentable point.

Cheers,
Ben

7

On Thu, Mar 1, 2012 at 12:16 PM, Ben R. [email protected] wrote:

Just to clarify, I’m not suggesting we get rid of the Doxygen
documentation. I think it documents the C++ side of things really
well. What I’m suggesting is that we have two sets of documentation.
Doxygen-generated docs for the C++ stuff, and sphinx-generated docs
for the python interface.

It looks like you have all of the C++ blocks moved over into the
Sphinx-generated docs, too, though, which duplicates what’s in Doxygen.
Having two sets of documentation seems a bit confusing to me, but I can
see
how a C+±only developer might get confused about stuff mentioned in the
manual that’s only available if using Python.

So with the latter argument in mind, I guess we could (and maybe should)
have separate documents for the two cases.

At the moment the subheaders are coming from the \ingroup tag
indirectly, in that I use that them to generate the initial
documentation, but it won’t update automatically if they are changed.
I think automating the documentation generation to the level where
these would update automatically would complicate things more than it
would help.

I think Josh worked out in cmake how to regenerate the swigdoc output if
any of the files are changed, so it seems like a similar thing should be
applicable here, too.

I’ll continue extending and tidying up the documentation and let you
know when it’s at a more presentable point.

Cheers,
Ben

Thanks,
Tom

8

Ben,
Yes, I was definitely confused. I think this is promising. So you’re
saying
that it gets populated by Doxygen markup that’s already in the files? So
we
don’t have to redo any of the documentation that we already have, right?

Also, I’ve been adding Doxygen pages (.dox files) with more
descriptions,
examples, etc. into the Doxygen manual. I really like this way as it
keeps
things all together as part of the code and the automatically generated
manual. If we can keep these as well, that’s great. And I’m assuming
that
the subheaders like “Signal Sources” and “Signal Sinks” are taken from
the
\ingroup tags?

Another thing that I’ve been doing with the Doxygen manual is keeping
older
versions of it alive on the website so that people can look at the
manual
for their particular version of GNU Radio (
http://gnuradio.org/redmine/projects/gnuradio/wiki/Old-docs). So again,
I’d
like to be able to easily host these. Looking at your URL, it looks like
it’ll be simple.

So yes, I say continue on this path. Taking what Martin and Michael said
about fixing some of the structure/styling would really help it, too.

Thanks!
Tom

9

On Thu, Mar 1, 2012 at 2:00 PM, Ben R. [email protected] wrote:

random internal stuff that is floating around in the module namespace.
Automating the decision on whether to include or exclude these
objects in the documentation, and what category to place them in would
be non-trivial. This is why using a tool like epydoc does not work
for us.

At the moment, when the html is regenerated, the docstrings themselves
will be updated, but the organization of the objects into categories
will not change unless the sphinx source files have been manually
edited.

Ok, I see. Is there a way to just stick all of these into an
“uncategorized” section? We could then edit the offenders to give them a
proper group.

Tom

10

On Thu, Mar 1, 2012 at 11:20 AM, Tom R. [email protected] wrote:

Sphinx-generated docs, too, though, which duplicates what’s in Doxygen.
Having two sets of documentation seems a bit confusing to me, but I can see
how a C+±only developer might get confused about stuff mentioned in the
manual that’s only available if using Python.

So with the latter argument in mind, I guess we could (and maybe should)
have separate documents for the two cases.

Yeah, ideally it would be nice to have one set of documentation that
covers the C++ and python levels, but I can’t think of a way to do
that, that doesn’t do a bad job of one or the other. Creating two
sets of documentation is the path of least resistance to finally
getting a reference for the python interface out there.

of the files are changed, so it seems like a similar thing should be
applicable here, too.

Although I generated the source files for sphinx to avoid tediousness,
I also did a bunch of manual editing afterwards. There are lots of
objects in the python modules that aren’t swiged signal processing
blocks and so don’t have \ingroup tags, for example python
hierarchical blocks, constellation objects, utility functions, and
random internal stuff that is floating around in the module namespace.
Automating the decision on whether to include or exclude these
objects in the documentation, and what category to place them in would
be non-trivial. This is why using a tool like epydoc does not work
for us.

At the moment, when the html is regenerated, the docstrings themselves
will be updated, but the organization of the objects into categories
will not change unless the sphinx source files have been manually
edited.

11

On Thu, Mar 1, 2012 at 12:05 PM, Tom R. [email protected] wrote:

would help.
blocks and so don’t have \ingroup tags, for example python
edited.

Ok, I see. Is there a way to just stick all of these into an “uncategorized”
section? We could then edit the offenders to give them a proper group.

Tom

I you’re really keen on getting the block categorization automated
then I’m sure I can come up with a way. Sphinx is extendable so I
would probably create a new sphinx markup command that includes all
the blocks for a given category. In this way the sphinx source file
would still be manually edited if you want to add a new utility
function, or python hierarchical block to the documentation but the
swiged signal processing blocks would be added automatically.

12

On Thu, Mar 1, 2012 at 2:21 PM, Ben R. [email protected] wrote:

these would update automatically would complicate things more than it
I also did a bunch of manual editing afterwards. There are lots of
will be updated, but the organization of the objects into categories

I you’re really keen on getting the block categorization automated
then I’m sure I can come up with a way. Sphinx is extendable so I
would probably create a new sphinx markup command that includes all
the blocks for a given category. In this way the sphinx source file
would still be manually edited if you want to add a new utility
function, or python hierarchical block to the documentation but the
swiged signal processing blocks would be added automatically.

We just want to automate any document generation as much as possible so
it’s always up-to-date when you build it.

Tom