# State of offline signal analysis tools in 2014?

**URL:** <https://www.ruby-forum.com/t/state-of-offline-signal-analysis-tools-in-2014/230993>\
**Category:** GNU Radio\
**Created:** [July 16, 2014, 4:54pm UTC](https://www.ruby-forum.com/t/state-of-offline-signal-analysis-tools-in-2014/230993 "2014-07-16T16:54:28Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![addis\_a](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/c4cdca/32.png) [@addis\_a](https://www.ruby-forum.com/u/addis_a)\
**Post date:** [July 16, 2014, 4:54pm UTC](https://www.ruby-forum.com/t/state-of-offline-signal-analysis-tools-in-2014/230993/1 "2014-07-16T16:54:28Z")

</div>

GNU Radio is a great tool for applications and dynamic experimentation,  
but it doesn’t have a lot of support for static/offline analysis of  
time-series data. I.e. I’ve captured some signal data and I want to  
explore its properties interactively so I can figure out what I want to  
do with it in GNU Radio.

The sort of capabilities I’m looking for include: Read time-series data  
from files of different formats (some too large to fit in physical  
memory). Display the data, optionally applying linear transformations.  
Interactively pan and zoom. Jump forwards and backwards among  
time-registered events. Enable/disable/time-shift data overlays. Export  
selected data to new files. Calculate and display statistics and other  
non-linear transformations of selected data.

Ideally I’d like an open-source analysis framework that I can extend in  
Python or C++; something like the Midas DSP tool family. I’m aware of  
some Qt widgets like QtCustomPlot, and generic frameworks like  
matplotlib and octave, but not of any ready-to-use applications or  
frameworks that already provide the basic functionality described  
above. The keywords I’ve tossed at Google haven’t produced any obvious  
solutions, and discussions I find in the archives here are a couple  
years old and seem to summarize as “use maplotlib/octave”.

Is any such framework available now or in development? If not, is  
anybody interested in joining me offline to discuss the requirements and  
design for such a thing?

Thanks.

Peter

---

<div class="post-metadata">

**Author:** ![Peter\_ASBigot](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/e0b2c6/32.png) [@Peter\_ASBigot](https://www.ruby-forum.com/u/Peter_ASBigot)\
**Post date:** [July 16, 2014, 5:22pm UTC](https://www.ruby-forum.com/t/state-of-offline-signal-analysis-tools-in-2014/230993/2 "2014-07-16T17:22:09Z")

</div>

On Wed, Jul 16, 2014 at 09:52:43AM -0500, Peter A. Bigot wrote:

> Is any such framework available now or in development? If not, is  
> anybody interested in joining me offline to discuss the requirements  
> and design for such a thing?

Something like this very nearly happened as a GSoC project this year.  
See:

[http://gnuradio.org/redmine/projects/gnuradio/wiki/GSoC#Advanced-Signal-Visualization-and-Analysis](http://gnuradio.org/redmine/projects/gnuradio/wiki/GSoC#Advanced-Signal-Visualization-and-Analysis)

Jared B. has some preliminary work in Python, mostly on packet  
detection and demodulation:

> **[GitHub - jboone/tpms: Tire Pressure Monitoring System decoding tools.](https://github.com/jboone/tpms)**
>
> Tire Pressure Monitoring System decoding tools.

Mike

---

<div class="post-metadata">

**Author:** ![Peter\_ASBigot](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/e0b2c6/32.png) [@Peter\_ASBigot](https://www.ruby-forum.com/u/Peter_ASBigot)\
**Post date:** [July 16, 2014, 5:22pm UTC](https://www.ruby-forum.com/t/state-of-offline-signal-analysis-tools-in-2014/230993/3 "2014-07-16T17:22:51Z")

</div>

Hi Peter,

GNU Radio is based very much on the idea of a data stream, so it might  
not actually be the tool of choice for static analysis.  
However, there is quite a lot which can be done with on-board tools, so  
let me comment in-text.  
On 16.07.2014 16:52, Peter A. Bigot wrote:

> GNU Radio is a great tool for applications and dynamic  
> experimentation, but it doesn’t have a lot of support for  
> static/offline analysis of time-series data. I.e. I’ve captured some  
> signal data and I want to explore its properties interactively so I  
> can figure out what I want to do with it in GNU Radio.
> 
> The sort of capabilities I’m looking for include:

> Read time-series data from files of different formats (some too large  
> to fit in physical memory).  
> So far, only raw samples in machine float format are supported, and the  
> GNU Radio-specific metadata/samples interleaved format along with the  
> Wav audio file format. As far as I know, all of the sources/sinks for  
> these file formats don’t need to store data in RAM but read/write it  
> sequentially.

If you feel like there is something obviously missing in this list, you  
could just use the awesome powers of python and/or C++ to read your  
favourite file format, write a database adapter or a twitter source; the  
reason we don’t have things like native CSV or HDF support is that I  
guess noone cared to implement a source for these formats, because they  
don’t lend themselves to streaming very well, and not because it’s hard.

Anyway, there’s a series of small tools for sample files, called  
gr\_{plot,plot\_fft,spectrogram}\_, that at least allow you to  
visualize recorded data easily, included in GNU Radio.

> Display the data, optionally applying linear transformations.  
> Well the problem here is that our visual sinks usually want to  
> periodically update the display, and that GNU Radio flow graphs usually  
> terminate when sources have finished producing items (e.g. when the  
> source file has been read completely). Many of these issues can be  
> worked around be setting your file source to repeat and “pausing” the  
> graphical sink when you see something interesting, after throttling your  
> item flow enough to make the signal observable by the naked eye.

The linear transformation thing is something you’d have to implement in  
a DSPish manner, and most probably can be done.

> Interactively pan and zoom.  
> Most of the graphical sinks can do that  
> Jump forwards and backwards among time-registered events.  
> Nope, I’m afraid that won’t work with the stream-oriented architecture  
> of GNU Radio.  
> Enable/disable/time-shift data overlays.  
> Again, if you feed a graphical sink with a signal and a time-shifted  
> version, you get a DSPified method of doing your visualization  
> Export selected data to new files.  
> not really available (yet?).  
> Calculate and display statistics and other non-linear transformations  
> of selected data.  
> Depends. Again, if you can translate your statistics to a signal  
> processing algorithm, then it’s almost certainly already been done or is  
> “easy” to do.
> 
> Ideally I’d like an open-source analysis framework that I can extend  
> in Python or C++; something like the Midas DSP tool family.  
> Not aware of these, sorry, and google turned up some defense program  
> along with large audio mixers. Do you have a URL to refer to?  
> I’m aware of some Qt widgets like QtCustomPlot, and generic frameworks  
> like matplotlib and octave, but not of any ready-to-use applications  
> or frameworks that already provide the basic functionality described  
> above.  
> I think you should take a look at things like R, GNUplot etc.  
> Anyway, this is a very interesting topic, and I would really enjoy  
> hearing from cool software that does what you describes in a manner that  
> could e.g. be explained to EE first-semester students or so.  
> The keywords I’ve tossed at Google haven’t produced any obvious  
> solutions, and discussions I find in the archives here are a couple  
> years old and seem to summarize as “use maplotlib/octave”.  
> I’m afraid my 2014 reply will disappoint you a little… it’s “if you  
> know what characteristics you’re looking for, go for a few lines of  
> python; if you don’t know, go for python and some additional lines”.  
> Actually, I’ve grown so used to numpy/scipy/matplotlib/pyqtgraph that I  
> wouldn’t trade it for Matlab (I have access to that and rarely use it),  
> especially because python is something I would consider a real language  
> whereas the matlab syntax and the matlab interpreter performance…  
> well, matlab has fantastic documentation.

> Is any such framework available now or in development? If not, is  
> anybody interested in joining me offline to discuss the requirements  
> and design for such a thing?  
> Count me in, as this is relevant to my work.

Greetings,  
Marcus

---

<div class="post-metadata">

**Author:** ![Peter\_ASBigot](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/e0b2c6/32.png) [@Peter\_ASBigot](https://www.ruby-forum.com/u/Peter_ASBigot)\
**Post date:** [July 16, 2014, 5:34pm UTC](https://www.ruby-forum.com/t/state-of-offline-signal-analysis-tools-in-2014/230993/4 "2014-07-16T17:34:39Z")

</div>

I’ve never used it for RF work, but pandas is a very powerful framework  
for  
working with timeseries and multi-dimensional data.

Very Respectfully,

Dan CaJacob

On Wed, Jul 16, 2014 at 11:20 AM, Marcus Müller  
[removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)

---

<div class="post-metadata">

**Author:** ![Peter\_ASBigot](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/e0b2c6/32.png) [@Peter\_ASBigot](https://www.ruby-forum.com/u/Peter_ASBigot)\
**Post date:** [July 16, 2014, 5:41pm UTC](https://www.ruby-forum.com/t/state-of-offline-signal-analysis-tools-in-2014/230993/5 "2014-07-16T17:41:34Z")

</div>

Two suggestions from my side:  
If you want to use Python, you can use the Spyder IDE  
([Google Code Archive - Long-term storage for Google Code Project Hosting.](https://code.google.com/p/spyderlib/)). Spyder is mainly designed for  
scientific programming. It even has built-in plotting capabilities.  
Another package I have used for that purpose is kst-plot. It is fast and  
has a lot of features: [http://kst-plot.kde.org](http://kst-plot.kde.org)  
The good thing is that KST unlike many other KDE applications does not  
rely on a full KDE installation, but uses just QT libraries.  
There is a PPA for Ubuntu users:  
[kst PPA : “kst developers” team](https://launchpad.net/~kst-plot/+archive/ubuntu/ppa)

Mark

---

<div class="post-metadata">

**Author:** ![Peter\_ASBigot](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/e0b2c6/32.png) [@Peter\_ASBigot](https://www.ruby-forum.com/u/Peter_ASBigot)\
**Post date:** [July 17, 2014, 11:54am UTC](https://www.ruby-forum.com/t/state-of-offline-signal-analysis-tools-in-2014/230993/6 "2014-07-17T11:54:19Z")

</div>

Thanks for the recommendations.

I should clarify that I am a software engineer, not a signals engineer,  
and my recurring need to visualize time-series data is often satisfied  
without having to invoke DSP. An example of the sort of thing I  
frequently want to do is to interactively explore data collected from  
multiple wireless sensors (e.g., six months of temperature and humidity  
data collected at 1-minute intervals). Technically signals, but simple  
shifts, scales, and basic windowed statistics are more useful than FFTs  
and complex filters.

Sometimes I do need DSP techniques to extract the data from third-party  
wireless transmissions, hence my current dabbling with GNU Radio, but  
it’s the data not the extraction process that’s the primary focus. For  
example, the reason I’m using GNU Radio is my need to demodulate packets  
from a WS-2080 Weather Station and some other 433 MHz OOK sensors. Yes,  
I’ve tried rtl433 and rtlsdr-433m-sensor; the signal I’m capturing is  
too noisy or I don’t have frequency/bandwidth/filter settings right.  
The specific use case that introduced my question about analysis tools  
is my desire to interactively identify regions of interest from wideband  
captures and then re-play that data repeatedly through various  
processing chains, tweaking parameters until I get something that  
reliably produces the underlying data.

@mossman: The GSoC project is very close to what I’d want for  
DSP-oriented analysis. If it existed it’d probably handle the  
motivating example above. It’s too domain-specific for generalized  
time-series visualization, though, so I’d still have an unmet need.

@marcus.mueller: Thanks for the details. My experience is that a  
metered stream/dataflow-oriented architecture is simply unsuited to the  
sort of offline analysis I’m trying to do. The ability to jump forwards  
and backwards in time is crucial, as is the ability to decouple the  
signal rate from the data processing rate. Example: Qt Time Raster  
schedules its updates based on wall clock not sample time, so speeding  
up or slowing down the rate of data through the system changes the  
displayed images, and running unthrottled drops all the information.

@dan.cajacob: pandas reminds me a lot of R. The intro video showed it’d  
be good for CLI-based data manipulation, but my initial need is to  
explore data graphically.

@mdammer: kst-plot’s web site shows some promising graphs, and it built  
cleanly (though it doesn’t obey CMAKE\_INSTALL\_PREFIX), but I’ve been  
unable to locate examples and documentation that would allow me to  
evaluate its capabilities quickly.

@madengr: Lack of source code for baudline and its focus on DSP knocks  
it out of contention for my general needs.

On 07/16/2014 09:52 AM, Peter A. Bigot wrote:

> The sort of capabilities I’m looking for include: Read time-series  
> data from files of different formats (some too large to fit in  
> physical memory). Display the data, optionally applying linear  
> transformations. Interactively pan and zoom. Jump forwards and  
> backwards among time-registered events. Enable/disable/time-shift  
> data overlays. Export selected data to new files. Calculate and  
> display statistics and other non-linear transformations of selected data.

A rough course towards the tool I imagine would be:

- Collect/develop a suite of Qt/C++ widgets for graphical data display  
and manipulation that are not sensitive to the processing rate, don’t  
have a unidirectional concept of time, can be accessed from Python,  
and can be combined to build something with the capabilities listed  
above.

- Use a command-line interface like pandas/R to display original files,  
extract regions of interest, apply transformations, and repeat until  
satori.

- Use GNU Radio Companion to glue components together to form  
domain-specific analysis applications.

That’s a lot of yak shaving just to get a reliable OOK packet extractor,  
so it probably won’t happen.

Thanks again for the suggestions.

Peter

---

<div class="post-metadata">

**Author:** ![Peter\_ASBigot](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/e0b2c6/32.png) [@Peter\_ASBigot](https://www.ruby-forum.com/u/Peter_ASBigot)\
**Post date:** [July 16, 2014, 7:44pm UTC](https://www.ruby-forum.com/t/state-of-offline-signal-analysis-tools-in-2014/230993/7 "2014-07-16T19:44:46Z")

</div>

Peter,

Have you tried Baudline? It’s not open source, but it is free, and will  
do  
allot of what you want. Looks like there will be a new version soon  
with  
64-bit support and I/Q demod. It’s also quite easy to pipe GR file sink  
to  
it via a FIFO (gr-baz has a sink for just that), but it will also do  
off-line analysis also.

> **[baudline signal analyzer - FFT spectrogram](http://www.baudline.com/)**
>
> A time frequency browser designed for spectrum analysis data visualization.

Thanks,  
Lou  
KD4HSO

Peter A. Bigot wrote

> The sort of capabilities I’m looking for include: Read time-series data  
> from files of different formats (some too large to fit in physical  
> memory). Display the data, optionally applying linear transformations.  
> Interactively pan and zoom. Jump forwards and backwards among  
> time-registered events. Enable/disable/time-shift data overlays. Export  
> selected data to new files. Calculate and display statistics and other  
> non-linear transformations of selected data.

–  
View this message in context:  
[http://gnuradio.4.n7.nabble.com/state-of-offline-signal-analysis-tools-in-2014-tp49376p49385.html](http://gnuradio.4.n7.nabble.com/state-of-offline-signal-analysis-tools-in-2014-tp49376p49385.html)  
Sent from the GnuRadio mailing list archive at [Nabble.com](http://Nabble.com).

---

<div class="post-metadata">

**Author:** ![Peter\_ASBigot](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/e0b2c6/32.png) [@Peter\_ASBigot](https://www.ruby-forum.com/u/Peter_ASBigot)\
**Post date:** [July 17, 2014, 12:37pm UTC](https://www.ruby-forum.com/t/state-of-offline-signal-analysis-tools-in-2014/230993/8 "2014-07-17T12:37:54Z")

</div>

-----BEGIN PGP SIGNED MESSAGE-----  
Hash: SHA1

Hi Peter,

You might have a look at Python with Numpy/Scipy/matplotlib. Processed  
data according to your needs may be observed interactively with  
matplotlib and numpy/scipy do a good job at offline analysis. You’d  
have to process your data again and again anyway whenever you want to  
check something else.

happy hacking  
Johannes

On 17.07.2014 11:52, Peter A. Bigot wrote:

> captures and then re-play that data repeatedly through various  
> metered stream/dataflow-oriented architecture is simply unsuited to  
> is to explore data graphically.
> 
> > The sort of capabilities I’m looking for include: Read
> 
> - Collect/develop a suite of Qt/C++ widgets for graphical data  
> domain-specific analysis applications.  
> [Discuss-gnuradio Info Page](https://lists.gnu.org/mailman/listinfo/discuss-gnuradio)

-----BEGIN PGP SIGNATURE-----  
Version: GnuPG v1  
Comment: Using GnuPG with Thunderbird - [http://www.enigmail.net/](http://www.enigmail.net/)

iQIcBAEBAgAGBQJTx6dJAAoJEO7fmkDsqywMQSgQAIEM6ujO7XUGXlKSiH0EFBAt  
7zUBbERcjIdNCv4/15kuH/sjNTfk9tUhu5aIK8M3FyRKyC3kIBkDLJBiZhMdxQ14  
sHldxKpt2kkKW6wYXLHfAV2qqauWM1lTfB9VYwjbtj8Wej55OwA6JUR6jtHS8F3O  
yZ+0obljFsChLSV8Y2DEHl66Of9SnyUciKIhBCt6MF4v48WeVwbkSeo04Ff31abE  
f3VlNcoNubug20boz8xMTVjanIei47b8zD1HDkroDaJikOWoboL/zdDnYJbfWiSY  
L16x5GjovOOvnw3ASkIKqO4iea4mqbvEiPu3OV1qHYzhb1TsVa32evMy/b8tO2HX  
CRcYs6lzs0YOiNDmUl2QUvP+Uaz81MLKmbk2EeA5MDpeLcvpAbjrd0bPkaL0DVRO  
fPwIReIFJuRri74d4O54nyNYQgONhl++J+K+tKLFmZ36UHFeDhTZ+odiLf+qN7vU  
ZujT0fANAww+hMZO4R4QZCUyDvcXthuwvaGY4T0nEerAv4Pu5jlCKKJOZKze3gzb  
kUoMNvn1pN2vNglihIaU4cNQauO24Lx0MoNOHED9Ar/le7QctYmiFp9oCSmokKSO  
JLheZsslvN86wWVQ6uFWRpa1+hOrG/JX1Ayxm9qoRmBtoV2L6c2dTDvZrasnnGlf  
pUAk1sdiqba9+dpZdkDQ  
=M0RI  
-----END PGP SIGNATURE-----

---

<div class="post-metadata">

**Author:** ![Peter\_ASBigot](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/e0b2c6/32.png) [@Peter\_ASBigot](https://www.ruby-forum.com/u/Peter_ASBigot)\
**Post date:** [July 17, 2014, 2:02pm UTC](https://www.ruby-forum.com/t/state-of-offline-signal-analysis-tools-in-2014/230993/9 "2014-07-17T14:02:32Z")

</div>

On 7/16/14 10:52 AM, Peter A. Bigot wrote:

> time-registered events. Enable/disable/time-shift data overlays. Export  
> selected data to new files. Calculate and display statistics and other  
> non-linear transformations of selected data.

If you have access to a Mac, DataGraph has much of this. We use it  
a lot to interactively manipulate large quantities of logged data.

> **[Visual Data Tools | Software for macOS](https://www.visualdatatools.com/DataGraph/)**
>
> Visual Data Tools creates powerful, affordable data analysis and visualization software for macOS — including DataGraph and ImageTank.

@(^.^)@ Ed