Forum: GNU Radio Unknown cause of Latency with USRP

Hello all,

I'm seeing a lot of latency in streaming from a webcam with GNURadio and 
the USRP.
The transmit chain looks like:
ffmpeg | ./modulator > tx.fifo &
then I run
sudo ./usrp_stream_tx.py

modulator is an external code I am running that is not part of gnuradio. 
uhd_stream_tx is a python script I made with GRC and it reads tx.fifo, 
resamples the signal, and then outputs to the USRP.

The receive chain looks like:
sudo ./usrp_stream_rx.py &
then I run
cat rx.fifo | ./demodulator | ffplay -

demodulator is an external code that is not part of gnuradio. 
uhd_stream_rx is a python script that reads samples from the usrp and 
writes them to rx.fifo.

First I run the receive chain, then the transmit chain with about a 1-2 
second delay in between executing the receive and transmit commands. 
When waiving my hand in front of the webcam, the above gives me latency 
in the ffplay video player of about 5-10 seconds initially, and this 
keeps growing in time to be quite high (even up to 30-45 seconds). 
Someone else named Alex Csete had tried a similar DVB transmission and 
said he had seen 10 seconds of lag initially with his setup. Does anyone 
know what is causing this building lag? Could it be the fifo files in 
the chain?


The external modulator and demodulator code runs in real-time at fixed 
data rates that the usrp can handle when sampling a signal at 1MHz. The 
ffmpeg encoder bitrate is set to be about the same as the modulator 
code.


I also tried the same thing without the USRP by connecting a cat5e cable 
between the transmitter and receiver laptops. Then I replace 
./usrp_stream_rx.py and ./usrp_stream_tx.py in the tx/rx chains with 
netcat commands using nc -l 1234 for the receiver, and nc 192.68.0.3 for 
the transmitter. This works great and I don't see any of the 5-10 second 
latency as with the USRP.


I don't know how or if gnuradio interblock buffers or fusb settings play 
in to the latency I'm seeing? The sampling rate for the TX and RX is 1 
MHz so it seems samples are pushed through at a fast enough rate and 
this latency shouldn't be buffer related. I don't see any underrun or 
overrun errors on the transmitter or receiver so the signal seems to be 
running real-time.


I appreciate any advice. I'm out of ideas and have searched a lot on 
latency related to GNURadio and most of the latency I've read up on 
seems to be in the microsecond to millisecond ranges.

Thanks very much, Tom

>
> I appreciate any advice.  I'm out of ideas and have searched a lot on
> latency related to GNURadio and most of the latency I've read up on
> seems to be in the microsecond to millisecond ranges.
>
> Thanks very much, Tom
>
>

Does the application that produces transmit samples send bursts of
samples only when there is data to send, or is the transmitter always
being fed?

I ask because gnuradio can potentially have a lot of buffering, and if
your transmit app is producing continuous samples or producing samples
faster than the rate consumed, the buffering will just become full, and
it will take buffer_size/sample_rate seconds to see a change occur
through the pipeline.

If this is the issue, I recommend some kind of bursty transmission.
Some neat examples of this in precog
https://github.com/jmalsbury/pre-cog/wiki

If you need to continuously transmit, you might want to maintain a
window of buffering where the application throttles back production of
samples. One way to do this would be to simply use the consumption of rx
samples to throttle the production of tx samples.

Another alternative would be to use the gr_block api to shrink the
buffer size of each block in the transmit chain -- to minimize the
capability of gnuradio buffering data -- if the app must rely on
backpressure from gnuradio

-josh

Hello Josh,
The transmitter sends data when there is enough to fill a packet. The 
packets are flushed about every 200msec consistently. The ffmpeg encoder 
seems to have a continuous output that is pretty steady since it is 
encoding webcam images at a pretty reasonably constant bitrate with the 
settings I applied.

When I look on the oscilloscope output of the USRP tx board, the stream 
shows no gaps or pauses at all and the packets look spaced properly in 
time as if I had written the transmitter output directly to a file.

Are the solutions you recommended something I can try implementing with 
GRC (for instance throttle or the gr_block api)? Making the external 
modulator/demodulator work in a burst type mode is something I probably 
don't know how to do. I know how to run the modulator/demodulator code 
but I didn't develop it. The only thing I can think of adjusting is the 
amount of data blocks inside a packet. I can increase or reduce that 
number and that would increase/decrease the time duration between 
flushing of packets.

Thanks very much for your help, Tom




________________________________
 From: Josh Blum <josh@ettus.com>
To: discuss-gnuradio@gnu.org
Sent: Tuesday, January 22, 2013 10:54 PM
Subject: Re: [Discuss-gnuradio] Unknown cause of Latency with USRP


>
> I appreciate any advice. I'm out of ideas and have searched a lot on
> latency related to GNURadio and most of the latency I've read up on
> seems to be in the microsecond to millisecond ranges.
>
> Thanks very much, Tom
>
>

Does the application that produces transmit samples send bursts of
samples only when there is data to send, or is the transmitter always
being fed?

I ask because gnuradio can potentially have a lot of buffering, and if
your transmit app is producing continuous samples or producing samples
faster than the rate consumed, the buffering will just become full, and
it will take buffer_size/sample_rate seconds to see a change occur
through the pipeline.

If this is the issue, I recommend some kind of bursty transmission.
Some neat examples of this in precog
https://github.com/jmalsbury/pre-cog/wiki

If you need to continuously transmit, you might want to maintain a
window of buffering where the application throttles back production of
samples. One way to do this would be to simply use the consumption of rx
samples to throttle the production of tx samples.

Another alternative would be to use the gr_block api to shrink the
buffer size of each block in the transmit chain -- to minimize the
capability of gnuradio buffering data -- if the app must rely on
backpressure from gnuradio

-josh

What is the bit rate of the source and what is the bit rate of the data
transmission?

Matt

Hello Matt,
The ffmpeg encoder is set to 45 kbps bitrate and the 
transmitter/receiver is set to about 45 kbps as well.
The ffmpeg encoder doesn't converge at 45 kbps exactly, but it comes 
close. The signals are sampled at a pretty high rate of 500 kHz and the 
GRC scripts resample at 1 MHz so the USRP is run at 1MS/s rates.
Thank you, Tom




________________________________
 From: Matt Ettus <matt@ettus.com>
To: Tom Hendrick <sdtom182@yahoo.com>
Cc: "josh@ettus.com" <josh@ettus.com>; "discuss-gnuradio@gnu.org" 
<discuss-gnuradio@gnu.org>
Sent: Wednesday, January 23, 2013 10:34 AM
Subject: Re: [Discuss-gnuradio] Unknown cause of Latency with USRP




What is the bit rate of the source and what is the bit rate of the data 
transmission?

Matt



On Wed, Jan 23, 2013 at 10:32 AM, Tom Hendrick <sdtom182@yahoo.com> 
wrote:

Hello Josh,
>The transmitter sends data when there is enough to fill a packet. The packets are 
flushed about every 200msec consistently. The ffmpeg encoder seems to have a 
continuous output that is pretty steady since it is encoding webcam images at a 
pretty reasonably constant bitrate with the settings I applied.
>
>When I look on the oscilloscope output of the USRP tx board, the stream shows no 
gaps or pauses at all and the packets look spaced properly in time as if I had 
written the transmitter output directly to a file.
>
>Are the solutions you recommended something I can try implementing with GRC (for 
instance throttle or the gr_block api)? Making the external modulator/demodulator 
work in a burst type mode is something I probably don't know how to do. I know how 
to run the
 modulator/demodulator code but I didn't develop it. The only thing I 
can think of adjusting is the amount of data blocks inside a packet. I 
can increase or reduce that number and that would increase/decrease the 
time duration between flushing of packets.

There is your problem.  If you have a stream with approximately 45kbps 
that
you are trying to put through one at 45kbps, then you will eventually 
build
up an infinite backlog.  Set your transmitter and receiver to something
like 50 kbps and this won't happen.

Matt

Hello Matt,
I went back and tried your suggestion of setting the transmitter at 
50kbps and keeping the ffmpeg stream at 45 kbps. This actually gets rid 
of almost all the latency buildup however the problem is that the 
transmit GRC script running the USRP is now consistently giving 
underruns errors. This is causing the received video signal to have a 
lot of distortions.

It seems like the video stream rate has to be lower than the 
transmit/receive rate like you suggested but then this causes a problem 
because the USRP seems to not get the samples fast enough to avoid 
underruns. I'm not sure if there even is a solution to this that I could 
easily implement with GRC.

Thanks again, Tom




________________________________
 From: Tom Hendrick <sdtom182@yahoo.com>
To: Matt Ettus <matt@ettus.com>
Cc: "discuss-gnuradio@gnu.org" <discuss-gnuradio@gnu.org>; 
"josh@ettus.com" <josh@ettus.com>
Sent: Wednesday, January 23, 2013 10:40 AM
Subject: Re: [Discuss-gnuradio] Unknown cause of Latency with USRP


Hello Matt,
The ffmpeg encoder is set to 45 kbps bitrate and the 
transmitter/receiver is set to about 45 kbps as well.
The ffmpeg encoder doesn't converge at 45 kbps exactly, but it comes 
close. The signals are sampled at a pretty high rate of 500 kHz and the 
GRC scripts resample at 1 MHz so the USRP is run at 1MS/s rates.
Thank you, Tom




________________________________
 From: Matt Ettus <matt@ettus.com>
To: Tom Hendrick <sdtom182@yahoo.com>
Cc: "josh@ettus.com" <josh@ettus.com>; "discuss-gnuradio@gnu.org" 
<discuss-gnuradio@gnu.org>
Sent: Wednesday, January 23, 2013 10:34 AM
Subject: Re: [Discuss-gnuradio] Unknown cause of Latency with USRP




What is the bit rate of the source and what is the bit rate of the data 
transmission?

Matt



On Wed, Jan 23, 2013 at 10:32 AM, Tom Hendrick <sdtom182@yahoo.com> 
wrote:

Hello Josh,
>The transmitter sends data when there is enough to fill a packet. The packets are 
flushed about every 200msec consistently. The ffmpeg encoder seems to have a 
continuous output that is pretty steady since it is encoding webcam images at a 
pretty reasonably constant bitrate with the settings I applied.
>
>When I look on the oscilloscope output of the USRP tx board, the stream shows no 
gaps or pauses at all and the packets look spaced properly in time as if I had 
written the transmitter output directly to a file.
>
>Are the solutions you recommended something I can try implementing with GRC (for 
instance throttle or the gr_block api)? Making the external modulator/demodulator 
work in a burst type mode is something I probably don't know how to do. I know how 
to run the
 modulator/demodulator code but I didn't develop it. The only thing I 
can think of adjusting is the amount of data blocks inside a packet. I 
can increase or reduce that number and that would increase/decrease the 
time duration between flushing of packets.

Hello Matt,
Thank you. I had thought the same a while ago and tried the opposite of 
your suggestion by setting the ffmpeg encoder to 35 kbps and leaving the 
transmitter/receiver at 45kbps and I still had seen the latency. Even if 
I left the encoder at 45 kbps and transmitter/receiver at 45kbps how 
come I don't see the latency buildup with netcat and a cat5e cable? This 
led me to believe it was a problem with my gnuradio setup.

Thanks again, Tom





________________________________
 From: Matt Ettus <matt@ettus.com>
To: Tom Hendrick <sdtom182@yahoo.com>
Cc: "josh@ettus.com" <josh@ettus.com>; "discuss-gnuradio@gnu.org" 
<discuss-gnuradio@gnu.org>
Sent: Wednesday, January 23, 2013 10:50 AM
Subject: Re: [Discuss-gnuradio] Unknown cause of Latency with USRP





There is your problem. If you have a stream with approximately 45kbps 
that you are trying to put through one at 45kbps, then you will 
eventually build up an infinite backlog. Set your transmitter and 
receiver to something like 50 kbps and this won't happen.

Matt



On Wed, Jan 23, 2013 at 10:40 AM, Tom Hendrick <sdtom182@yahoo.com> 
wrote:

Hello Matt,
> From: Matt Ettus <matt@ettus.com>
>What is the bit rate of the source and what is the bit rate of the data 
transmission?
>>
>>When I look on the oscilloscope output of the USRP tx board, the stream shows no 
gaps or pauses at all and the packets look spaced properly in time as if I had 
written the transmitter output directly to a file.
>>
>>Are the solutions you recommended something I can try implementing with GRC (for 
instance throttle or the gr_block api)? Making the external modulator/demodulator 
work in a burst type mode is something I probably don't know how to do. I know how 
to run the
 modulator/demodulator code but I didn't develop it. The only thing I 
can think of adjusting is the amount of data blocks inside a packet. I 
can increase or reduce that number and that would increase/decrease the 
time duration between flushing of packets.

On 01/23/2013 02:29 PM, Tom Hendrick wrote:
> avoid underruns. I'm not sure if there even is a solution to this
> that I could easily implement with GRC.


In this case you are sending finite bursts of samples. A burst should
end with an of burst tag when its known that there are no more samples
available (for example). This tell the USRP TX machinery not to expect
more samples, otherwise, it generates an underflow message.

Technically GRC is agnostic to this because its just connections. The
block feeding the gnuradio stream should probably be generating this.
There is an example of this in the pre-cog link I sent you (does it in
python). Also, c++ example:
http://gnuradio.org/cgit/gnuradio.git/tree/gr-uhd/...

-josh

Thank you Matt and Josh for the suggestions.

When I look at the output of the RF connector with an oscilloscope, when 
there are underruns the signal is randomly broken up with gaps even 
though I am flushing out data packet by packet to GNURadio. Is adding an 
SOB and EOB tag to the burstthe only way to fix the breaking up of the 
signal? The underrun messages themselves do not bother me.

I thought the underruns from the USRP would happen only between sending 
packets so there would only be random gaps in the signal between the 
packets not inside the packet itself. What I'm seeing is random gaps 
inside of the data blocks of a packet causing detection issues on the 
receiver side. Without the underruns I don't see this. I will look into 
the SOB and EOB tags as suggested but thought I would ask if the 
behavior I saw was expected.

Thank you, Tom



________________________________
 From: Josh Blum <josh@ettus.com>
To: Tom Hendrick <sdtom182@yahoo.com>
Cc: Matt Ettus <matt@ettus.com>; "discuss-gnuradio@gnu.org" 
<discuss-gnuradio@gnu.org>
Sent: Wednesday, January 23, 2013 1:52 PM
Subject: Re: [Discuss-gnuradio] Unknown cause of Latency with USRP



On 01/23/2013 02:29 PM, Tom Hendrick wrote:
> avoid underruns. I'm not sure if there even is a solution to this
> that I could easily implement with GRC.


In this case you are sending finite bursts of samples. A burst should
end with an of burst tag when its known that there are no more samples
available (for example). This tell the USRP TX machinery not to expect
more samples, otherwise, it generates an underflow message.

Technically GRC is agnostic to this because its just connections. The
block feeding the gnuradio stream should probably be generating this.
There is an example of this in the pre-cog link I sent you (does it in
python). Also, c++ example:
http://gnuradio.org/cgit/gnuradio.git/tree/gr-uhd/...

-josh

On 01/24/2013 04:07 PM, Tom Hendrick wrote:
> Thank you Matt and Josh for the suggestions.
>
> When I look at the output of the RF connector with an oscilloscope,
> when there are underruns the signal is randomly broken up with gaps
> even though I am flushing out data packet by packet to GNURadio.  Is
> adding an SOB and EOB tag to the burstthe only way to fix the
> breaking up of the signal?  The underrun messages themselves do not
> bother me.
>

In a bursty transmission like this, underflow messages can only be fixed
by setting EOB. When an underflow happens, the USRP remains in transmit
state for some time window rather that switching at the EOB. If you are
relying on some sort of antenna switching, then this could be an issue
if you are receiving from the same device.

> I thought the underruns from the USRP would happen only between
> sending packets so there would only be random gaps in the signal
> between the packets not inside the packet itself.  What I'm seeing is
> random gaps inside of the data blocks of a packet causing detection
> issues on the receiver side.  Without the underruns I don't see this.
> I will look into the SOB and EOB tags as suggested but thought I
> would ask if the behavior I saw was expected.
>

The underflows should only happen in places where the device is starved.
So probably in between packets of data from your source. So I wouldnt
expect random gaps in the middle of useful data.

-josh

Hello Josh,
Thanks again I don't need to do any tx switching. I have a dedicated 
USRP for the transmitter and a dedicated USRP for the receiver for a 1 
way transmission for now.

I'm guessing I won't need to worry about the SOB and EOB unless I want 
to see the underruns go away.

This isn't an issue except for the fact the signal is broken up with 
gaps inside of each packet. I don't know why that is happening. I'm 
pretty sure the data is flushed packet by packet. Is there anything else 
you can think of that could cause this? I can go back and double check 
to make sure the transmitter is indeed flushing out data only once a 
packet is made.

Thank you, Tom




________________________________
 From: Josh Blum <josh@ettus.com>
To: Tom Hendrick <sdtom182@yahoo.com>
Cc: "discuss-gnuradio@gnu.org" <discuss-gnuradio@gnu.org>
Sent: Thursday, January 24, 2013 2:36 PM
Subject: Re: [Discuss-gnuradio] Unknown cause of Latency with USRP



On 01/24/2013 04:07 PM, Tom Hendrick wrote:
> Thank you Matt and Josh for the suggestions.
>
> When I look at the output of the RF connector with an oscilloscope,
> when there are underruns the signal is randomly broken up with gaps
> even though I am flushing out data packet by packet to GNURadio. Is
> adding an SOB and EOB tag to the burstthe only way to fix the
> breaking up of the signal? The underrun messages themselves do not
> bother me.
>

In a bursty transmission like this, underflow messages can only be fixed
by setting EOB. When an underflow happens, the USRP remains in transmit
state for some time window rather that switching at the EOB. If you are
relying on some sort of antenna switching, then this could be an issue
if you are receiving from the same device.

> I thought the underruns from the USRP would happen only between
> sending packets so there would only be random gaps in the signal
> between the packets not inside the packet itself. What I'm seeing is
> random gaps inside of the data blocks of a packet causing detection
> issues on the receiver side. Without the underruns I don't see this.
> I will look into the SOB and EOB tags as suggested but thought I
> would ask if the behavior I saw was expected.
>

The underflows should only happen in places where the device is starved.
So probably in between packets of data from your source. So I wouldnt
expect random gaps in the middle of useful data.

-josh

Hello all,

I've done some troubleshooting on an underrun problem I was having where 
the signal seemed to get broken up with
random gaps. The underruns happen when I set the data rate of my stream
lower than the external modulator/demodulator's transmission rate. The 
external modulator code writes to a file and is read by GNURadio scripts 
I made in GRC. I have a
simple transmit GRC script which reads from a file, then the signal goes 
through a rational resampler block, then a float to complex block, then 
a
constant multiplier (2^15-1), then to the USRP sink.

If I have
the file sampling rate in the external modulator and GRC script set to 
500 kS/sec and the USRP sampling rate at 1 MHz, the system works 
perfectly. I see packet by packet on the oscilloscope and when there
are underruns there is a wider gap between packets and no random gaps
inside the packets themselves.

When I set the file sampling rate
in the modulator and GRC script to 125 kS/sec, it works perfectly when
the USRP sampling rate is at 1 MHz, however if I set the USRP sampling
rate to 500 kS/sec or 250 kS/sec I start to see random gaps inside
packets. If I keep the USRP sampling rate at 500kS/sec or 250kS/sec but
 set the data stream to a higher data rate, I get no underruns and the
 signal looks great, except of course I will have an unwanted building 
latency on
the receiver side, since the data rate is higher than the transmission 
rate.

Does anyone know what could be
causing the gaps inside the packets? I expect the USRP to give zero 
signal only when it is not receiving anything from the external 
modulator code. The modulator flushes an entire packet at a time to the 
file read by the GRC script. Is it something related to the buffering
in between GNURadio blocks? I want to run the USRP at 250kS/sec so I can 
reduce the processor's load.

Thank you for your help, Tom