Unknown cause of Latency with USRP

GNU Radio
1

Hello all,

I’m seeing a lot of latency in streaming from a webcam with GNURadio and
the USRP.
The transmit chain looks like:
ffmpeg | ./modulator > tx.fifo &
then I run
sudo ./usrp_stream_tx.py

modulator is an external code I am running that is not part of gnuradio.
uhd_stream_tx is a python script I made with GRC and it reads tx.fifo,
resamples the signal, and then outputs to the USRP.

The receive chain looks like:
sudo ./usrp_stream_rx.py &
then I run
cat rx.fifo | ./demodulator | ffplay -

demodulator is an external code that is not part of gnuradio.
uhd_stream_rx is a python script that reads samples from the usrp and
writes them to rx.fifo.

First I run the receive chain, then the transmit chain with about a 1-2
second delay in between executing the receive and transmit commands.
When waiving my hand in front of the webcam, the above gives me latency
in the ffplay video player of about 5-10 seconds initially, and this
keeps growing in time to be quite high (even up to 30-45 seconds).
Someone else named Alex Csete had tried a similar DVB transmission and
said he had seen 10 seconds of lag initially with his setup. Does anyone
know what is causing this building lag? Could it be the fifo files in
the chain?

The external modulator and demodulator code runs in real-time at fixed
data rates that the usrp can handle when sampling a signal at 1MHz. The
ffmpeg encoder bitrate is set to be about the same as the modulator
code.

I also tried the same thing without the USRP by connecting a cat5e cable
between the transmitter and receiver laptops. Then I replace
./usrp_stream_rx.py and ./usrp_stream_tx.py in the tx/rx chains with
netcat commands using nc -l 1234 for the receiver, and nc 192.68.0.3 for
the transmitter. This works great and I don’t see any of the 5-10 second
latency as with the USRP.

I don’t know how or if gnuradio interblock buffers or fusb settings play
in to the latency I’m seeing? The sampling rate for the TX and RX is 1
MHz so it seems samples are pushed through at a fast enough rate and
this latency shouldn’t be buffer related. I don’t see any underrun or
overrun errors on the transmitter or receiver so the signal seems to be
running real-time.

I appreciate any advice. I’m out of ideas and have searched a lot on
latency related to GNURadio and most of the latency I’ve read up on
seems to be in the microsecond to millisecond ranges.

Thanks very much, Tom

2

I appreciate any advice. I’m out of ideas and have searched a lot on
latency related to GNURadio and most of the latency I’ve read up on
seems to be in the microsecond to millisecond ranges.

Thanks very much, Tom

Does the application that produces transmit samples send bursts of
samples only when there is data to send, or is the transmitter always
being fed?

I ask because gnuradio can potentially have a lot of buffering, and if
your transmit app is producing continuous samples or producing samples
faster than the rate consumed, the buffering will just become full, and
it will take buffer_size/sample_rate seconds to see a change occur
through the pipeline.

If this is the issue, I recommend some kind of bursty transmission.
Some neat examples of this in precog
https://github.com/jmalsbury/pre-cog/wiki

If you need to continuously transmit, you might want to maintain a
window of buffering where the application throttles back production of
samples. One way to do this would be to simply use the consumption of rx
samples to throttle the production of tx samples.

Another alternative would be to use the gr_block api to shrink the
buffer size of each block in the transmit chain – to minimize the
capability of gnuradio buffering data – if the app must rely on
backpressure from gnuradio

-josh

3

What is the bit rate of the source and what is the bit rate of the data
transmission?

Matt

4

Hello Josh,
The transmitter sends data when there is enough to fill a packet. The
packets are flushed about every 200msec consistently. The ffmpeg encoder
seems to have a continuous output that is pretty steady since it is
encoding webcam images at a pretty reasonably constant bitrate with the
settings I applied.

When I look on the oscilloscope output of the USRP tx board, the stream
shows no gaps or pauses at all and the packets look spaced properly in
time as if I had written the transmitter output directly to a file.

Are the solutions you recommended something I can try implementing with
GRC (for instance throttle or the gr_block api)? Making the external
modulator/demodulator work in a burst type mode is something I probably
don’t know how to do. I know how to run the modulator/demodulator code
but I didn’t develop it. The only thing I can think of adjusting is the
amount of data blocks inside a packet. I can increase or reduce that
number and that would increase/decrease the time duration between
flushing of packets.

Thanks very much for your help, Tom

From: Josh B. [email protected]
To: [email protected]
Sent: Tuesday, January 22, 2013 10:54 PM
Subject: Re: [Discuss-gnuradio] Unknown cause of Latency with USRP

I appreciate any advice. I’m out of ideas and have searched a lot on
latency related to GNURadio and most of the latency I’ve read up on
seems to be in the microsecond to millisecond ranges.

Thanks very much, Tom

Does the application that produces transmit samples send bursts of
samples only when there is data to send, or is the transmitter always
being fed?

I ask because gnuradio can potentially have a lot of buffering, and if
your transmit app is producing continuous samples or producing samples
faster than the rate consumed, the buffering will just become full, and
it will take buffer_size/sample_rate seconds to see a change occur
through the pipeline.

If this is the issue, I recommend some kind of bursty transmission.
Some neat examples of this in precog
https://github.com/jmalsbury/pre-cog/wiki

If you need to continuously transmit, you might want to maintain a
window of buffering where the application throttles back production of
samples. One way to do this would be to simply use the consumption of rx
samples to throttle the production of tx samples.

Another alternative would be to use the gr_block api to shrink the
buffer size of each block in the transmit chain – to minimize the
capability of gnuradio buffering data – if the app must rely on
backpressure from gnuradio

-josh

5

Hello Matt,
The ffmpeg encoder is set to 45 kbps bitrate and the
transmitter/receiver is set to about 45 kbps as well.
The ffmpeg encoder doesn’t converge at 45 kbps exactly, but it comes
close. The signals are sampled at a pretty high rate of 500 kHz and the
GRC scripts resample at 1 MHz so the USRP is run at 1MS/s rates.
Thank you, Tom

From: Matt E. [email protected]
To: Tom H. [email protected]
Cc: “[email protected]” [email protected]; “[email protected]”
[email protected]
Sent: Wednesday, January 23, 2013 10:34 AM
Subject: Re: [Discuss-gnuradio] Unknown cause of Latency with USRP

What is the bit rate of the source and what is the bit rate of the data
transmission?

Matt

On Wed, Jan 23, 2013 at 10:32 AM, Tom H. [email protected]
wrote:

Hello Josh,

The transmitter sends data when there is enough to fill a packet. The packets are
flushed about every 200msec consistently. The ffmpeg encoder seems to have a
continuous output that is pretty steady since it is encoding webcam images at a
pretty reasonably constant bitrate with the settings I applied.

When I look on the oscilloscope output of the USRP tx board, the stream shows no
gaps or pauses at all and the packets look spaced properly in time as if I had
written the transmitter output directly to a file.

Are the solutions you recommended something I can try implementing with GRC (for
instance throttle or the gr_block api)? Making the external modulator/demodulator
work in a burst type mode is something I probably don’t know how to do. I know how
to run the
modulator/demodulator code but I didn’t develop it. The only thing I
can think of adjusting is the amount of data blocks inside a packet. I
can increase or reduce that number and that would increase/decrease the
time duration between flushing of packets.

6

There is your problem. If you have a stream with approximately 45kbps
that
you are trying to put through one at 45kbps, then you will eventually
build
up an infinite backlog. Set your transmitter and receiver to something
like 50 kbps and this won’t happen.

Matt

7

Hello Matt,
I went back and tried your suggestion of setting the transmitter at
50kbps and keeping the ffmpeg stream at 45 kbps. This actually gets rid
of almost all the latency buildup however the problem is that the
transmit GRC script running the USRP is now consistently giving
underruns errors. This is causing the received video signal to have a
lot of distortions.

It seems like the video stream rate has to be lower than the
transmit/receive rate like you suggested but then this causes a problem
because the USRP seems to not get the samples fast enough to avoid
underruns. I’m not sure if there even is a solution to this that I could
easily implement with GRC.

Thanks again, Tom

From: Tom H. [email protected]
To: Matt E. [email protected]
Cc: “[email protected]” [email protected];
“[email protected]” [email protected]
Sent: Wednesday, January 23, 2013 10:40 AM
Subject: Re: [Discuss-gnuradio] Unknown cause of Latency with USRP

Hello Matt,
The ffmpeg encoder is set to 45 kbps bitrate and the
transmitter/receiver is set to about 45 kbps as well.
The ffmpeg encoder doesn’t converge at 45 kbps exactly, but it comes
close. The signals are sampled at a pretty high rate of 500 kHz and the
GRC scripts resample at 1 MHz so the USRP is run at 1MS/s rates.
Thank you, Tom

From: Matt E. [email protected]
To: Tom H. [email protected]
Cc: “[email protected]” [email protected]; “[email protected]”
[email protected]
Sent: Wednesday, January 23, 2013 10:34 AM
Subject: Re: [Discuss-gnuradio] Unknown cause of Latency with USRP

What is the bit rate of the source and what is the bit rate of the data
transmission?

Matt

On Wed, Jan 23, 2013 at 10:32 AM, Tom H. [email protected]
wrote:

Hello Josh,

The transmitter sends data when there is enough to fill a packet. The packets are
flushed about every 200msec consistently. The ffmpeg encoder seems to have a
continuous output that is pretty steady since it is encoding webcam images at a
pretty reasonably constant bitrate with the settings I applied.

When I look on the oscilloscope output of the USRP tx board, the stream shows no
gaps or pauses at all and the packets look spaced properly in time as if I had
written the transmitter output directly to a file.

Are the solutions you recommended something I can try implementing with GRC (for
instance throttle or the gr_block api)? Making the external modulator/demodulator
work in a burst type mode is something I probably don’t know how to do. I know how
to run the
modulator/demodulator code but I didn’t develop it. The only thing I
can think of adjusting is the amount of data blocks inside a packet. I
can increase or reduce that number and that would increase/decrease the
time duration between flushing of packets.

8

Hello Matt,
Thank you. I had thought the same a while ago and tried the opposite of
your suggestion by setting the ffmpeg encoder to 35 kbps and leaving the
transmitter/receiver at 45kbps and I still had seen the latency. Even if
I left the encoder at 45 kbps and transmitter/receiver at 45kbps how
come I don’t see the latency buildup with netcat and a cat5e cable? This
led me to believe it was a problem with my gnuradio setup.

Thanks again, Tom

From: Matt E. [email protected]
To: Tom H. [email protected]
Cc: “[email protected]” [email protected]; “[email protected]”
[email protected]
Sent: Wednesday, January 23, 2013 10:50 AM
Subject: Re: [Discuss-gnuradio] Unknown cause of Latency with USRP

There is your problem. If you have a stream with approximately 45kbps
that you are trying to put through one at 45kbps, then you will
eventually build up an infinite backlog. Set your transmitter and
receiver to something like 50 kbps and this won’t happen.

Matt

On Wed, Jan 23, 2013 at 10:40 AM, Tom H. [email protected]
wrote:

Hello Matt,

From: Matt E. [email protected]
What is the bit rate of the source and what is the bit rate of the data
transmission?

When I look on the oscilloscope output of the USRP tx board, the stream shows no
gaps or pauses at all and the packets look spaced properly in time as if I had
written the transmitter output directly to a file.

Are the solutions you recommended something I can try implementing with GRC (for
instance throttle or the gr_block api)? Making the external modulator/demodulator
work in a burst type mode is something I probably don’t know how to do. I know how
to run the
modulator/demodulator code but I didn’t develop it. The only thing I
can think of adjusting is the amount of data blocks inside a packet. I
can increase or reduce that number and that would increase/decrease the
time duration between flushing of packets.

9

Thank you Matt and Josh for the suggestions.

When I look at the output of the RF connector with an oscilloscope, when
there are underruns the signal is randomly broken up with gaps even
though I am flushing out data packet by packet to GNURadio. Is adding an
SOB and EOB tag to the burstthe only way to fix the breaking up of the
signal? The underrun messages themselves do not bother me.

I thought the underruns from the USRP would happen only between sending
packets so there would only be random gaps in the signal between the
packets not inside the packet itself. What I’m seeing is random gaps
inside of the data blocks of a packet causing detection issues on the
receiver side. Without the underruns I don’t see this. I will look into
the SOB and EOB tags as suggested but thought I would ask if the
behavior I saw was expected.

Thank you, Tom

From: Josh B. [email protected]
To: Tom H. [email protected]
Cc: Matt E. [email protected]; “[email protected]”
[email protected]
Sent: Wednesday, January 23, 2013 1:52 PM
Subject: Re: [Discuss-gnuradio] Unknown cause of Latency with USRP

On 01/23/2013 02:29 PM, Tom H. wrote:

avoid underruns. I’m not sure if there even is a solution to this
that I could easily implement with GRC.

In this case you are sending finite bursts of samples. A burst should
end with an of burst tag when its known that there are no more samples
available (for example). This tell the USRP TX machinery not to expect
more samples, otherwise, it generates an underflow message.

Technically GRC is agnostic to this because its just connections. The
block feeding the gnuradio stream should probably be generating this.
There is an example of this in the pre-cog link I sent you (does it in
python). Also, c++ example:
http://gnuradio.org/cgit/gnuradio.git/tree/gr-uhd/examples/c++/

-josh

10

On 01/23/2013 02:29 PM, Tom H. wrote:

avoid underruns. I’m not sure if there even is a solution to this
that I could easily implement with GRC.

In this case you are sending finite bursts of samples. A burst should
end with an of burst tag when its known that there are no more samples
available (for example). This tell the USRP TX machinery not to expect
more samples, otherwise, it generates an underflow message.

Technically GRC is agnostic to this because its just connections. The
block feeding the gnuradio stream should probably be generating this.
There is an example of this in the pre-cog link I sent you (does it in
python). Also, c++ example:
http://gnuradio.org/cgit/gnuradio.git/tree/gr-uhd/examples/c++/

-josh

11

On 01/24/2013 04:07 PM, Tom H. wrote:

Thank you Matt and Josh for the suggestions.

When I look at the output of the RF connector with an oscilloscope,
when there are underruns the signal is randomly broken up with gaps
even though I am flushing out data packet by packet to GNURadio. Is
adding an SOB and EOB tag to the burstthe only way to fix the
breaking up of the signal? The underrun messages themselves do not
bother me.

In a bursty transmission like this, underflow messages can only be fixed
by setting EOB. When an underflow happens, the USRP remains in transmit
state for some time window rather that switching at the EOB. If you are
relying on some sort of antenna switching, then this could be an issue
if you are receiving from the same device.

I thought the underruns from the USRP would happen only between
sending packets so there would only be random gaps in the signal
between the packets not inside the packet itself. What I’m seeing is
random gaps inside of the data blocks of a packet causing detection
issues on the receiver side. Without the underruns I don’t see this.
I will look into the SOB and EOB tags as suggested but thought I
would ask if the behavior I saw was expected.

The underflows should only happen in places where the device is starved.
So probably in between packets of data from your source. So I wouldnt
expect random gaps in the middle of useful data.

-josh

12

Hello Josh,
Thanks again I don’t need to do any tx switching. I have a dedicated
USRP for the transmitter and a dedicated USRP for the receiver for a 1
way transmission for now.

I’m guessing I won’t need to worry about the SOB and EOB unless I want
to see the underruns go away.

This isn’t an issue except for the fact the signal is broken up with
gaps inside of each packet. I don’t know why that is happening. I’m
pretty sure the data is flushed packet by packet. Is there anything else
you can think of that could cause this? I can go back and double check
to make sure the transmitter is indeed flushing out data only once a
packet is made.

Thank you, Tom

From: Josh B. [email protected]
To: Tom H. [email protected]
Cc: “[email protected]” [email protected]
Sent: Thursday, January 24, 2013 2:36 PM
Subject: Re: [Discuss-gnuradio] Unknown cause of Latency with USRP

On 01/24/2013 04:07 PM, Tom H. wrote:

Thank you Matt and Josh for the suggestions.

When I look at the output of the RF connector with an oscilloscope,
when there are underruns the signal is randomly broken up with gaps
even though I am flushing out data packet by packet to GNURadio. Is
adding an SOB and EOB tag to the burstthe only way to fix the
breaking up of the signal? The underrun messages themselves do not
bother me.

In a bursty transmission like this, underflow messages can only be fixed
by setting EOB. When an underflow happens, the USRP remains in transmit
state for some time window rather that switching at the EOB. If you are
relying on some sort of antenna switching, then this could be an issue
if you are receiving from the same device.

I thought the underruns from the USRP would happen only between
sending packets so there would only be random gaps in the signal
between the packets not inside the packet itself. What I’m seeing is
random gaps inside of the data blocks of a packet causing detection
issues on the receiver side. Without the underruns I don’t see this.
I will look into the SOB and EOB tags as suggested but thought I
would ask if the behavior I saw was expected.

The underflows should only happen in places where the device is starved.
So probably in between packets of data from your source. So I wouldnt
expect random gaps in the middle of useful data.

-josh

13

Hello all,

I’ve done some troubleshooting on an underrun problem I was having where
the signal seemed to get broken up with
random gaps. The underruns happen when I set the data rate of my stream
lower than the external modulator/demodulator’s transmission rate. The
external modulator code writes to a file and is read by GNURadio scripts
I made in GRC. I have a
simple transmit GRC script which reads from a file, then the signal goes
through a rational resampler block, then a float to complex block, then
a
constant multiplier (2^15-1), then to the USRP sink.

If I have
the file sampling rate in the external modulator and GRC script set to
500 kS/sec and the USRP sampling rate at 1 MHz, the system works
perfectly. I see packet by packet on the oscilloscope and when there
are underruns there is a wider gap between packets and no random gaps
inside the packets themselves.

When I set the file sampling rate
in the modulator and GRC script to 125 kS/sec, it works perfectly when
the USRP sampling rate is at 1 MHz, however if I set the USRP sampling
rate to 500 kS/sec or 250 kS/sec I start to see random gaps inside
packets. If I keep the USRP sampling rate at 500kS/sec or 250kS/sec but
set the data stream to a higher data rate, I get no underruns and the
signal looks great, except of course I will have an unwanted building
latency on
the receiver side, since the data rate is higher than the transmission
rate.

Does anyone know what could be
causing the gaps inside the packets? I expect the USRP to give zero
signal only when it is not receiving anything from the external
modulator code. The modulator flushes an entire packet at a time to the
file read by the GRC script. Is it something related to the buffering
in between GNURadio blocks? I want to run the USRP at 250kS/sec so I can
reduce the processor’s load.

Thank you for your help, Tom