Expected behavior during underruns

GNU Radio
1

Dear list,

What is the expected behavior when we get underruns with the UHD and the
flow control using the USRP2?

  1. When we get underruns, does the usrp2 send some random samples until
    the host computer manage to feed the usrp2 and then starts to consume
    the ‘old’ samples from the host computer?

Or

  1. When we get underruns, does the usrp2 send some random samples while
    throwing away old samples until the host computer manage to feed the
    usrp2 with samples to be sent in the future?

Or does it work in another way?

Br,
Patrik

2

On 12/09/2010 02:11 AM, Patrik E. wrote:

Dear list,

What is the expected behavior when we get underruns with the UHD and the flow
control using the USRP2?

The host will get an async packet marked underflow and print a U. You
can get the timestamp of this packet through the recv_async_msg
interface.

The USRP2 will continue to send subsequent packets ASAP until the burst
has ended. It is possible to configure the device to discard all packets
until a new burst begins.

When underflowing, there are no random samples, the TX state machine
goes into an IDLE mode. The effect of this depends on the daughterboard.
But its basically turning the transmitter off.

-JOsh

3

On Thu, Dec 9, 2010 at 5:11 AM, Patrik E.
[email protected] wrote:

Or does it work in another way?
Based on my observations with a scope, there is no output during
underruns; the usrp2 never sends random samples. Also, the usrp2
doesn’t drop samples, so late samples will transmit immediately upon
late arrival. My applications deal with continuous streaming, which
requires occasionally dropping samples to maintain synchronization,
and that occurs on the host.

I should note that most of my testing was before host based flow
control was merged. I haven’t noticed any significant changes
afterwards, so I assume that the underrun behaviour is unchanged.

Thomas

4

Dear list,
2) When we get underruns, does the usrp2 send some random
continuous streaming, which requires occasionally dropping
samples to maintain synchronization, and that occurs on the host.

I should note that most of my testing was before host based
flow control was merged. I haven’t noticed any significant
changes afterwards, so I assume that the underrun behaviour
is unchanged.

Thomas

Thanks for your answer Thomas!

Our observations shows also that the usrp2 doesn’t drop any samples when
an underrun occur. If we drop samples at the host, can we get a
continous stream of sample at 25Msamp/s that is synchronous? What is
your maximum continous bandwidth?

I’ve been looking in the source code and tried to find out where to
implement the sample dropping. My thoughts so far is that it should be
somewhere in vrt_packet_handler.hpp and/or io_impl.cpp in the send
functions and get_send_buffs function.

The underruns are received(detected) in the pirate loop in io_impl.cpp,
correct?! It’s not obvious for me how to figure out which samples that
I should dropp to get the rest of the samples synchronized again. Can
you give me some hints?

Br,
Patrik

5

Thanks for your answer Thomas!

Our observations shows also that the usrp2 doesn’t drop any samples
when an underrun occur. If we drop samples at the host, can we get a
continous stream of sample at 25Msamp/s that is synchronous? What is
your maximum continous bandwidth?

Underflows occur on transmit, nothing can drop because this is a lack of
samples.

Overflow occurs on receive. The host cannot back-pressure the USRP2, so
samples backup into the host and are dropped in the kernel at the packet
level.

Every packet has a timestamp which can be used to determine exactly what
packets have gone missing in a continuous stream. See the benchmark_rx
example which demonstrates this.

I’ve been looking in the source code and tried to find out where to
implement the sample dropping. My thoughts so far is that it should
be somewhere in vrt_packet_handler.hpp and/or io_impl.cpp in the send
functions and get_send_buffs function.

Is there something that you think the streaming API is missing please
let me know. I think you can accomplish what you need with the API
without modifying the underlying implementation.

The underruns are received(detected) in the pirate loop in
io_impl.cpp, correct?! It’s not obvious for me how to figure out
which samples that I should dropp to get the rest of the samples
synchronized again. Can you give me some hints?

Asynchronous messages (which include transmit underflow) are received in
this loop and pushed into a queue. You can retrieve these events with
device::recv_async_msg(…)

-Josh

6

Underflows occur on transmit, nothing can drop because this is a lack
of samples.

That’s true! But I don’t want the rest of the samples to be delayed due
to the underrun.

If I understand correctly, the transmit situation that Patrik
describes is behaviour after the underflow. For a
continuous stream, the effect to subsequent samples is an
added time offset - equal to the duration of the underflow -
relative to the original start of the stream.

Yes you understood me correct and it is this offset I want to be zero
again after an underrun. But I have a continuous stream. For example: if
I expect that eth-packets (1) should be moved to the USRP2 and then
transmitted, but if I get an underrun the sequence can be like this (2)
then there is a offset between the real-time for p2 and the actual
transmitting time. The solution is then to drop p2 (and maybe p3) in the
host so the sequence becomes (3)(if we also drop p3 p1,U,E,p4,p5
E=empty) where p4,p5 are transmitted at the original expected
time(compare (1) with (3)).

p1,p2,p3,p4,p5 (1) excpected sequence
p1,U ,p2,p3,p4,p5 (2) with underrun
p1,U ,p3,p4,p5 (3) with underrun and dropping

I’ve been looking in the source code and tried to find out where to
implement the sample dropping. My thoughts so far is that it should
be somewhere in vrt_packet_handler.hpp and/or io_impl.cpp in the send
functions and get_send_buffs function.

Is there something that you think the streaming API is missing please
let me know. I think you can accomplish what you need with the API
without modifying the underlying implementation.

Yes, sure I could probably solve this with the API. But I utilize that
the send-function in the API fragments the payload(samples) to fit in
eth-packets to the usrp2. For example: if I send 25Msamp @ 25Msamp/s
with one call to the send function those samples will be split into
several eth-packets and transmitted during 1 second if we don’t get any
underruns (if we get underruns they are transmitted but it takes more
time than 1 second), correct? If I handle the underruns with the API, I
have to drop some samples and wait for some part of 1 second to get
synchronized alignment on the next second after that. This creates a gap
of some part of second where I’m not transmitting.

It would be really nice if the send-function in the API have a TDMA mode
where fragments of samples are dropped when an underrun occur so that
the majority of the samples are aligned to the start_of_burst_time (see
above example (1) and (3)). With this feature it is not necessary for
the programmer to know if an underrun has occurred and how to handle
this. Instead the programmer knows that if an underrun occurred some
samples are dropped but the rest of the sample are transmitted at the
expected time without any offset. Is this doable?

-Patrik

7

It would be really nice if the send-function in the API have a TDMA
mode where fragments of samples are dropped when an underrun occur so
that the majority of the samples are aligned to the
start_of_burst_time (see above example (1) and (3)). With this
feature it is not necessary for the programmer to know if an underrun
has occurred and how to handle this. Instead the programmer knows
that if an underrun occurred some samples are dropped but the rest of
the sample are transmitted at the expected time without any offset.
Is this doable?

This should already be the case. If you timestamp every packet, then the
underflow recovery is handled by hardware. That is, the FPGA will drop
until the timestamp is <= current time, and then wait until the
timestamp == current time, so realignment is automatic.

Are you timestamping every packet? This calculation should be
last_time_spec + time_spec_t(0, num_samps_in_last_packet,
host_sample_rate)

Let me know if you are not seeing this behavior.
Thanks,
-Josh

8

On Tue, Dec 14, 2010 at 6:34 PM, Josh B. [email protected] wrote:

Our observations shows also that the usrp2 doesn’t drop any samples
when an underrun occur. If we drop samples at the host, can we get a
continous stream of sample at 25Msamp/s that is synchronous? What is
your maximum continous bandwidth?

Underflows occur on transmit, nothing can drop because this is a lack of
samples.

If I understand correctly, the transmit situation that Patrik
describes is behaviour after the underflow. For a continuous stream,
the effect to subsequent samples is an added time offset - equal to
the duration of the underflow - relative to the original start of the
stream.

For certain real-time applications, paired spectrum TDMA for example,
this makes underruns devastating. In these cases, samples that were
supposed to be transmitted when the underrun occurred, but didn’t for
whatever reason, are irrelevant and need to be dropped rather than
transmitted best effort. AFAIK, the only way to currently handle this
is to catch the underrun at the host and drop a handful of packets so
that the next start-of-burst reaches the USRP2 ahead of the deadline.

I’ve been looking in the source code and tried to find out where to
implement the sample dropping. My thoughts so far is that it should
be somewhere in vrt_packet_handler.hpp and/or io_impl.cpp in the send
functions and get_send_buffs function.

Is there something that you think the streaming API is missing please
let me know. I think you can accomplish what you need with the API
without modifying the underlying implementation.

I agree. Poking around the guts is a bad place to start and unnecessary.

The OpenBTS USRP2 implementation uses get_time_now() to resync the
host when an underrun occurs. Obviously, you shouldn’t bother sending
a packet if the difference between usrp time and the transmit
timestamp is negative or near zero.

http://github.com/ttsou/openbts-uhd/blob/master/public-trunk/Transceiver/UHDDevice.cpp

Thomas

9

On Wed, Dec 15, 2010 at 2:04 PM, Josh B. [email protected] wrote:

This should already be the case. If you timestamp every packet, then the
underflow recovery is handled by hardware. That is, the FPGA will drop
until the timestamp is <= current time, and then wait until the
timestamp == current time, so realignment is automatic.

I believe you mean timestamp is >= current time. Regardless though,
this is not the case based on my observations.

Are you timestamping every packet? This calculation should be
last_time_spec + time_spec_t(0, num_samps_in_last_packet, host_sample_rate)

Yes. I timestamp every packet. I also mark every packet
start_of_burst. I wonder if late unmarked packets will still transmit,
though this would not remove the necessity of detecting underruns on
the host.

Thomas

10

This should already be the case. If you timestamp every
packet, then the underflow recovery is handled by hardware.
That is, the FPGA will drop until the timestamp is <= current
time, and then wait until the timestamp == current time, so
realignment is automatic.

Are you timestamping every packet? This calculation should be
last_time_spec + time_spec_t(0, num_samps_in_last_packet,
host_sample_rate)

Yes, I timestamp every packet sent by the send-function but my
observation shows that the underrun realignment is not automatic. I also
think that the underrun recovery works for packets, but it doesn’t work
for fragments of packets.

Consider the case when you have a packet P1 with timestamp 3.0 and the
number of samples are equal to 1 second in transmission duration. P1 is
then fragmented into several fragments by the UHD, call them
f1,f2,f3,f4,f5. The timestamp and start_of_burst is set at f1 in P1
correct? And the remaining fragments doesn’t get any timestamp and are
set to be transmitted immediate, correct? The f1 will block the rest of
the fragments until the usrp2 time is equal to the timestamp of f1 and
then all of the fragments will be transmitted in a sequence if we do not
get an underrun (1).

If we instead get an underrun after that the first fragment is moved to
the usrp2 either nothing or a CW will be transmitted for a while between
f1 and f2 which will introduce an unknown and unwanted offset (2). To
eliminate this offset the host can drop f2 and timestamp all of the
fragments so that the fragments in the end of the transmission happen at
the expected time (3).

f1,f2,f3,f4,f5 (1) expected sequence
f1,U ,f2,f3,f4,f5 (2) with underrun
f1,U ,f3,f4,f5 (3) with underrun and dropping

As we are talking about the fragments of a packet now, this must be
handled in vrt_packet_handler.hpp and/or io_impl.cpp.

http://code.ettus.com/redmine/ettus/projects/uhd/repository/revisions/71e1763332141603e9edba097fd19b00e9a76ab8/entry/host/lib/transport/vrt_packet_handler.hpp

Look at the state SEND_MODE_FULL_BUFF at line 414

//calculate new flags for the fragments
427: if_packet_info.has_tsi = metadata.has_time_spec and
first_fragment;
428: if_packet_info.has_tsf = if_packet_info.has_tsi;
429: if_packet_info.sob = metadata.start_of_burst and
first_fragment;
430: if_packet_info.eob = metadata.end_of_burst and
final_fragment;

To me it look likes the first fragment only has the timestamp for the
packet…

Thanks for your answer by the way!

-Patrik

11

Yes, I timestamp every packet sent by the send-function but my
observation shows that the underrun realignment is not automatic. I
also think that the underrun recovery works for packets, but it
doesn’t work for fragments of packets.

Correct. Its at the packet level (thats UDP packet level 360~ish
samples).

If we instead get an underrun after that the first fragment is moved
handled in vrt_packet_handler.hpp and/or io_impl.cpp.

I think you will find that hardware recovery is going to be faster then
anything you you manipulate in software.

I think the problem is that you are sending very large bursts with only
the first packet having a time stamp. Can you try your own fragmentation
into smaller sub-packets, each sub-packet with a timestamp?
tx_timed_samples.cpp has an example of fragmenting a buffer into smaller
packets.

http://code.ettus.com/redmine/ettus/projects/uhd/repository/revisions/71e1763332141603e9edba097fd19b00e9a76ab8/entry/host/lib/transport/vrt_packet_handler.hpp

Look at the state SEND_MODE_FULL_BUFF at line 414 //calculate new
flags for the fragments 427: if_packet_info.has_tsi =
metadata.has_time_spec and first_fragment; 428:
if_packet_info.has_tsf = if_packet_info.has_tsi; 429:
if_packet_info.sob = metadata.start_of_burst and first_fragment;
430: if_packet_info.eob = metadata.end_of_burst and
final_fragment;

Correct, this could be modified to time-stamp every packet in the burst.
But a pain to do as the code there knows nothing about the sample rate.

-Josh

12

Hello,

I was running Hydra MIMO testbench on USRP and got stream of this
message:

usrp_source: overrun
drop 16384 bytes

Does anyone know what it means by overrun?
Thanks,

Thomas

From: Josh B. [email protected]
To: Patrik E. [email protected]
Cc: “[email protected]” [email protected]
Date: 12/16/2010 12:29 PM
Subject: Re: [Discuss-gnuradio] Expected behavior during
underruns
Sent by: discuss-gnuradio-bounces+thomas.h.kim=removed_email_address@domain.invalid

Yes, I timestamp every packet sent by the send-function but my
observation shows that the underrun realignment is not automatic. I
also think that the underrun recovery works for packets, but it
doesn’t work for fragments of packets.

Correct. Its at the packet level (thats UDP packet level 360~ish
samples).

If we instead get an underrun after that the first fragment is moved
to the usrp2 either nothing or a CW will be transmitted for a while
between f1 and f2 which will introduce an unknown and unwanted offset
(2). To eliminate this offset the host can drop f2 and timestamp all
of the fragments so that the fragments in the end of the transmission
happen at the expected time (3).

f1,f2,f3,f4,f5 (1) expected sequence
f1,U ,f2,f3,f4,f5 (2) with
underrun f1,U ,f3,f4,f5 (3) with underrun
and dropping

As we are talking about the fragments of a packet now, this must be
handled in vrt_packet_handler.hpp and/or io_impl.cpp.

I think you will find that hardware recovery is going to be faster then
anything you you manipulate in software.

I think the problem is that you are sending very large bursts with only
the first packet having a time stamp. Can you try your own fragmentation
into smaller sub-packets, each sub-packet with a timestamp?
tx_timed_samples.cpp has an example of fragmenting a buffer into smaller
packets.

http://code.ettus.com/redmine/ettus/projects/uhd/repository/revisions/71e1763332141603e9edba097fd19b00e9a76ab8/entry/host/lib/transport/vrt_packet_handler.hpp

Look at the state SEND_MODE_FULL_BUFF at line 414 //calculate new
flags for the fragments 427: if_packet_info.has_tsi =
metadata.has_time_spec and first_fragment; 428:
if_packet_info.has_tsf = if_packet_info.has_tsi; 429:
if_packet_info.sob = metadata.start_of_burst and first_fragment;
430: if_packet_info.eob = metadata.end_of_burst and
final_fragment;

Correct, this could be modified to time-stamp every packet in the burst.
But a pain to do as the code there knows nothing about the sample rate.

-Josh