USRP2 spontaneously dies while transmitting

Hi all-

Occasionally, while transmitting, my USRP2 will die – that is, it
stops reliably transmitting and then it drops off the network and
doesn’t come back until restarted. Any ideas? I’m running some
vaguely recent firmware, but I’ve also seen this problem with the
version from the factory.

(This is one of the beta units, which is supposed to be identical to
the modern version for these purposes.)

Here’s what triggered it most recently:

$ tx_samples -i 20 -r -I /data/cal_signal
Failed to enable realtime scheduling
usrp2: failed to enable realtime scheduling
Daughterboard configuration:
baseband_freq=0.000000
duc_freq=0.000000
residual_freq=0.000000
inverted=no

^C
$ tx_samples -i 20 -r -I /data/cal_signal
Failed to enable realtime scheduling
terminate called after throwing an instance of ‘std::runtime_error’
what(): No USRPs found on interface eth0
Aborted

$ tx_samples -i 20 -r -I /data/cal_signal
Failed to enable realtime scheduling
terminate called after throwing an instance of ‘std::runtime_error’
what(): No USRPs found on interface eth0
Aborted

Thanks,
Andy

FWIW, I’m currently struggling with the very same issue. I just flashed
to
the newest firmware/fpga with no improvement. The issue manifests at
every
interpolation I’ve tried.

Jordan

On Tue, 2009-08-04 at 14:46 -0600, Jordan J Riggs wrote:

FWIW, I’m currently struggling with the very same issue. I just
flashed to the newest firmware/fpga with no improvement. The issue
manifests at every interpolation I’ve tried.

Can you look at:

http://gnuradio.org/trac/wiki/USRP2GigEReports

…and see whether your configuration is listed?

The issue may be related to flow control handling on your motherboard
chipset.

Johnathan

I’m not sure how to tell exactly which chipset I’m using, but I’m
running
brand-new Dell M6400 laptops. That page says that Broadcom BCM5755M on
Dell
mobos works fine.

Jordan

On Tue, Aug 4, 2009 at 3:11 PM, Johnathan C. <

On Tue, Aug 04, 2009 at 04:38:56PM -0400, Andrew L. wrote:

what(): No USRPs found on interface eth0
Aborted

Thanks,
Andy

First off, I highly suggest that you add yourself to group usrp if
you’re not already, then add this line to /etc/security/limits.conf:

@usrp - rtprio 50

Do you have the TTL serial port on the USRP2 connected? If so, what
was the last thing it said?

Eric

On Tue, Aug 4, 2009 at 5:52 PM, Eric B.[email protected] wrote:

the modern version for these purposes.)
inverted=no
terminate called after throwing an instance of ‘std::runtime_error’
@usrp - rtprio 50
Already tried that and it didn’t help.

I’m using an Intel 82566DC-2 (e1000e) attached to a switch, which is a
terrible idea, but I don’t really mind occasional flakiness.

My main objection is that I have to reboot the USRP2 when it dies.
I’d expect it to recover.

Also, tx_samples often freaks out and saturates the network, even at
interpolation 20. IMO that shouldn’t happen no matter what happens to
priorities, the link, the network card, or the USRP.

Do you have the TTL serial port on the USRP2 connected? If so, what
was the last thing it said?

Nope. I might try that eventually, but I probably won’t get a chance
for a week or two.

–Andy

On Tue, Aug 04, 2009 at 05:55:27PM -0400, Andrew L. wrote:

I’m using an Intel 82566DC-2 (e1000e) attached to a switch, which is a
terrible idea, but I don’t really mind occasional flakiness.

My main objection is that I have to reboot the USRP2 when it dies.
I’d expect it to recover.

Also, tx_samples often freaks out and saturates the network, even at
interpolation 20. IMO that shouldn’t happen no matter what happens to
priorities, the link, the network card, or the USRP.

You’re running it through a switch. We tell you not to do that.
Your switch is probably declining to be flow controlled by the USRP2,
hence there’s no back pressure being applied to the host, thus the
unbounded amount of data sent from the host.

Rule 1: Don’t connect your USRP2 via a switch

Rule 2: If you think you know what you’re doing and understand the ins
and outs of ethernet PAUSE frames and asymmetric flow control,
and can get your switch to accept flow control from the USRP,
then it may be possible to make it work through a switch.

Rule 3: See Rule 1.

Do you have the TTL serial port on the USRP2 connected? If so, what
was the last thing it said?

Nope. I might try that eventually, but I probably won’t get a chance
for a week or two.

Eric

I also edited limits.conf as instructed, no luck.
I’m running a Broadcom Corporation NetXtreme BCM5761e Gigabit Ethernet
PCIe
(rev 10)
No switch.

Jordan

On Tue, Aug 4, 2009 at 6:05 PM, Eric B.[email protected] wrote:

   and can get your switch to accept flow control from the USRP,
   then it _may_ be possible to make it work through a switch.

Rule 3: See Rule 1.

Would you accept a patch to error out on send if sustained data rates
more than, say, 150% of what they should be are seen?

Also, I fully expect (and don’t mind) some drops due to running
through a switch. I’m only running this thing when the network is
otherwise nearly idle and even then I’m at high interpolation. It
still seems buggy that the USRP2 crashes as a result.

Is the USRP2 ethernet protocol documented somewhere? Quick googling
didn’t find it.

–Andy

On Tue, Aug 04, 2009 at 07:44:21PM -0400, Andrew L. wrote:

   and outs of ethernet PAUSE frames and asymmetric flow control,
   and can get your switch to accept flow control from the USRP,
   then it _may_ be possible to make it work through a switch.

Rule 3: See Rule 1.

Would you accept a patch to error out on send if sustained data rates
more than, say, 150% of what they should be are seen?

I’d definitely consider one. Be sure it doesn’t burn many cpu cycles.
When running with interp = 4, even on big iron, we don’t have much to
spare.

Also, I fully expect (and don’t mind) some drops due to running
through a switch. I’m only running this thing when the network is
otherwise nearly idle and even then I’m at high interpolation. It
still seems buggy that the USRP2 crashes as a result.

I agree that it shouldn’t crash or disappear. Feel free to debug and
fix it.

Is the USRP2 ethernet protocol documented somewhere? Quick googling
didn’t find it.

The existing protocol was stopgap kludge that escaped into the wild.
The packet formats are defined in
usrp2/firmware/include/usrp2_eth_packet.h

The entire on-the-wire protocol will be getting reworked to use VRT.

Eric

Timothy,

If you make any progress on this I’d LOVE to hear about it. This problem
is
putting a serious crimp on the work I’m doing.

Jordan

I’m also having trouble transmitting on USRP2’s with Dell M6400’s - I’ve
put
off posting until I’ve had a chance to look into the problem - not using
a
switch and the transmitting USRP fails fairly quickly – occasionally
lasting about 15minutes (Usually this appear happen after the USRP has
been
used to do something else recently - but that could be my imagination)

The ethernet controller in these is the Broadcom NetExtreme BCM5761e
according to lspci which isnt currently mentioned on the link a few
emails
back.

I’d be interested in the mentioned patch (even if it reduces performance
too
much for inclusion in the standard source) to avoid this problem if one
gets
written before I finish playing with Rx side stuff.

Cheers,

Tim

On Tue, Aug 4, 2009 at 10:18 PM, Jordan J Riggs [email protected]
wrotes