MIMO Trigger (was Re: USRP2 questions)

GNU Radio
1

this case)
2 - All systems need to have a common time reference (often a 1 PPS
signal) so that they know which sample in one system corresponds to
which sample in the other
3 - All the data needs to come from / go to the same place

(etc.)

Matt,

I would like to suggest that what you have described is necessary but
not sufficient for some (most?) MIMO or multi-channel applications and
would like to describe the other pieces needed for a more complete
solution.

For many applications differential phase shift and time delay between
channels that is not observable without known external signals for
calibration and that changes every time you reset or power cycle is
unacceptable. To ensure the phase shift and time delay are stable and
repeatable the following additional steps are necessary. These steps
are ordered in increasing channel-to-channel precision.

A. Output buffers either need to be flushed to start filling when the
trigger occurs, or the next sample after the trigger needs to be marked
in some fashion so that it is identified in the data stream(s).

B. The state of the decimation counters for each USRP2 needs to be
either reset or read when triggered. An alternative is to count the
number of 100MHz clocks before the next internal sample is output to the
ethernet buffer. Not compensating for this could lead to a large time
delay variation betweeen channels depending on the decimation factor.
This is translated to a phase shift at the center frequency. For
example, we like to run narrow-band outputs with a sample rate of 50kHz.
This is a decimation of 2,000 relative to the 100MHz clock. With a
30MHz center frequency input each sample is 30/100, or 3/10 of a cycle
so a decimation difference of 2,000 clocks (20 microseconds) is as much
as 600 cycles of the RF signals.

C. The phase of the internal numerical oscillator needs to be
synchronized between USRP2 systems. This would normally be through a
common, synchronized reset, but could also be done by reading the phase
register for the numerical oscillators when the trigger/1pps occurs.
This could lead to as much as a 360 degree phase variation if not
compensated.

D. The 1PPS/trigger input needs to be resynchronized internal to the
USRP2 so that it is stable with respect to 100MHz clock edges. This
implies meeting setup and hold times for the 100MHz clock edges. Since
the 100MHz clock edges are not observable (or are they?) external to the
USRP2 the 1pps needs to be synchronized to the 10MHz reference clock and
then internal to the USRP2 resynchronized to the 100MHz such that it is
stable with respect to the 10MHz. It needs to meet 100MHz setup and
hold times internal to the FPGA. This could lead to a
channel-to-channel phase variation of 3/10 of a cycle at 30MHz,
sufficient that beams created from the multiple channels are not formed
properly with potentially significant loss in gain and high sidelobes.

A through C could be straightforward through a master synchronized reset
or by including metadata with the data block.
D is not straightforward since it requires an analysis of the timing
internal to the USRP2. The ability to implement either item D or the
master synchronized reset will not be clear until the schematic is
published. With a 10MHz external clock input I expect the best we will
be able to do for channel-to-channel coherence is to align triggers to
the 10MHz clock edge for resolution of 100nsec.

Without these synchronizations the apparent phase or time delay will
wander even for a single channel of normally synchronous data from one
trigger to another.

When you are able to make the schematic available I would be happy to
help develop the multi-channel synchronization approach. My preference
is your scenario 3 since it is the most scaleable.

Thanks,

Frank

2

Robey, Frank wrote:

Matt,

I would like to suggest that what you have described is necessary but not sufficient for some (most?) MIMO or multi-channel applications and would like to describe the other pieces needed for a more complete solution.

For many applications differential phase shift and time delay between channels that is not observable without known external signals for calibration and that changes every time you reset or power cycle is unacceptable. To ensure the phase shift and time delay are stable and repeatable the following additional steps are necessary. These steps are ordered in increasing channel-to-channel precision.

I believe we have done all of these.

In the definitions that I am going by, MIMO and phased-array have
different requirements.

MIMO requires:
    Samples are taken synchronously
    Which samples from each stream were taken at the same time is

known
All frequency conversion operations (i.e. mixing) use local
oscillators with an arbitrary, possibly unknown, but constant phase
relationship.

Phased arrays require everything that MIMO requires PLUS:
   The LO phase relationship must be known in addition to being

constant
All differential phase delays in the system (cable lengths,
amplifer phase shift, etc.) must be known, often by careful matching
and/or calibration

The MIMO issues are all taken care of by the motherboard (USRP1 or
USRP2) so long as the daughterboards all lock their local oscillators to
the motherboard clock. All of them, except for the TVRX, do this.

A. Output buffers either need to be flushed to start filling when the trigger occurs, or the next sample after the trigger needs to be marked in some fashion so that it is identified in the data stream(s).

This is done implicitly. All samples from all antennas are time-stamped
so you know which ones from each antenna correspond to each other.

B. The state of the decimation counters for each USRP2 needs to be either reset or read when triggered. An alternative is to count the number of 100MHz clocks before the next internal sample is output to the ethernet buffer. Not compensating for this could lead to a large time delay variation betweeen channels depending on the decimation factor. This is translated to a phase shift at the center frequency. For example, we like to run narrow-band outputs with a sample rate of 50kHz. This is a decimation of 2,000 relative to the 100MHz clock. With a 30MHz center frequency input each sample is 30/100, or 3/10 of a cycle so a decimation difference of 2,000 clocks (20 microseconds) is as much as 600 cycles of the RF signals.

This is how we do it. All decimation and interpolation counters (and
everything else in the signal processing chain) are triggered by the
master system timer, which is coordinated between systems.

C. The phase of the internal numerical oscillator needs to be synchronized between USRP2 systems. This would normally be through a common, synchronized reset, but could also be done by reading the phase register for the numerical oscillators when the trigger/1pps occurs. This could lead to as much as a 360 degree phase variation if not compensated.

This is also done.

D. The 1PPS/trigger input needs to be resynchronized internal to the USRP2 so that it is stable with respect to 100MHz clock edges. This implies meeting setup and hold times for the 100MHz clock edges. Since the 100MHz clock edges are not observable (or are they?) external to the USRP2 the 1pps needs to be synchronized to the 10MHz reference clock and then internal to the USRP2 resynchronized to the 100MHz such that it is stable with respect to the 10MHz. It needs to meet 100MHz setup and hold times internal to the FPGA. This could lead to a channel-to-channel phase variation of 3/10 of a cycle at 30MHz, sufficient that beams created from the multiple channels are not formed properly with potentially significant loss in gain and high sidelobes.

In the case of a MIMO setup over the MIMO cables, this is handled
internally by the FPGA, since all signals are generated from the same
100 MHz clock. In the case of “Scenario 3” from my previous post, in
which every system in synced by the 1 PPS and the 10 MHz reference, the
1 PPS input reference signal is required to be synchronous with the 10
MHz reference signal. This allows each USRP to sample it with its own
100 MHz clock without any setup/hold issues. Remember, the all the 100
MHz signals are phase locked to the same 10 MHz reference. Thus, each
system has the same conception of time.

Note that not all of the common GPS disciplined oscillators out there
produce a 1 PPS signal which is synchronous to the 10 MHz signal.
Nearly all do, but the CNS Systems CNS Clock II does not, so it would
not be a good candidate to use.

A through C could be straightforward through a master synchronized reset or by including metadata with the data block.
D is not straightforward since it requires an analysis of the timing internal to the USRP2. The ability to implement either item D or the master synchronized reset will not be clear until the schematic is published. With a 10MHz external clock input I expect the best we will be able to do for channel-to-channel coherence is to align triggers to the 10MHz clock edge for resolution of 100nsec.

The schematics are all published on the Wiki (see the USRP2 FAQ).

I assure you that we always get our 100 MHz cycles lined up properly.

Without these synchronizations the apparent phase or time delay will wander even for a single channel of normally synchronous data from one trigger to another.

I would say that all of the above, A through D are required in order to
call it MIMO, and I would not have designed the system any other way.
That being said, there is still some work to do inside the FPGA,
firmware, and host software.

When you are able to make the schematic available I would be happy to help develop the multi-channel synchronization approach. My preference is your scenario 3 since it is the most scaleable.

That is great! The schematics are on the wiki at
http://gnuradio.org/trac/wiki/USRP2

Please let me know if you have further questions or want to discuss
anything in more detail.

Matt