Updated packet format on USRP inband signaling

Would those of you with an interest in USRP inband signaling, please
take a look at the latest proposed packet format. Now’s a good time
to change things :wink:

It’s in trunk/usrp/doc/inband-signaling-usb

http://gnuradio.org/trac/browser/gnuradio/trunk/usrp/doc/inband-signaling-usb

Thanks,
Eric

Some preliminary questions:

How are the operations linked with the transmit sequences? Are
operations sent down in bulk, or one at a time? Will the RX chain
simply start and return 504-byte length packets until the next TX
timestamp is sent down? Could this starve the USB bandwidth?

Brian

On 2/25/07, Eric B. [email protected] wrote:

On Sun, Feb 25, 2007 at 07:29:01PM -0500, Brian P. wrote:

Some preliminary questions:

How are the operations linked with the transmit sequences?

I’m not sure I understand this question.

I am not sure how the operations being sent down are associated with a
payload. Are they just 32-bit aligned in either an in or out packet?
If so, should there be something in the header to say how many
operations are there before the actual modulated data starts?

Are operations sent down in bulk, or one at a time?

You can send as many as will fit in the payload.
Unless you doing something like hopping, I suspect that command
packets are relatively infrequent.

It looks like the assumption I made just previously is accurate - the
operations are in the payload along with modulation data.

Brian

On Sun, Feb 25, 2007 at 08:10:06PM -0500, Brian P. wrote:

If so, should there be something in the header to say how many
operations are there before the actual modulated data starts?

If Chan == 0x1f the payload contains control operations, otherwise the
payload contains homogeneous samples (the type of which is specified
by the contents of some register(s) that were set earlier).

Are operations sent down in bulk, or one at a time?

You can send as many as will fit in the payload.
Unless you doing something like hopping, I suspect that command
packets are relatively infrequent.

It looks like the assumption I made just previously is accurate - the
operations are in the payload along with modulation data.

No, they’re distinct, based on the Chan field in the header.

My thinking behind this was to keep life simple for the common case:
If Chan != 0x1f, clock payload into appropriate signal processing
pipeline.
If Chan == 0x1f, do the potentially slow, complicated work…

Eric

On Sun, Feb 25, 2007 at 05:28:47PM -0800, Eric B. wrote:

payload. Are they just 32-bit aligned in either an in or out packet?

Unless you doing something like hopping, I suspect that command

Eric

Would you prefer that both types of data occurred in the same packet?

Eric

On Sun, Feb 25, 2007 at 07:29:01PM -0500, Brian P. wrote:

Some preliminary questions:

How are the operations linked with the transmit sequences?

I’m not sure I understand this question.

Are operations sent down in bulk, or one at a time?

You can send as many as will fit in the payload.
Unless you doing something like hopping, I suspect that command
packets are relatively infrequent.

Will the RX chain simply start and return 504-byte length packets
until the next TX timestamp is sent down?

I think this depends on how everything is configured. I think you’ll
have to do something (set some register) to enable the Rx at all.
Likewise for Tx. As to how the Rx and Tx interact, I think that’s
still to be worked out. The right answer may depend on the
application.

E.g., a half-duplex “push to talk” radio (simplex or trunked) would
probably run the Rx all the time, except when it was transmitting.

A packet based radio may run some power measurement / RSSI code in the
FPGA and only return Rx data when it wasn’t transmitting and the power
exceeded some threshold.

Could this starve the USB bandwidth?

Only if the sum total of IN + OUT is > 32MB/s

Eric

On Sun, Feb 25, 2007 at 08:30:19PM -0500, Brian P. wrote:

On 2/25/07, Eric B. [email protected] wrote:

My thinking behind this was to keep life simple for the common case:
If Chan != 0x1f, clock payload into appropriate signal processing pipeline.
If Chan == 0x1f, do the potentially slow, complicated work…

Sorry - my misunderstanding. This makes sense.

Would you want to provision 2 control channels - one for each AD9862
on the USRP?

No, just a single control channel. There are 2^10 register addresses
available :wink:

Eric

I am definitely interested in the capabilities of the in-band
signaling, but my lack of experience is going to show through in this
e-mail. =)

I am not sure I understand the meaning behind the 5-bit channel entry:
I thought that this might refer to channels within the total
bandwidth of the incoming / outgoing signals…you said that these
are “logically independent of the others” samples, and I was
thinking this was referring perhaps to frequency division of some
sort (like bin values in an FFT). Could you help me understand what
you mean by channels here?

Is the control channel (0x1f), configuring the hardware(setting
center frequency, registers for PGA gain…possibly an AGC in the
future, etc…using the I2C & SPI…I’m actually not very familiar
with SPI & I2C…)?

Just from looking through some of the usrp host files (prims,
standard, etc.) it looks like the SPI is for the devices on the
motherboard, and the I2C is for the daughterboards…is this correct?

Also, I was wondering about Timestamping…In the document you wrote

“If a packet reaches the head of the transmit queue, and the current
time is later than the timestamp, an error is assumed to have occurred
and the packet is discarded.”

so, if the packet that is ready to be transmitted is early, then it
should be discarded? Maybe I am misunderstanding how the time stamp
is going to be done. If the timestamp (for the OUT packet) is when
the data “should” go out of the D/A, then why wouldn’t it just wait
for the time and send . Would that be too risky…meaning things
might get backed up waiting for an incorrect time stamp that is
marked two hours into the future? Wouldn’t the packets following
also be “early” as well, or could that be corrected?

Thanks for any clarification that you can give,

David S.

On Mon, Feb 26, 2007 at 01:22:22AM -0500, David S. wrote:

I am definitely interested in the capabilities of the in-band
signaling, but my lack of experience is going to show through in this
e-mail. =)

No problem.

I am not sure I understand the meaning behind the 5-bit channel entry:
I thought that this might refer to channels within the total
bandwidth of the incoming / outgoing signals…you said that these
are “logically independent of the others” samples, and I was
thinking this was referring perhaps to frequency division of some
sort (like bin values in an FFT). Could you help me understand what
you mean by channels here?

It’s just away to think about how to route samples to/from the A/D’s
and D/A’s. (5-bits is probably overkill, but I’m dreaming of a day
when Matt provides us h/w that can do 16x16 MIMO.) If you’re running
two daughterboards, you may want to send an independent stream of
samples to each. Now, it may be that you want the samples interleaved
(minimizing latency) on a single logical channel, but it also may be
that you are running the two daughterboards fully independently of
each other – you could imagine them running with different interp or
decim rates (not implemented currently.) In that case, one stream of
samples would be assigned to Chan 0, the other to Chan 1.

Is the control channel (0x1f), configuring the hardware(setting
center frequency, registers for PGA gain…possibly an AGC in the
future, etc…using the I2C & SPI…I’m actually not very familiar
with SPI & I2C…)?

Yes. SPI and I2C are relatively low speed buses commonly used to read
and write values into pieces of h/w (chips). The differ in the number
of wires, addressing, and the way the bits are transmitted, but
basically they’re two common ways of solving a similar problem.

Just from looking through some of the usrp host files (prims,
standard, etc.) it looks like the SPI is for the devices on the
motherboard, and the I2C is for the daughterboards…is this correct?

Everything goes to both the motherboard and the daughterboards.
SPI is used to configure the AD9862s, the EEPROMs are controlled over
I2C. The daughteboard-specific code uses the appropriate interface to
control the chips on the daughterboard.

Also, I was wondering about Timestamping…In the document you wrote

“If a packet reaches the head of the transmit queue, and the current
time is later than the timestamp, an error is assumed to have occurred
and the packet is discarded.”

so, if the packet that is ready to be transmitted is early, then it
should be discarded? Maybe I am misunderstanding how the time stamp
is going to be done.

If a packet is ready to be transmitted and it’s early, it’s held
until time to transmit. If it’s late, it’s discarded.

If the timestamp (for the OUT packet) is when the data “should” go
out of the D/A, then why wouldn’t it just wait for the time and
send.

It does. Using wall clock time for illustrative purposes, if the
current time is 8:30 and the packet timestamp says 8:25, the packet
got here late, and we drop it :wink:

Would that be too risky…meaning things
might get backed up waiting for an incorrect time stamp that is
marked two hours into the future? Wouldn’t the packets following
also be “early” as well, or could that be corrected?

FWIW, a 32-bit counter, clocked at 64MHz, rolls over every 67 seconds.

There’s really very little RAM to store the packets on the FPGA. The
host needs to send them in proper order. The timestamp is used to
ensure
that they don’t go to the D/A’s until the right instant. This is
useful for hitting a TDMA time slot.

Thanks for any clarification that you can give,

I hope this helped!

David S.

Eric

On 2/25/07, Eric B. [email protected] wrote:

My thinking behind this was to keep life simple for the common case:
If Chan != 0x1f, clock payload into appropriate signal processing pipeline.
If Chan == 0x1f, do the potentially slow, complicated work…

Sorry - my misunderstanding. This makes sense.

Would you want to provision 2 control channels - one for each AD9862
on the USRP?

Brian

On Feb 26, 2007, at 2:04 AM, Eric B. wrote:

As Mr. Burns once said…Excellent.

is going to be done.

Using wall clock time for illustrative purposes, if the
current time is 8:30 and the packet timestamp says 8:25, the packet
got here late, and we drop it :wink:

Ah, I was definitely thinking that time went backwards…sorry about
that =)

Thanks for any clarification that you can give,

I hope this helped!

THANKS ERIC! you’ve cleared up alot of things for me!

David S.

Eric B. wrote:

Would those of you with an interest in USRP inband signaling, please
take a look at the latest proposed packet format. Now’s a good time
to change things :wink:

It’s in trunk/usrp/doc/inband-signaling-usb

TIMESTAMP FORMAT:
What timestamp format do you have in mind.
32 bit samplenumbers, or something else.
And are samplenumbers, input-samples, output-samples or 64 Mhz clocks.
64 Mhz clocks would be the most accurate, but could be awkward to work
with when using fractional interpolation or decimation rates.
(I am working on these)
When using samplenumbers of input-samples or output-samples, you have
the problem that different channels could work at different rates.
When using seconds or milliseconds as timestanmps you have the problem
of reduced accuracy.

In the multi_usrp fpga code which I use to synchronise multiple usrps I
use a 32 bit samplenumember to align different channels from different
usrps.
(The samplenumber counters are synchronised in hardware using GPIO 15 on
daughterboard RA)

In my pre-inbandsignaling code I am now using a whole channel which just
sends out samplenumbers.
This way I can align channels from different usrps, sample-accurate.

Using in-band signaling I could get away with just a timestamp per
packet, but then it must have at least 1 sample granularity.

TIMESTAMP IN CONTROL CHANNEL:
What I am missing is a desciption on how the timestamp in the control
channel operations is used.

I suggest it would have the following meaning:
On (a bundle of) write and I2C READ and WRITE and SPI READ and WRITE
operations it would mean:

Start these operation at specified timestamp.
If the timestamp is 0xffffffff it is interpreted as “Now”.

On READ_REPLY operations it would mean:
These value were read (all reads were finished) at given timestamp.
If the timestamp is 0xffffffff it is interpreted as , “An error has
occured, or timeout”.

This would give the following use:
In a lot of applications you really want to know the exact sample at
which some setting was changed.

For example frequency hopping:
Change DDC frequency to 16.0 Mhz at timestamp 1000
Change DDC frequency to 17.0 Mhz at timestamp 1100

RX:
Then when receiving and the samples come in, I know sample 1000 to 1100
have RX DDC freq 16.0 MHz and sample 1100 and beyond have RX DDC freq
17.0 Mhz

TX: send samples 0000 to 1000 knowing it will be transmitted with 16.0
MHz,
send samples 1100 and beyond knowing it will be transmitted at 17
MHz.

Another usefull application is sample-accurate control of the GPIO
lines.
(set bit GPIO15 high at samplenumber 10000, set it low again at
samplenumber 10001)
This again could be used for the multi_usrp code to start synchronising
the samplenumber counters (timestamps)

ERROR OR STATUS CONTROL:
Maybe we also want some more status info (error-control) for operations.
A single bit in all replies could indicate if an error had occured.

Or maybe even more advanced:
ERROR: an error has occured
INVALID: an invalid value was send to this register. (Better do this in
software)
TIMEOUT: a timeout has occured
IO_ERROR: I2C or SPI communication failed

MASKED_WRITE:
What I also miss is a masked write to registers:
Write Register:

   Opcode:     OP_WRITE_REG_MASKED

  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |     Opcode    |       6       |    mbz    |     Reg Number    |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                        MASK                                   |
  |                        Register Value                         |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

A Masked write would save you from needing to do a read, modify, write
over the BUS when you only want to change a single bit.

TYPO:
I also found a little typo:

typo:
226 SPI Write:
227
228 Opcode: OP_SPI_WRITE
229 Enables: Which SPI enables to assert (mask)
230 Format: Specifies format of SPI data and Opt Header
Bytes
231 Opt Header Bytes: 2-byte field containing optional Tx bytes;
see Format
232 Data: The bytes to write to the I2C bus

must probably be:
232 Data: The bytes to write to the SPI bus

I hope these suggestions are of use.

Martin

On Mon, Feb 26, 2007 at 09:44:37PM +0100, Martin D. wrote:

And are samplenumbers, input-samples, output-samples or 64 Mhz clocks.
64 Mhz clocks would be the most accurate,

The master clock on the USRP. For the current models, 64MHz.

but could be awkward to work with when using fractional
interpolation or decimation rates. (I am working on these) When
using samplenumbers of input-samples or output-samples, you have the
problem that different channels could work at different rates. When
using seconds or milliseconds as timestanmps you have the problem of
reduced accuracy.

Agreed, that’s why I think the master clock is the time base.
It also gives an idea of time that is independent of how any of the
signal processing paths may be configured.

In the multi_usrp fpga code which I use to synchronise multiple
usrps I use a 32 bit samplenumember to align different channels from
different usrps. (The samplenumber counters are synchronised in
hardware using GPIO 15 on daughterboard RA)

In my pre-inbandsignaling code I am now using a whole channel which just sends out samplenumbers.
This way I can align channels from different usrps, sample-accurate.

Using in-band signaling I could get away with just a timestamp per
packet, but then it must have at least 1 sample granularity.

Yes. This shouldn’t be a problem if we use the master clock as the time
base.

TIMESTAMP IN CONTROL CHANNEL:
What I am missing is a desciption on how the timestamp in the control channel operations is used.

I was thinking that it specifies the time at which the command packet
sequencer starts working through the sub-packets.

I suggest it would have the following meaning:
On (a bundle of) write and I2C READ and WRITE and SPI READ and WRITE operations it would mean:

Start these operation at specified timestamp.
If the timestamp is 0xffffffff it is interpreted as “Now”.

Yes.

On READ_REPLY operations it would mean:
These value were read (all reads were finished) at given timestamp.

Agreed. On any reply packet, the timestamp is the time at which the
command
(or group of commands) finished.

If the timestamp is 0xffffffff it is interpreted as , “An error has occured, or timeout”.

I think I’d rather not overload timestamp for error signaling.

This would give the following use:
In a lot of applications you really want to know the exact sample at which some setting was changed.

For example frequency hopping:
Change DDC frequency to 16.0 Mhz at timestamp 1000
Change DDC frequency to 17.0 Mhz at timestamp 1100

Yes.

RX:

Then when receiving and the samples come in, I know sample 1000 to
1100 have RX DDC freq 16.0 MHz and sample 1100 and beyond have RX
DDC freq 17.0 Mhz

TX: send samples 0000 to 1000 knowing it will be transmitted with 16.0 MHz,
send samples 1100 and beyond knowing it will be transmitted at 17 MHz.

Another usefull application is sample-accurate control of the GPIO
lines. (set bit GPIO15 high at samplenumber 10000, set it low again
at samplenumber 10001) This again could be used for the multi_usrp
code to start synchronising the samplenumber counters (timestamps)

Yes.

ERROR OR STATUS CONTROL:
Maybe we also want some more status info (error-control) for operations.
A single bit in all replies could indicate if an error had occured.

Or maybe even more advanced:
ERROR: an error has occured
INVALID: an invalid value was send to this register. (Better do this in software)
TIMEOUT: a timeout has occured
IO_ERROR: I2C or SPI communication failed

OK. We could grab another bit from the header, or reuse one that only
applies to data packets. Perhaps we ought to define 4 cases for the
header flags:

DATA IN
DATA OUT
CONTROL IN
CONTROL OUT

  |                        Register Value                         |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

A Masked write would save you from needing to do a read, modify,
write over the BUS when you only want to change a single bit.

Good idea. I’ll add it.

232 Data: The bytes to write to the I2C bus

must probably be:
232 Data: The bytes to write to the SPI bus

Thanks!

I hope these suggestions are of use.

Definitely! Thanks for taking the time to read through it.

Martin

Eric

On Mon, Feb 26, 2007 at 04:39:32PM -0500, Brian P. wrote:

     |     Opcode    |       6       |    mbz    |     Reg Number    

A Masked write would save you from needing to do a read, modify,
write over the BUS when you only want to change a single bit.

Good idea. I’ll add it.

Can’t the host figure out what the new value should be using its
shadowed values? Under which circumstances would you want to use the
mask without the host already knowing the contents of the register if
it had already been written?

If there are multiple processes running on the host(s) that can send
commands, then shadows don’t work out well.

Also, although we’ve been talking about USB, I’m hoping to use what we
come up with in this conversation for the Gigabit ethernet interface
too. So far, the main difference I see is that ethernet packets are
variable length.

Another concern I had was with multiple read commands and the
timestamp. If a control packet is sent down with multiple read
commands and a timestamp of NOW - the first read is done ASAP and the
timestamp field filled with the current time. Is this the way it
should work, or should the timestamp represent the time the last piece
of data was read? Should there be some other type of message which
states a second timestamp at the end of the received control packet to
say how long it took for the operation to complete?

I’m for keeping it as simple as possible. Absent a compelling use case
for some other behavior, I suggest that we pick some intepretation
(either the beginning or the end of processing) for the timestamp on
the reply packet. I’m leaning towards “beginning”, which is similar
to the interpretation of the timestamp on IN packets.

One could also imagine a scenario where one of the registers that was
readable was mapped to the time base. Then your READ_REG_REPLY
would have that time in it :wink:

Eric

Thanks Dave,

Great ideas.

Eric

I’m considering the case whereby users might want to attach a USRPng
to a network, then either L2/L3 to multiple processing frontends.

Thanks,
Tim

On Mon, Feb 26, 2007 at 10:07:46PM -0500, David I. Emery wrote:

I’m thinking layer 2 with a unique Ethernet packet type (probably use
some abandoned packet type). That said, there’s nothing stopping us
from doing UDP, except the additional bandwidth. I don’t see any need
to do TCP.

I think we can take advantage of Gigabit Ethernet’s flow control
mechanism (pause frames).

Eric

On 2/26/07, Eric B. [email protected] wrote:

     |                        MASK                                   |
     |                        Register Value                         |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

A Masked write would save you from needing to do a read, modify,
write over the BUS when you only want to change a single bit.

Good idea. I’ll add it.

Can’t the host figure out what the new value should be using its
shadowed values? Under which circumstances would you want to use the
mask without the host already knowing the contents of the register if
it had already been written?

Another concern I had was with multiple read commands and the
timestamp. If a control packet is sent down with multiple read
commands and a timestamp of NOW - the first read is done ASAP and the
timestamp field filled with the current time. Is this the way it
should work, or should the timestamp represent the time the last piece
of data was read? Should there be some other type of message which
states a second timestamp at the end of the received control packet to
say how long it took for the operation to complete?

Brian

On 2/25/07, Eric B. [email protected] wrote:

That sounds good. Would you interpret it as a delta-t from the
timestamp in the header? I’m thinking about the case where there are
multiple DELAY ops in the payload.

It would probably be best to interpret it as a delta from the current
time tick where each operation would be 1 tick.

If you did it as a delta from the timestamp header, you would have to
calculate those intermediate deltas - and 32-bit subtractors inside an
FPGA can run pretty slowly even with the fast carry chains.

Something like:
OP_WRITE_REG
OP_DELAY[19]
OP_I2C_WRITE

I could see this showing OP_I2C_WRITE occuring 20 ticks after
OP_WRITE_REG.

What do you think?

Eric

Brian

On Sun, Feb 25, 2007 at 08:53:51PM -0500, Brian P. wrote:

FPGA can run pretty slowly even with the fast carry chains.

Something like:
OP_WRITE_REG
OP_DELAY[19]
OP_I2C_WRITE

I could see this showing OP_I2C_WRITE occuring 20 ticks after OP_WRITE_REG.

What do you think?

I guess I’m concerned that some of the OPS could take more than a
single tick. E.g., I2C or SPI ops. Thoses buses run at something
like 100kHz to 400kHz. If all execution delays are known, then
OP_DELAY seems fine (and easy).

Eric