Hdtv hacking

valdar · April 7, 2006, 9:30pm

Well, (this is a boy trying a grown-ups job I managed to get
two new modules to compile in gr-atsc, atsc_interleaver and
atsc_deinterleaver, just by copying atsc_rs_encoder and atsc_rd_decoder
(and accounting for atsci_reed_solomon having one class
and two methods(?) encode and decode, whereas atsci_data_interleaver
has two classes with one method each - IIUC).

So, basically all I did was change the names and put
in for atsc_interleaver.cc:

for (int i = 0; i < noutput_items; i++){
d_interleaver.interleave (out[i], in[i]);

and in atsc_deinterleaver.cc

for (int i = 0; i < noutput_items; i++){
d_deinterleaver.deinterleave (out[i], in[i]);

Anyway, after putting in another python test_loopback_002(self):
with the two stages interleaver/deinterleaver it will fail
make check , printing out a bunch of numbers.

–Chuck
learning a lot

valdar · April 7, 2006, 11:07pm

On Fri, 2006-04-07 at 15:28 -0400, Charles S. wrote:

Well, (this is a boy trying a grown-ups job I managed to get
two new modules to compile in gr-atsc, atsc_interleaver and
atsc_deinterleaver, just by copying atsc_rs_encoder and atsc_rd_decoder

Anyway, after putting in another python test_loopback_002(self):
with the two stages interleaver/deinterleaver it will fail
make check , printing out a bunch of numbers.

Ah, I need a clue how to do this:

When you loop the deinterleaver back, you’ll need to account for the
52 segment delay through it. Using gr.skiphead is probably the
easiest solution (thanks Martin!).

As this sure didn’t help:

    src = vector_source_ts(self.fg, src_data)
    rand = atsc.randomizer()
    rs_enc = atsc.rs_encoder()
    inter = atsc.interleaver()

=?=> skip = gr.skiphead(atsc.sizeof_atsc_mpeg_packet_rs_encoded,52)
deinter = atsc.deinterleaver()
rs_dec = atsc.rs_decoder()
derand = atsc.derandomizer()
dst = vector_sink_ts(self.fg)
self.fg.connect(src, rand, rs_enc, inter, skip, deinter, rs_dec,
derand, dst)

apologies if the above is laughably inept

–Chuck

valdar · April 7, 2006, 11:41pm

On Fri, Apr 07, 2006 at 03:28:17PM -0400, Charles S. wrote:

for (int i = 0; i < noutput_items; i++){
d_interleaver.interleave (out[i], in[i]);

and in atsc_deinterleaver.cc

for (int i = 0; i < noutput_items; i++){
d_deinterleaver.deinterleave (out[i], in[i]);

Looks good.

Anyway, after putting in another python test_loopback_002(self):
with the two stages interleaver/deinterleaver it will fail
make check , printing out a bunch of numbers.

OK. The most likely reason it doesn’t pass, is that there is a 52
segment delay between what you write into the interleaver, and what you
get out of the deinterleaver. To work around the behavior, drop the
first 52 segments of the output and then compare the remainder with
the input.

That is:

Drop first 52 segments of output
Ignore last 52 segments of input
The resulting pieces should be the same length and value.

Raw input and output:

 Input:  A----------------------------------------ZXXXXXX
 Output: 000000A----------------------------------------Z

Where XXXXXX and 0000000 represent 52 segments of data.

After adjustment:

 Input:  A----------------------------------------Z
 Output: A----------------------------------------Z

–Chuck
learning a lot

I did too!
Eric

valdar · April 7, 2006, 11:44pm

On Fri, Apr 07, 2006 at 05:05:24PM -0400, Charles S. wrote:

Ah, I need a clue how to do this:
rs_enc = atsc.rs_encoder()
apologies if the above is laughably inept
I think you’re close.

derand, dst)
Then you’ll still need to drop the last 52 segments of the source
before comparing.

Eric

valdar · April 8, 2006, 1:12am

On Apr 7, 2006, at 6:49 PM, Charles S. wrote:

work - cool! (putting skip AFTER inter, deinter). I’ll finish the
test code and move on to Trellis Encoder / Viterbi Decoder.

I have a trellis encoder and the V decoder is in debugging (by me) -
hope to have version 1 finished by the end of this weekend. These
are generic encoder / decoder blocks, taking arbitrary # of input
streams and element sizes and reproducing that on the output.
Throughput of the encoder using a (2 output, 2 input, 2 memory /
input) code (meaning 16 states) is around 10 Mb/s without optimizing
the programming. As the decoder isn’t operational, I don’t know how
fast it will be … but it’s mush more complex than the encoder (no
surprise there) and thus will definitely be slower than the encoder.

I don’t know if these would be of interest for use by ATSC, but I’m
happy to provide them once I know they work. Would also make for
good beta testing I will provide them to the overall GR project
once I know they work and are reasonable optimized. - MLD

valdar · April 8, 2006, 3:22am

On Fri, Apr 07, 2006 at 06:49:07PM -0400, Charles S. wrote:

work - cool! (putting skip AFTER inter, deinter). I’ll finish the
test code and move on to Trellis Encoder / Viterbi Decoder.

That’s great.
Did you figure out how to compare them without writing to files?

The latest benchmark I have for the 0.9 atsc is 10 minutes of data
takes about 14 hours to decode on an Athlon 3200+ (2200 MHz real),
and over HALF that time is spent resampling and up-mixing the data
to emulate an mc4020 to atsc_rx.

Did you do the 5:2 resampling using the rational_resampler, or did you
interpolate then decimate in two steps?

Great work!

Eric

valdar · April 8, 2006, 12:51am

The most likely reason it doesn’t pass, is that there is a 52
segment delay between what you write into the interleaver, and what you
get out of the deinterleaver. To work around the behavior, drop the
first 52 segments of the output and then compare the remainder with
the input.

Well - writing expected_results and result_data out to files
and manually comparing show they do match, so interleaver/deinterleaver
work - cool! (putting skip AFTER inter, deinter). I’ll finish the
test code and move on to Trellis Encoder / Viterbi Decoder.

The latest benchmark I have for the 0.9 atsc is 10 minutes of data
takes about 14 hours to decode on an Athlon 3200+ (2200 MHz real),
and over HALF that time is spent resampling and up-mixing the data
to emulate an mc4020 to atsc_rx.

valdar · April 8, 2006, 10:05am

On Fri, Apr 07, 2006 at 07:11:21PM -0400, Michael D. wrote:

I don’t know if these would be of interest for use by ATSC, but I’m
happy to provide them once I know they work. Would also make for
good beta testing I will provide them to the overall GR project
once I know they work and are reasonable optimized. - MLD

Maximum likelyhood decoding with the Viterbi algorithm is needed
almost everywhere - a generic one for gnuradio would be great.

Jens

valdar · April 8, 2006, 4:11am

On Fri, Apr 07, 2006 at 07:11:21PM -0400, Michael D. wrote:

the programming. As the decoder isn’t operational, I don’t know how
fast it will be … but it’s mush more complex than the encoder (no
surprise there) and thus will definitely be slower than the encoder.

I don’t know if these would be of interest for use by ATSC, but I’m
happy to provide them once I know they work. Would also make for
good beta testing I will provide them to the overall GR project
once I know they work and are reasonable optimized. - MLD

Thanks!

Eric

valdar · April 8, 2006, 4:15pm

On Fri, 2006-04-07 at 18:20 -0700, Eric B. wrote:

On Fri, Apr 07, 2006 at 06:49:07PM -0400, Charles S. wrote:

The latest benchmark I have for the 0.9 atsc is 10 minutes of data
takes about 14 hours to decode on an Athlon 3200+ (2200 MHz real),
and over HALF that time is spent resampling and up-mixing the data
to emulate an mc4020 to atsc_rx.

Did you do the 5:2 resampling using the rational_resampler, or did you
interpolate then decimate in two steps?

currently it’s using the rational_resampler on one machine, then pipes
and sockets to atsc_rx on another to parallelize things a bit. Has been
running all night Resampler on an Athlon 3200+ at 99% cpu and
atsc_rx on a 1.4Ghz Pentium-M running about 70%.

valdar · April 9, 2006, 1:28pm

Michael D. wrote:

On Apr 7, 2006, at 6:49 PM, Charles S. wrote:

work - cool! (putting skip AFTER inter, deinter). I’ll finish the
test code and move on to Trellis Encoder / Viterbi Decoder.

There is a very fast (SSE,SSE2) viterbi decoder by Phil Karns.
(He also wrote a reed-solomon decoder)
They are available here:
http://www.ka9q.net/code/fec/

Greetings,
Martin

valdar · April 9, 2006, 8:17pm

On Sun, Apr 09, 2006 at 01:26:50PM +0200, Martin D. wrote:

Michael D. wrote:

On Apr 7, 2006, at 6:49 PM, Charles S. wrote:

work - cool! (putting skip AFTER inter, deinter). I’ll finish the
test code and move on to Trellis Encoder / Viterbi Decoder.

There is a very fast (SSE,SSE2) viterbi decoder by Phil Karns.
(He also wrote a reed-solomon decoder)
They are available here:
Forward Error Correcting Codes

FYI, we’re using an early version of his RS code.

Eric

valdar · April 10, 2006, 7:50pm

Phil’s work is quite famous, and his programs are of very good quality
for C programs. Not the best commented, but they do work for the
platforms for which they are designed (generic or SSE/2 optimized).
Fortunately or not, I use a PPC PowerMac, and hence the SSE/2 stuff is
of no use to me. Maybe I’ll get an IntelMac once the hardware has
stabilized and I have the resources

My encoder and decoder blocks will augment his programs somewhat in
various ways including adding many useful comments, user options, and
computational advances in decoding, all without compromising
throughput. - MLD

valdar · April 10, 2006, 8:09pm

On Fri, 2006-04-07 at 18:20 -0700, Eric B. wrote:

On Fri, Apr 07, 2006 at 06:49:07PM -0400, Charles S. wrote:

The most likely reason it doesn’t pass, is that there is a 52
segment delay between what you write into the interleaver, and what you
get out of the deinterleaver. To work around the behavior, drop the
first 52 segments of the output and then compare the remainder with
the input.

That’s great.
Did you figure out how to compare them without writing to files?

Yes, using gr.skiphead(atsc.sizeof_atsc_mpeg_packet_rs_encoded,52)
right after inter/deinterleaving, and also trim 52 * 188 off the
end of expected_result with

expected_result = src_data[0:len(src_data)-(52*188)]

I’m a little cornfused with the data sizes, in light of this:

“An MPEG transport stream packet is 188 bytes long. Internally we use a
packet that is 256 bytes long to help with buffer alignment. This
function adds the appropriate trailing padding to convert each packet
from 188 to 256 bytes.”

and these:

atsc.sizeof_atsc_data_segment
1024

atsc.sizeof_atsc_data_segment_pad
188

atsc.sizeof_atsc_mpeg_packet
256

atsc.sizeof_atsc_mpeg_packet_no_sync
256

atsc.sizeof_atsc_mpeg_packet_no_sync_pad
65

atsc.sizeof_atsc_mpeg_packet_pad
68

atsc.sizeof_atsc_mpeg_packet_rs_encoded
256

atsc.sizeof_atsc_mpeg_packet_rs_encoded_pad
45

atsc.sizeof_atsc_soft_data_segment
4096

I would hav expected “atsc_mpeg_packet” to be 188 and
“atsc_mpeg_packet_pad” to be 256 ??

Anyway, seems we should use one of those instead of 188 in
the expected_result trim statement above.

Other than that, make check runs and passes, ready to check
into cvs.

–Chuck