Slow down rate of Python source block

addis_a · July 31, 2014, 6:23pm

Dear All,

I have a Python block that produces packets of size 1536 bytes. Due to
various reasons, the latter parts of my flow graph are very slow (this
is desired and cannot be changed). After producing 510 packets, I get
the following error.

“handler caught exception: operands could not be broadcast together with
shapes (1527) (1536)
Traceback (most recent call last):
File “/usr/local/lib/python2.7/dist-packages/gnuradio/gr/gateway.py”,
line 55, in eval
try: self._callback()
File “/usr/local/lib/python2.7/dist-packages/gnuradio/gr/gateway.py”,
line 160, in __gr_block_handle
) for i in self.__out_indexes],
File “/usr/local/lib/python2.7/dist-packages/trl/blsd_enc_b.py”, line
198, in work
out_cA[0:len(cAm)] = cAm
ValueError: operands could not be broadcast together with shapes (1527)
(1536)
thread[thread-per-block[1]: <block blsd_enc_b (2)>]: caught unrecognized
exception”

Debugging more carefully, I can see that:

len(cAm) = 1536 , len(out_cA) = 32768

for the first 490 packets, and then the length of ‘out_cA’ begins to
reduce

packet_num = 490
len(cAm) = 1536 , len(out_cA) = 32247
packet_num = 491
len(cAm) = 1536 , len(out_cA) = 30711
packet_num = 492
len(cAm) = 1536 , len(out_cA) = 29175
packet_num = 493
len(cAm) = 1536 , len(out_cA) = 27639
packet_num = 494
len(cAm) = 1536 , len(out_cA) = 26103
packet_num = 495
len(cAm) = 1536 , len(out_cA) = 24567
packet_num = 496
len(cAm) = 1536 , len(out_cA) = 23031
packet_num = 497
len(cAm) = 1536 , len(out_cA) = 21495
packet_num = 498
len(cAm) = 1536 , len(out_cA) = 19959
packet_num = 499
len(cAm) = 1536 , len(out_cA) = 18423
packet_num = 500
len(cAm) = 1536 , len(out_cA) = 16887
packet_num = 501
len(cAm) = 1536 , len(out_cA) = 15351
packet_num = 502
len(cAm) = 1536 , len(out_cA) = 13815
packet_num = 503
len(cAm) = 1536 , len(out_cA) = 12279
packet_num = 504
len(cAm) = 1536 , len(out_cA) = 10743
packet_num = 505
len(cAm) = 1536 , len(out_cA) = 9207
packet_num = 506
len(cAm) = 1536 , len(out_cA) = 7671
packet_num = 507
len(cAm) = 1536 , len(out_cA) = 6135
packet_num = 508
len(cAm) = 1536 , len(out_cA) = 4599
packet_num = 509
len(cAm) = 1536 , len(out_cA) = 3063
packet_num = 510
len(cAm) = 1536 , len(out_cA) = 1527

I believe this is because the latter parts of the flow graph are
blocking, their buffers become full, and this backs up to my source
block. This is confirmed by connecting my source block to null sinks,
which then allows the source block to run infinitely (well, a long
while!).

Is there a way I can slow up this block? I tried using a throttle on the
output, but this doesn’t help (in fact it seems to make it worse!).

The full code of the block is attached.

Thanks!

David

NOTE: The information in this email and any attachments may be
confidential and/or legally privileged. This message may be read, copied
and used only by the intended recipient. If you are not the intended
recipient, please destroy this message, delete any copies held on your
system and notify the sender immediately.

Toshiba Research Europe Limited, registered in England and Wales
(2519556). Registered Office 208 Cambridge Science Park, Milton Road,
Cambridge CB4 0GZ, England. Web: www.toshiba.eu/research/trl

divotdave · July 31, 2014, 8:13pm

On Thu, Jul 31, 2014 at 12:21 PM, David Halls
[email protected]
wrote:

File “/usr/local/lib/python2.7/dist-packages/gnuradio/gr/gateway.py”, line
thread[thread-per-block[1]: <block blsd_enc_b (2)>]: caught unrecognized
exception"

Debugging more carefully, I can see that:

len(cAm) = 1536 , len(out_cA) = 32768

Just a quick response without really studying the problem or your code.
The
dynamic scheduler in GR is getting in your way, and the throttle block
is
definitely not the right way to help you. You need to either tell the
scheduler what you need it to send your block or handle it internally.
There are three ways to solve these issues:

Use set_output_multiple in the constructor, which will only allow the
scheduler to send you chunks of data of some multiple of the number you
pass that. I’ve seen this slow down the scheduler in other situations,
but
it sounds like you’re going slow, anyways, so this shouldn’t cause a
problem.
Make your input signature your packet_length so each item will be a
vector of that length. This would not be my preferred way, but we’ve
played
that game before.
Handle it internally. Buffer up the input until you have enough to
produce what you need. You’d need to inherit from gr.basic_block here
and
do more management of the data and buffers yourself.

Tom

divotdave · August 1, 2014, 11:25am

From: [email protected] [[email protected]] on behalf of Tom
Rondeau [[email protected]]
Sent: 31 July 2014 19:11
To: David Halls
Cc: [email protected]
Subject: Re: [Discuss-gnuradio] Slow down rate of Python source block

On Thu, Jul 31, 2014 at 12:21 PM, David Halls
<[email protected]mailto:[email protected]>
wrote:
Dear All,

I have a Python block that produces packets of size 1536 bytes. Due to
various reasons, the latter parts of my flow graph are very slow (this
is desired and cannot be changed). After producing 510 packets, I get
the following error.

“handler caught exception: operands could not be broadcast together with
shapes (1527) (1536)
Traceback (most recent call last):
File “/usr/local/lib/python2.7/dist-packages/gnuradio/gr/gateway.py”,
line 55, in eval
try: self._callback()
File “/usr/local/lib/python2.7/dist-packages/gnuradio/gr/gateway.py”,
line 160, in __gr_block_handle
) for i in self.__out_indexes],
File “/usr/local/lib/python2.7/dist-packages/trl/blsd_enc_b.py”, line
198, in work
out_cA[0:len(cAm)] = cAm
ValueError: operands could not be broadcast together with shapes (1527)
(1536)
thread[thread-per-block[1]: <block blsd_enc_b (2)>]: caught unrecognized
exception”

Debugging more carefully, I can see that:

len(cAm) = 1536 , len(out_cA) = 32768

Just a quick response without really studying the problem or your code.
The dynamic scheduler in GR is getting in your way, and the throttle
block is definitely not the right way to help you. You need to either
tell the scheduler what you need it to send your block or handle it
internally. There are three ways to solve these issues:

Use set_output_multiple in the constructor, which will only allow the
scheduler to send you chunks of data of some multiple of the number you
pass that. I’ve seen this slow down the scheduler in other situations,
but it sounds like you’re going slow, anyways, so this shouldn’t cause a
problem.
Make your input signature your packet_length so each item will be a
vector of that length. This would not be my preferred way, but we’ve
played that game before.
Handle it internally. Buffer up the input until you have enough to
produce what you need. You’d need to inherit from gr.basic_block here
and do more management of the data and buffers yourself.

Tom

Tom,

Thanks for your reply. Unfortunately as it is a Python block, I cannot
use ‘set_output_multiple’. For 2. do you mean set the input signature of
the blocks fed from the source block. This could be possible but quite a
few different blocks (some standard GR blocks) are fed from it. Could
you provide some more details, or a link to similar implementation for
option 3. - again is this possible from within a Python block?

Regards,

David

for the first 490 packets, and then the length of ‘out_cA’ begins to
reduce

packet_num = 490
len(cAm) = 1536 , len(out_cA) = 32247
packet_num = 491
len(cAm) = 1536 , len(out_cA) = 30711
packet_num = 492
len(cAm) = 1536 , len(out_cA) = 29175
packet_num = 493
len(cAm) = 1536 , len(out_cA) = 27639
packet_num = 494
len(cAm) = 1536 , len(out_cA) = 26103
packet_num = 495
len(cAm) = 1536 , len(out_cA) = 24567
packet_num = 496
len(cAm) = 1536 , len(out_cA) = 23031
packet_num = 497
len(cAm) = 1536 , len(out_cA) = 21495
packet_num = 498
len(cAm) = 1536 , len(out_cA) = 19959
packet_num = 499
len(cAm) = 1536 , len(out_cA) = 18423
packet_num = 500
len(cAm) = 1536 , len(out_cA) = 16887
packet_num = 501
len(cAm) = 1536 , len(out_cA) = 15351
packet_num = 502
len(cAm) = 1536 , len(out_cA) = 13815
packet_num = 503
len(cAm) = 1536 , len(out_cA) = 12279
packet_num = 504
len(cAm) = 1536 , len(out_cA) = 10743
packet_num = 505
len(cAm) = 1536 , len(out_cA) = 9207
packet_num = 506
len(cAm) = 1536 , len(out_cA) = 7671
packet_num = 507
len(cAm) = 1536 , len(out_cA) = 6135
packet_num = 508
len(cAm) = 1536 , len(out_cA) = 4599
packet_num = 509
len(cAm) = 1536 , len(out_cA) = 3063
packet_num = 510
len(cAm) = 1536 , len(out_cA) = 1527

I believe this is because the latter parts of the flow graph are
blocking, their buffers become full, and this backs up to my source
block. This is confirmed by connecting my source block to null sinks,
which then allows the source block to run infinitely (well, a long
while!).

Is there a way I can slow up this block? I tried using a throttle on the
output, but this doesn’t help (in fact it seems to make it worse!).

The full code of the block is attached.

Thanks!

David

NOTE: The information in this email and any attachments may be
confidential and/or legally privileged. This message may be read, copied
and used only by the intended recipient. If you are not the intended
recipient, please destroy this message, delete any copies held on your
system and notify the sender immediately.

Toshiba Research Europe Limited, registered in England and Wales
(2519556). Registered Office 208 Cambridge Science Park, Milton Road,
Cambridge CB4 0GZ, England. Web:
www.toshiba.eu/research/trlhttp://www.toshiba.eu/research/trl

This email has been scanned for email related threats and delivered
safely by Mimecast.
For more information please visit http://www.mimecast.com

Discuss-gnuradio mailing list
[email protected]mailto:[email protected]
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

NOTE: The information in this email and any attachments may be
confidential and/or legally privileged. This message may be read, copied
and used only by the intended recipient. If you are not the intended
recipient, please destroy this message, delete any copies held on your
system and notify the sender immediately.

Toshiba Research Europe Limited, registered in England and Wales
(2519556). Registered Office 208 Cambridge Science Park, Milton Road,
Cambridge CB4 0GZ, England. Web: www.toshiba.eu/research/trl

divotdave · August 14, 2014, 8:50pm

From: [email protected] [mailto:[email protected]] On Behalf Of
Tom R.
Sent: 01 August 2014 14:31
To: David Halls
Cc: [email protected]
Subject: Re: [Discuss-gnuradio] Slow down rate of Python source block

On Fri, Aug 1, 2014 at 5:24 AM, David Halls
<[email protected]mailto:[email protected]>
wrote:

From: [email protected]mailto:[email protected]
[[email protected]mailto:[email protected]] on behalf of Tom
Rondeau [[email protected]mailto:[email protected]]
Sent: 31 July 2014 19:11
To: David Halls
Cc: [email protected]mailto:[email protected]
Subject: Re: [Discuss-gnuradio] Slow down rate of Python source block

On Thu, Jul 31, 2014 at 12:21 PM, David Halls
<[email protected]mailto:[email protected]<mailto:[email protected]mailto:[email protected]>>
wrote:
Dear All,

I have a Python block that produces packets of size 1536 bytes. Due to
various reasons, the latter parts of my flow graph are very slow (this
is desired and cannot be changed). After producing 510 packets, I get
the following error.

“handler caught exception: operands could not be broadcast together with
shapes (1527) (1536)
Traceback (most recent call last):
File “/usr/local/lib/python2.7/dist-packages/gnuradio/gr/gateway.py”,
line 55, in eval
try: self._callback()
File “/usr/local/lib/python2.7/dist-packages/gnuradio/gr/gateway.py”,
line 160, in __gr_block_handle
) for i in self.__out_indexes],
File “/usr/local/lib/python2.7/dist-packages/trl/blsd_enc_b.py”, line
198, in work
out_cA[0:len(cAm)] = cAm
ValueError: operands could not be broadcast together with shapes (1527)
(1536)
thread[thread-per-block[1]: <block blsd_enc_b (2)>]: caught unrecognized
exception”

Debugging more carefully, I can see that:

len(cAm) = 1536 , len(out_cA) = 32768

Just a quick response without really studying the problem or your code.
The dynamic scheduler in GR is getting in your way, and the throttle
block is definitely not the right way to help you. You need to either
tell the scheduler what you need it to send your block or handle it
internally. There are three ways to solve these issues:

Use set_output_multiple in the constructor, which will only allow the
scheduler to send you chunks of data of some multiple of the number you
pass that. I’ve seen this slow down the scheduler in other situations,
but it sounds like you’re going slow, anyways, so this shouldn’t cause a
problem.
Make your input signature your packet_length so each item will be a
vector of that length. This would not be my preferred way, but we’ve
played that game before.
Handle it internally. Buffer up the input until you have enough to
produce what you need. You’d need to inherit from gr.basic_block here
and do more management of the data and buffers yourself.

Tom

Tom,

Thanks for your reply. Unfortunately as it is a Python block, I cannot
use ‘set_output_multiple’. For 2. do you mean set the input signature of
the blocks fed from the source block. This could be possible but quite a
few different blocks (some standard GR blocks) are fed from it. Could
you provide some more details, or a link to similar implementation for
option 3. - again is this possible from within a Python block?

Regards,

David

David,

set_output_multiple is exposed through the Python gateway code, so you
should be able to call self.set_output_multiple inside the constructor…

For 2, yes, I mean that you can adjust the io signature of the block,
essentially taking in a vector so that 1 item is now a full packet. But
yes, this makes it more difficult to integrate with other blocks; one
reason I said this isn’t a preferred method.

And for 3, the only think that I can think of is the qtgui sinks, like
the freq sink that passes a chunk of data the length of the FFT size and
buffers it internally. The work functions for these types of blocks tend
to become more complex, though, and difficult to read. But you can
certainly do it inside a Python block; you just may need to do hard
copies out of the input buffers to make sure the data is actually moved
and stored locally.

Tom

Hi Tom,

I tried self.set_output_multiple and this doesn’t work in Python, I
believe for the same reasons as

"The “set_min_noutput_items()” is not yet a supported python statement
[1] as per March 06, 2014.
Also, I believe this statement should be placed in the member functions
(constructor, work function, callback function etc) of your custom
block, don’t put it at the wrong place.
In alternative, you may want to replace it with “set_output_multiple()”
if this is more appropriate.

http://gnuradio.4.n7.nabble.com/self-set-min-noutput-items-is-not-a-valid-python-command-in-gnuradio-td46731.html"

So I can’t do Option 1), for now, Option 3) sounds very tricky, so I’ve
tried number 2)

Fortunately my source block feeds only to ‘unpacked_to_packed’ blocks, a
few of them, so I’ve created my own. I adjusted the io signature of the
input to be
…::make(1,1, packet_len*sizeof(unsigned char))

This gives Python runtime errors “itemsize mismatch: bsld_enc_b0:0 using
1, unpacked_to_packed1:0 using 1536”.

I then try to set the io signature of my source block but if I try:

out_sig=[packet_len*numpy.uint8]

I get

TypeError: unsupported operand type(s) for *: ‘int’ and ‘type’

and if I try

out_sig=[packet_len]

TypeError: data type not understood

Can you help me further?

Many thanks,

David

NOTE: The information in this email and any attachments may be
confidential and/or legally privileged. This message may be read, copied
and used only by the intended recipient. If you are not the intended
recipient, please destroy this message, delete any copies held on your
system and notify the sender immediately.

Toshiba Research Europe Limited, registered in England and Wales
(2519556). Registered Office 208 Cambridge Science Park, Milton Road,
Cambridge CB4 0GZ, England. Web: www.toshiba.eu/research/trl

divotdave · August 1, 2014, 3:32pm

On Fri, Aug 1, 2014 at 5:24 AM, David Halls
[email protected]
wrote:

On Thu, Jul 31, 2014 at 12:21 PM, David Halls <
Traceback (most recent call last):
(1536)
scheduler what you need it to send your block or handle it internally.
that game before.
Tom,
David

David,

set_output_multiple is exposed through the Python gateway code, so you
should be able to call self.set_output_multiple inside the constructor…

For 2, yes, I mean that you can adjust the io signature of the block,
essentially taking in a vector so that 1 item is now a full packet. But
yes, this makes it more difficult to integrate with other blocks; one
reason I said this isn’t a preferred method.

And for 3, the only think that I can think of is the qtgui sinks, like
the
freq sink that passes a chunk of data the length of the FFT size and
buffers it internally. The work functions for these types of blocks tend
to
become more complex, though, and difficult to read. But you can
certainly
do it inside a Python block; you just may need to do hard copies out of
the
input buffers to make sure the data is actually moved and stored
locally.

Tom