Slow message queues... How to speed up?

Hello!

I’m trying to write a program that read data from the usrp, then
preforms a
fft and plots
the values of the maximum frequency and correspondent amplitude in real
time.

I’ve come across this problem:

  • At first I was using vector_sink but it gave me problems with memory
    management, always
    crashing after a few moments.
  • Now I’ve implemented a message queue system. With a queue for the
    frequency values and
    one for the amplitude.
  • I’ve noticed that it became much slower (about 10 fft’s per second, 2
    per
    message)
  • I need to have a much higher number to do what I want in the final
    version

Is there any way to speed up the process? (Have more values per message
or
more messages per second maybe…)

Following is my code:

class top(gr.top_block):
def init(self):
gr.top_block.init(self)

#Settings
rx_subdev_spec = (0, 0)
decim = 256
freq = None
gain = 10

self.u = usrp.source_c(decim_rate=decim)
self.u.set_mux(usrp.determine_rx_mux_value(self.u, rx_subdev_spec))

# determine the daughterboard subdevice we're using
self.subdev = usrp.selected_subdev(self.u, rx_subdev_spec)

# FFT Parameters
fftsize = 4096
mywin = window.blackmanharris(fftsize)
fft = gr.fft_vcc(fftsize, True, mywin)

# Get Maximum Value
c2m = gr.complex_to_mag(fftsize)
imax = gr.argmax_fs(fftsize)    # index
amax = gr.max_ff(fftsize)      # amplitude

s2f1 = gr.short_to_float()
s2f2 = gr.short_to_float()
p0 = gr.probe_signal_f()
p1 = gr.probe_signal_f()

# Vector Sink/Source
ss2v = gr.stream_to_vector(gr.sizeof_gr_complex, fftsize)
v2ss = gr.vector_to_stream(gr.sizeof_float, fftsize)

# Message Settings
self.qsize = 10

self.msgq0 = gr.msg_queue(self.qsize)  # queue frequency
self.msgq1 = gr.msg_queue(self.qsize)  # queue amplitude

# Message sink for frequency(0) and amplitude(1)
m_sink0 = gr.message_sink(gr.sizeof_float, self.msgq0, True)
m_sink1 = gr.message_sink(gr.sizeof_float, self.msgq1, True)

self.connect(self.u, ss2v, fft, c2m, (imax,0), s2f1, m_sink0)
self.connect((imax,1), s2f2, p1)

self.connect(c2m, amax, m_sink1)

    # set initial values

if gain is None:
# if no gain was specified, use the mid-point in dB
  g = self.subdev.gain_range()
  gain = float(g[0]+g[1])/2

  if freq is None:
    freq = 10.7e6

    self.set_gain(gain)

    if not(self.set_freq(freq)):
      pass
else:
  if freq is None:
    freq = 10.7e6

    self.set_gain(gain)

    if not(self.set_freq(freq)):
      pass

def set_freq(self, target_freq):

r = usrp.tune(self.u, 0, self.subdev, target_freq)

return False

def set_gain(self, gain):
self.subdev.set_gain(gain)

def shift(x,size=4096):
“”"
Shifts indexes so that the first half corresponds to FS-FS/2
and the second half to FS+FS/2.
@ x: index number
“”"
if x <= (size/2):
return x+(size/2)-1
elif x > (size/2):
return x-(size/2)-1

def fill_buffer(buffer, data, buffer_size=1000):
“”"
Fill a buffer in a circular form. When buffer_size is reached
the oldest values are discarded. The number of values discarded
depend on the size of new ones to add to the buffer.
@ buffer: vector to add the values
@ data: list of values to add
@ buffer_size: 1000 by default
@ out: final buffer with data values added
“”"

if len(buffer) < buffer_size:
out = concatenate((buffer,data))
elif len(buffer) >= buffer_size:
out = concatenate((buffer[len(data):],data))
return out

def main ():

Start Top Block

tb = top()

#Start Flowgraph
tb.start()

Set Buffers

yf=[]
ya=[]
fftsize=4096

Read first message

fmsg = tb.msgq0.delete_head() # get first frequency message
amsg = tb.msgq1.delete_head() # get first amplitude message

raw_f = fmsg.to_string() # raw frequency data
raw_a = amsg.to_string() # raw amplitude data

f_data = numpy.fromstring(raw_f, numpy.float32, count =
int(fmsg.arg2())) #
converted frequency data
a_data = numpy.fromstring(raw_a, numpy.float32, count =
int(amsg.arg2())) #
converted amplitude data

fill frequency buffer

yf = fill_buffer(yf,f_data)

fill frequency buffer

ya = fill_buffer(ya,a_data)

Set Initial Plot Values

x = [i for i in range(len(yf))]

a_data = 10.0scipy.log10(a_data)
yf = [shift(i,fftsize) for i in yf] # index shifting
yf = [10.7e6+i
250e3/fftsize-125e3 for i in yf] # convertion from
index to
frequency

pylab.ion()

(…)

pylab.draw()

while(True):

fmsg = tb.msgq0.delete_head()  # get first frequency message
amsg = tb.msgq1.delete_head()  # get first amplitude message

raw_f = fmsg.to_string()  # raw frequency data
raw_a = amsg.to_string()  # raw amplitude data

f_data = numpy.fromstring(raw_f, numpy.float32, count = 

int(fmsg.arg2()))

converted frequency data

a_data = numpy.fromstring(raw_a, numpy.float32, count = 

int(amsg.arg2()))

converted amplitude data

a_data = 10.0*scipy.log10(a_data)
f_data = [shift(i,fftsize) for i in f_data]      # index shifting
f_data = [10.7e6+i*250e3/fftsize-125e3 for i in f_data]  # 

convertion from
index to frequency

# fill frequency buffer
yf = fill_buffer(yf, f_data)

# fill frequency buffer
ya = fill_buffer(ya, a_data)

(....)

pylab.draw()

if name == ‘main’:
main ()


View this message in context:
http://www.nabble.com/Slow-message-queues…-How-to-speed-up–tp22583297p22583297.html
Sent from the GnuRadio mailing list archive at Nabble.com.

On Wed, Mar 18, 2009 at 09:44:05AM -0700, E. Ornelas wrote:

management, always
crashing after a few moments.

  • Now I’ve implemented a message queue system. With a queue for the
    frequency values and
    one for the amplitude.
  • I’ve noticed that it became much slower (about 10 fft’s per second, 2 per
    message)
  • I need to have a much higher number to do what I want in the final version

Is there any way to speed up the process? (Have more values per message or
more messages per second maybe…)

First off, depending on the machine you are running this on, you may
or may not have enough cycles to get the job done.

You want to do as little as possible in python. That is, be sure that
there’s no python on the critical path.

Consider writing a C++ block that scans the fft output vectors and
computes the value that you want. It sounds like you want something
that looks at each fft output vector, and for each vector returns the
index of the bin with the most power, and the magnitude of that bin.

Eric