# Need some advice : scientific scripting language project

**URL:** <https://www.ruby-forum.com/t/need-some-advice-scientific-scripting-language-project/188843>\
**Category:** wxRuby\
**Created:** [May 19, 2010, 8:47am UTC](https://www.ruby-forum.com/t/need-some-advice-scientific-scripting-language-project/188843 "2010-05-19T08:47:36Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![emmanuel](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/3e96dc/32.png) [@emmanuel](https://www.ruby-forum.com/u/emmanuel)\
**Post date:** [May 19, 2010, 8:47am UTC](https://www.ruby-forum.com/t/need-some-advice-scientific-scripting-language-project/188843/1 "2010-05-19T08:47:36Z")

</div>

Hello,

Ok, I am writing a (pretty big) project trying to use ruby as the main  
scripting language for performing math computation, scientific display  
and display interaction.  
Here is roughly what I want

- a terminal window with a ruby interpreter in it (I want to be able to  
control all the completions/editions capabilities)
- a bench of ruby classes to manipulate signals/images,etc…
- I want everything to be really really fast (I’ll be manipulating huge  
signals/images)
- simple commands to display instances of these classes
- graphic (programable) interaction on these displays (e.g. moving the  
mouse on a such a window should display image values, clicking should  
zoom, etc…

Now my questions are the following

- Should I use irb ? Can I wrap it easily in a wxWidget window ? Can I  
reprogram some new capabilities (e.g., history completion, multi-line  
history commands…)

- A unix terminal window displays output text really really fast… what  
object should I use in wxRuby to display text as fast (wxTextCtrl is  
pretty slow, even if I use a buffer)

- I want to be able to interrupt long computations (clicking on a stop  
button or typing ctrl-c on the terminal window) ==\> I should be able to  
send SIGINT signal to the interpreter. Should I use 2 processes : one  
for the terminal and one for the rest ?  
The problem if I do that is that both processes will have a  
wxMainLoop… now what will be the best way for them to communicate ?

Thank you very much

Emmanuel

---

<div class="post-metadata">

**Author:** ![emmanuel](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/3e96dc/32.png) [@emmanuel](https://www.ruby-forum.com/u/emmanuel)\
**Post date:** [May 19, 2010, 1:13pm UTC](https://www.ruby-forum.com/t/need-some-advice-scientific-scripting-language-project/188843/2 "2010-05-19T13:13:12Z")

</div>

Hi Emmanuel

On 19/05/2010 07:47, Emmanuel E. wrote:

> Ok, I am writing a (pretty big) project trying to use ruby as the main  
> scripting language for performing math computation, scientific display  
> and display interaction.

Sounds like a big and interesting project. I originally got interested  
in wxRuby building research tools (for sociology) so I’d certainly be  
interested to hear your progress.

> Here is roughly what I want
> 
> - a terminal window with a ruby interpreter in it (I want to be able to  
> control all the completions/editions capabilities)

readline?

> - a bench of ruby classes to manipulate signals/images,etc…
> - I want everything to be really really fast (I’ll be manipulating huge  
> signals/images)

ruby/gsl? r-ruby?

> - simple commands to display instances of these classes
> - graphic (programable) interaction on these displays (e.g. moving the  
> mouse on a such a window should display image values, clicking should  
> zoom, etc…

wxRuby is well suited to this sort of application.

> Now my questions are the following
> 
> - Should I use irb ? Can I wrap it easily in a wxWidget window ? Can I  
> reprogram some new capabilities (e.g., history completion, multi-line  
> history commands…)

I don’t know of any examples of wrapping IRB in wxRuby, although it does  
provide a library interface. You might look into Wx::StyledTextCtrl  
which offers syntax highlighting and other more sophisticated support  
for programming code text.

> - A unix terminal window displays output text really really fast.. what  
> object should I use in wxRuby to display text as fast (wxTextCtrl is  
> pretty slow, even if I use a buffer)

The text display classes are Wx::TextCtrl, Wx::LogTextCtrl,  
Wx::StyledTextCtrl, Wx::RichTextCtrl and Wx::HtmlControl. I’d expect  
simple TextCtrl or StyledTextCtrl to be fastest, but I’ve never tested.

Judicious use of #freeze and #thaw might help to manage updates on  
screen. Generally Ruby is fairly slow, and wxRuby’s bridge to C++ adds  
additional overhead to each Wx method call, so you might improve speed  
by batching calls to the Wx API to update the TextCtrl’s context.

> - I want to be able to interrupt long computations (clicking on a stop  
> button or typing ctrl-c on the terminal window) ==\> I should be able to  
> send SIGINT signal to the interpreter. Should I use 2 processes : one  
> for the terminal and one for the rest ?  
> The problem if I do that is that both processes will have a  
> wxMainLoop… now what will be the best way for them to communicate ?

You have a couple of options:

- if the computational task can be broken down into small discrete  
elements, you can execute these in an evt\_idle handler, which runs when  
the GUI isn’t doing anything else. This is the easiest route because you  
can allow interruptions using normal wxRuby event handling techniques.

- you could use Ruby threads and run the computation in a slave thread.  
This is somewhat more complex because the slave thread should not  
directly update the GUI, but should signal its state by posting Wx  
events to the main GUI thread. A class that does this sort of thing is  
here (afraid it’s embedded in a fairly complex Wx app):

[http://weft-qda.rubyforge.org/svn/trunk/weft-qda/lib/weft/wxgui/gui\_subscriber.rb](http://weft-qda.rubyforge.org/svn/trunk/weft-qda/lib/weft/wxgui/gui_subscriber.rb)

If you use wxRuby and threads in Ruby 1.8 you need to explicitly  
allocate time to the slave thread - there are numerous discussions of  
this in the list archives.

- you could fork / pipe and run the computation as a separate process.  
There is no need for this to be a GUI process, I presume, so you don’t  
need to worry about a separate GUI main\_loop. One good thing about this  
route is that it will take advantage of multiple cores if they’re  
available, and will separate the GUI and analytic code. But it’ll  
involve working out how to pass data to the subsidiary process (eg  
popen3?) and manage and signal to it (SIGINT etc?). Given Ruby’s unix  
background, managing a child process is easier if you only need the  
program to work on \*nix platforms, not Windows.

hth  
alex

---

<div class="post-metadata">

**Author:** ![emmanuel](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/3e96dc/32.png) [@emmanuel](https://www.ruby-forum.com/u/emmanuel)\
**Post date:** [May 19, 2010, 3:10pm UTC](https://www.ruby-forum.com/t/need-some-advice-scientific-scripting-language-project/188843/3 "2010-05-19T15:10:14Z")

</div>

First of all, Thank you Alex for all your interesting comments !

> Alex F. wrote:  
> Hi Emmanuel
> 
> Sounds like a big and interesting project. I originally got interested  
> in wxRuby building research tools (for sociology) so I’d certainly be  
> interested to hear your progress.

I will let you know. Actually I already wrote a script language (called  
LastWave, [http://www.cmap.polytechnique.fr/~bacry/LastWave/index.html](http://www.cmap.polytechnique.fr/~bacry/LastWave/index.html))  
that has been quite used by a lot of persons… I started it a long time  
ago, so I did not use the modern script language (python or ruby)…  
it is version 3.0 now and using wxWidget… but it clearly needs to be  
rewritten using a good scripting language… it needs to be entirely  
redesigned…  
However I learned a lot, and I am pretty clear about some of the issues  
(memory, speed, some graphics issue).

I know there exist some language such as Matlab ($$$) or R (free), but I  
do think these languages suffer from some drawbacks (I could develop  
that point but it is not the subject!).  
Moreover, if I write such a soft using Ruby for instance, I could  
benefit from both Matlab an R (there are libraries to encapsulate their  
engine)…

> > Here is roughly what I want
> > 
> > - a terminal window with a ruby interpreter in it (I want to be able to  
> > control all the completions/editions capabilities)
> 
> readline?

I am not sure to understand… I want a window in which I can redirect  
any key being hit so that I can implement multi-line editing commands,  
history completion and so on …  
readline is a library ?

> > - a bench of ruby classes to manipulate signals/images,etc…
> > - I want everything to be really really fast (I’ll be manipulating huge  
> > signals/images)
> 
> ruby/gsl? r-ruby?

nope and this is clear from my experience… I know pretty precisely  
what I want … and it is not on the market… I’ll have to do it (I  
alredy started and it is not the hardes part at all).  
But yes, I want my soft to be able to communicate with the libraries  
that deal with the Matlab engine and the R engine

> > - simple commands to display instances of these classes
> > - graphic (programable) interaction on these displays (e.g. moving the  
> > mouse on a such a window should display image values, clicking should  
> > zoom, etc…
> 
> wxRuby is well suited to this sort of application.

Yes I know it is, since LastWave has been done fully on wxWidget !

> > Now my questions are the following
> > 
> > - Should I use irb ? Can I wrap it easily in a wxWidget window ? Can I  
> > reprogram some new capabilities (e.g., history completion, multi-line  
> > history commands…)
> 
> I don’t know of any examples of wrapping IRB in wxRuby, although it does  
> provide a library interface. You might look into Wx::StyledTextCtrl  
> which offers syntax highlighting and other more sophisticated support  
> for programming code text.

Is there a irb documentation (of the class I mean) anywhere on the web ?  
I could not find it!

> > - A unix terminal window displays output text really really fast.. what  
> > object should I use in wxRuby to display text as fast (wxTextCtrl is  
> > pretty slow, even if I use a buffer)
> 
> The text display classes are Wx::TextCtrl, Wx::LogTextCtrl,  
> Wx::StyledTextCtrl, Wx::RichTextCtrl and Wx::HtmlControl. I’d expect  
> simple TextCtrl or StyledTextCtrl to be fastest, but I’ve never tested.

No wxTextCtrl is what I used on LastWave and I have been doing some test  
on wxRuby recently. It is fast, but really nothing compared to a unix  
terminal…

> Judicious use of #freeze and #thaw might help to manage updates on  
> screen.  
> There are methods of which class ?

> Generally Ruby is fairly slow, and wxRuby’s bridge to C++ adds  
> additional overhead to each Wx method call, so you might improve speed  
> by batching calls to the Wx API to update the TextCtrl’s context.  
> Yes I agree and plan to do that (though I heard ruby 1.9 is much faster!  
> Do you think I should use python instead?)

> > - I want to be able to interrupt long computations (clicking on a stop  
> > button or typing ctrl-c on the terminal window) ==\> I should be able to  
> > send SIGINT signal to the interpreter. Should I use 2 processes : one  
> > for the terminal and one for the rest ?  
> > The problem if I do that is that both processes will have a  
> > wxMainLoop… now what will be the best way for them to communicate ?
> 
> You have a couple of options:
> 
> - if the computational task can be broken down into small discrete  
> elements, you can execute these in an evt\_idle handler, which runs when  
> the GUI isn’t doing anything else. This is the easiest route because you  
> can allow interruptions using normal wxRuby event handling techniques.

No, what you call the “comoutational part” is not only computational…  
It has graphics drawing (that must be fast) … I want to write  
something “a la matlab” so there is not a clear cut in between  
terminal/graphics and computations

> - you could use Ruby threads and run the computation in a slave thread.  
> This is somewhat more complex because the slave thread should not  
> directly update the GUI, but should signal its state by posting Wx  
> events to the main GUI thread. A class that does this sort of thing is  
> here (afraid it’s embedded in a fairly complex Wx app):
> 
> [http://weft-qda.rubyforge.org/svn/trunk/weft-qda/lib/weft/wxgui/gui\_subscriber.rb](http://weft-qda.rubyforge.org/svn/trunk/weft-qda/lib/weft/wxgui/gui_subscriber.rb)
> 
> If you use wxRuby and threads in Ruby 1.8 you need to explicitly  
> allocate time to the slave thread - there are numerous discussions of  
> this in the list archives.  
> Yes I tried (maybe too naively) doing threads… but it slowed down the  
> computations a lot .. again I maight have done it the wrong way !

> - you could fork / pipe and run the computation as a separate process.  
> There is no need for this to be a GUI process, I presume, so you don’t  
> need to worry about a separate GUI main\_loop. One good thing about this  
> route is that it will take advantage of multiple cores if they’re  
> available, and will separate the GUI and analytic code. But it’ll  
> involve working out how to pass data to the subsidiary process (eg  
> popen3?) and manage and signal to it (SIGINT etc?). Given Ruby’s unix  
> background, managing a child process is easier if you only need the  
> program to work on \*nix platforms, not Windows.

I want to work on ALL platforms. You think I sould not use 2 processes  
then ?  
Now how would the 2 processes communicate if they are both running  
wxMainLoop (one for the terminal and the other one for all other  
graphics…) ?

There is one thing VERY important for me : I need to be able to stop a  
computation if the user decides it’s taking too long for instance… and  
I don’t want the soft to crash !

Thank you again

Emmanuel

> hth  
> alex

---

<div class="post-metadata">

**Author:** ![emmanuel](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/3e96dc/32.png) [@emmanuel](https://www.ruby-forum.com/u/emmanuel)\
**Post date:** [May 19, 2010, 5:00pm UTC](https://www.ruby-forum.com/t/need-some-advice-scientific-scripting-language-project/188843/4 "2010-05-19T17:00:25Z")

</div>

On 19/05/2010 14:10, Emmanuel E. wrote:

> I will let you know. Actually I already wrote a script language (called  
> LastWave, [http://www.cmap.polytechnique.fr/~bacry/LastWave/index.html](http://www.cmap.polytechnique.fr/~bacry/LastWave/index.html))  
> that has been quite used by a lot of persons… I started it a long time  
> ago, so I did not use the modern script language (python or ruby)…  
> it is version 3.0 now and using wxWidget… but it clearly needs to be  
> rewritten using a good scripting language… it needs to be entirely  
> redesigned…

Looks good, although I don’t understand the topic of analysis 😉

> history completion and so on …  
> readline is a library ?

I was wondering if the library could be used to provide  
command-completion and history features - but on looking now maybe it’s  
quite tied to the terminal which hosts it.

> > ruby/gsl? r-ruby?
> 
> nope and this is clear from my experience… I know pretty precisely  
> what I want … and it is not on the market… I’ll have to do it (I  
> alredy started and it is not the hardes part at all).  
> But yes, I want my soft to be able to communicate with the libraries  
> that deal with the Matlab engine and the R engine

I’d be really interested to hear how you find the Ruby-R bindings, as  
I’ve been doing increasing amounts of stat analysis with R over the past  
year.

> > I don’t know of any examples of wrapping IRB in wxRuby, although it does  
> > provide a library interface. You might look into Wx::StyledTextCtrl  
> > which offers syntax highlighting and other more sophisticated support  
> > for programming code text.
> 
> Is there a irb documentation (of the class I mean) anywhere on the web ?  
> I could not find it!

I can’t find anything either - it doesn’t appear on [ruby-doc.org](http://ruby-doc.org) even  
though it’s part of the standard library. I think when I did a bit of  
work extending it I just relied on the source files.

The 1.9 module structure and method listing is here:  
[http://ruby-doc.org/ruby-1.9/classes/IRB.html](http://ruby-doc.org/ruby-1.9/classes/IRB.html)

> on wxRuby recently. It is fast, but really nothing compared to a unix  
> terminal…

No, I imagine not. Like to know whether there is a perceptible speed  
difference between wxTextCtrl (C++) and Wx::TextCtrl (Ruby) performing a  
comparable task.

> > Judicious use of #freeze and #thaw might help to manage updates on  
> > screen.
> 
> There are methods of which class ?

Wx::Window, so applies to all things that are drawn on screen.

> > Generally Ruby is fairly slow, and wxRuby’s bridge to C++ adds  
> > additional overhead to each Wx method call, so you might improve speed  
> > by batching calls to the Wx API to update the TextCtrl’s context.
> 
> Yes I agree and plan to do that (though I heard ruby 1.9 is much faster!  
> Do you think I should use python instead?)

Ruby 1.9 brought particular speed improvements to some numeric  
operations, which might be relevant to your project.

Python?! I don’t know. It probably is faster. But it’s not so much  
faster that it should outweigh your preference of language, and tools.  
wxPython is of course very well established, with a longer history than  
wxRuby.

> No, what you call the “comoutational part” is not only computational…  
> It has graphics drawing (that must be fast) … I want to write  
> something “a la matlab” so there is not a clear cut in between  
> terminal/graphics and computations

It sounds like you have a decomposable task - ie there are interim or  
partial results from part of the whole computation. So any option is  
open to you. It’s a question of what’s fast enough, and how it’s most  
comfortable to pass data (results) back to the GUI layer to be drawn on  
screen.

> > If you use wxRuby and threads in Ruby 1.8 you need to explicitly  
> > allocate time to the slave thread - there are numerous discussions of  
> > this in the list archives.
> 
> Yes I tried (maybe too naively) doing threads… but it slowed down the  
> computations a lot .. again I maight have done it the wrong way !

Yes, it’ll be pretty inefficient because the Ruby thread scheduler is  
being forced to jump (rather clumsily) between the GUI event loop and  
the computational task in the slave thread. You might test evt\_idle to  
see if it processes the same task faster.

> I want to work on ALL platforms. You think I sould not use 2 processes  
> then ?

Ordinary ruby will not take advantage of more than one core - but if you  
spawn a separate process it may run on a different core. This may be a  
big advantage for your use case.

> Now how would the 2 processes communicate if they are both running  
> wxMainLoop (one for the terminal and the other one for all other  
> graphics…) ?

You could perhaps open a subprocess with IO#popen  
([http://ruby-doc.org/core/classes/IO.html#M002242](http://ruby-doc.org/core/classes/IO.html#M002242)), write the data to be  
processed to the process’s STDIN, and then read the results back as  
they’re available on the process’s STDOUT (and do any graphical display  
that’s wanted).

> There is one thing VERY important for me : I need to be able to stop a  
> computation if the user decides it’s taking too long for instance… and  
> I don’t want the soft to crash !

Several ways - perhaps just closing the pipes to the subprocess will  
kill it; you could send it a system signal; or kill it with Process.  
Afraid I don’t know what is needed to make this portable to Windows.

a

---

<div class="post-metadata">

**Author:** ![emmanuel](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/3e96dc/32.png) [@emmanuel](https://www.ruby-forum.com/u/emmanuel)\
**Post date:** [May 20, 2010, 12:01am UTC](https://www.ruby-forum.com/t/need-some-advice-scientific-scripting-language-project/188843/5 "2010-05-20T00:01:23Z")

</div>

Emmanuel E. wrote:

> Thanks again for teh time you spend answering me !

You’re welcome.

> > No, I imagine not. Like to know whether there is a perceptible speed  
> > difference between wxTextCtrl (C++) and Wx::TextCtrl (Ruby) performing a  
> > comparable task.
> 
> No there is none

That’s useful to know, thanks.

> yes, so for that, the receiving process should have idle events reading  
> the stdin right to wait for inputs right?

Something like that, yes. Use evt\_idle if possible since you can  
directly update the GUI state in that handler. Dealing with threads,  
posting events, synchronisation etc is much more hassle.

> Do you know if this works well on Windows platform ?

IIRC Ruby’s own fork, popen and popen3 DON’T work on Windows. However  
there are definitely libraries which provide comparable functionality  
and similar API through the Win32 API.

> This is the signal thing you are afraid it might be not portable or even  
> more than that ?

I’m not sure but I think SIGINT / SIGKILL are \*nix specific. Just having  
the pipe might be enough to control the slave process and terminate  
when needed. Otherwise there are various other options for communicating  
between processes - have a search for ‘ruby IPC’.

best  
alex

---

<div class="post-metadata">

**Author:** ![emmanuel](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/3e96dc/32.png) [@emmanuel](https://www.ruby-forum.com/u/emmanuel)\
**Post date:** [May 20, 2010, 1:12am UTC](https://www.ruby-forum.com/t/need-some-advice-scientific-scripting-language-project/188843/6 "2010-05-20T01:12:48Z")

</div>

Not to jump in and ruin the discussion, quite good stuff. I just have  
one  
thing to comment on:  
popen3/popen/etc/etc.

When I was working on my Ruby IDE, I found the limitation of  
popen/popen3,  
and the very many different implementations of how to properly utilize  
the  
API to do what you wanted. So I created my own Library, called  
xprocess.  
It utilizes the same methods that are used across all platforms, and  
provides a singular, unified API, that is consistent, and the same  
across  
all platform.

I don’t have any Gem build of it, or published on RubyForge, as I only  
had  
the need for the library at the time. It will need to be compiled on  
any  
platform you wish to implement it on, but you can get it from  
[http://repos.ruby-im.net/xprocess](http://repos.ruby-im.net/xprocess) There is currently no documentation  
for  
it, but you can look in the test directory, and see a couple of samples  
of  
how to use it.

hth,

Mario

---

<div class="post-metadata">

**Author:** ![emmanuel](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/3e96dc/32.png) [@emmanuel](https://www.ruby-forum.com/u/emmanuel)\
**Post date:** [May 19, 2010, 9:06pm UTC](https://www.ruby-forum.com/t/need-some-advice-scientific-scripting-language-project/188843/7 "2010-05-19T21:06:53Z")

</div>

Thanks again for teh time you spend answering me !

> I’d be really interested to hear how you find the Ruby-R bindings, as  
> I’ve been doing increasing amounts of stat analysis with R over the past  
> year.

Ok, let me go into it further and I’ll develop that in an other post.

> No, I imagine not. Like to know whether there is a perceptible speed  
> difference between wxTextCtrl (C++) and Wx::TextCtrl (Ruby) performing a  
> comparable task.  
> No there is none

> > > Judicious use of #freeze and #thaw might help to manage updates on  
> > > screen.
> > 
> > There are methods of which class ?
> 
> Wx::Window, so applies to all things that are drawn on screen.  
> thanks !

> Ruby 1.9 brought particular speed improvements to some numeric  
> operations, which might be relevant to your project.
> 
> Python?! I don’t know. It probably is faster. But it’s not so much  
> faster that it should outweigh your preference of language, and tools.  
> wxPython is of course very well established, with a longer history than  
> wxRuby.

ok

> > Yes I tried (maybe too naively) doing threads… but it slowed down the  
> > computations a lot .. again I maight have done it the wrong way !
> 
> Yes, it’ll be pretty inefficient because the Ruby thread scheduler is  
> being forced to jump (rather clumsily) between the GUI event loop and  
> the computational task in the slave thread. You might test evt\_idle to  
> see if it processes the same task faster.

I’ll try!

> > I want to work on ALL platforms. You think I sould not use 2 processes  
> > then ?
> 
> Ordinary ruby will not take advantage of more than one core - but if you  
> spawn a separate process it may run on a different core. This may be a  
> big advantage for your use case.
> 
> > Now how would the 2 processes communicate if they are both running  
> > wxMainLoop (one for the terminal and the other one for all other  
> > graphics…) ?
> 
> You could perhaps open a subprocess with IO#popen  
> ([http://ruby-doc.org/core/classes/IO.html#M002242](http://ruby-doc.org/core/classes/IO.html#M002242)), write the data to be  
> processed to the process’s STDIN, and then read the results back as  
> they’re available on the process’s STDOUT (and do any graphical display  
> that’s wanted).

yes, so for that, the receiving process should have idle events reading  
the stdin right to wait for inputs right?

Do you know if this works well on Windows platform ?

> > There is one thing VERY important for me : I need to be able to stop a  
> > computation if the user decides it’s taking too long for instance… and  
> > I don’t want the soft to crash !
> 
> Several ways - perhaps just closing the pipes to the subprocess will  
> kill it; you could send it a system signal; or kill it with Process.  
> Afraid I don’t know what is needed to make this portable to Windows.

This is the signal thing you are afraid it might be not portable or even  
more than that ?

Emmanuel

---

<div class="post-metadata">

**Author:** ![emmanuel](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/3e96dc/32.png) [@emmanuel](https://www.ruby-forum.com/u/emmanuel)\
**Post date:** [May 20, 2010, 11:42am UTC](https://www.ruby-forum.com/t/need-some-advice-scientific-scripting-language-project/188843/8 "2010-05-20T11:42:17Z")

</div>

Ok,

let me describe what I have done and working very well on a macos X…  
I am going to sum up the code … (just have in mind that the goal is to  
get an interaction between a terminal window and computations/graphics  
in the “same” way as R for instance).

I still have a few questions (to both of you !) and any comments will be  
great !

=====  
The father process opens a child process (using popen mode “r+”) and  
opens a window with a TextCtrl in it and just waits with wxMainLoop.  
This process has basically two possible status : i) we are typing in a  
command (so this is a basic line-editor program) and as soon as we hit a  
return key, this command is sent to the child process using pipe.puts  
(pipe is the object returned by the popen command for now) and then it  
switches to the ii) waiting mode.  
In this last mode, the process (every 10ms) reads the stdout of the  
child process (pipe.read\_nonblock) and prints in the terminal window  
whatever is in it as long as a specific character sequence is found  
indicating that the command has been executed by the child process ==\>  
return to the command mode i)

=====  
The child process is very simple : a wxMainLoop (again!) and every 10ms  
an Idle procedure is ran.  
It reads the $stdin using $stdin.read\_nonblock. If there is something  
then it corresponds to a command that will be executed, e.g.,  
$ans = eval(command,TOPLEVEL\_BINDING)  
(I take care of eventual errors of course)  
And then, it just $stdout.puts the specific character sequence that  
indicates command execution is over and $stdout.flush

Notice that this process can open windows, draw graphics, handle  
interactions with these graphics …

++++++++++++++

That’s it.  
In this way, the display on the terminal is really fast (never got  
something as fast). Be aware that the (ruby) command that is executed  
with eval can use puts … and it will print on the terminal.  
The output buffering system is the one of the system…!

So for instance

100000.times {|i| puts i}

takes less that 2 seconds !  
Which is really fast.

Moreover if a ctrl-C is typed in the terminal window I can send a SIGINT  
to the child process (I do it using C raise) which interrupts the  
execution of the ruby command !?!

I was really happy… till I realized that, it will not work on Windows  
XP!

==\> First I looked on the web to see if there is anything that could  
replace the popen. I found popen4… BUT  
a- You have to pass a block (at the end of the block the process  
dies)… which is not good for me since I want the child process to live  
through the wxMainLoop  
b- (which is related to a- I think) I want to do a read\_nonblock (so I  
don’t reach the eof … )

==\> MARIO : does your library will help me ? Does it manage the  
stdin/stdout of the child process in a standard way (buffering and  
all)? Is it non blocking ? Can I read and write (non blocking)  
alternatively without closing the process ?

==\> Do you have any other idea (Alex, I guess you understand exactly  
what I want to do since you know R)

Thank you for everything !!

Emmanuel

---

<div class="post-metadata">

**Author:** ![emmanuel](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/3e96dc/32.png) [@emmanuel](https://www.ruby-forum.com/u/emmanuel)\
**Post date:** [May 20, 2010, 8:51pm UTC](https://www.ruby-forum.com/t/need-some-advice-scientific-scripting-language-project/188843/9 "2010-05-20T20:51:10Z")

</div>

Hello Emmanuel,

The XProcess library I developed, is asynchronous, and has methods in  
which  
to detect if there is data available on a pipe object or not. I setup  
the  
library specifically for the need to be able to be async to keep the GUI  
responsive while the sub-process is being executed.

As I said before, there are many different methods across Windows,  
Linux,  
and OS X, in which to create a pipe to a sub-process, and Windows lacks  
the  
major feature in which to allow sub-proceses naturally in ruby, which is  
Fork. What I have done, is created a single API interface in which to  
be  
able to create processes, and send and receive data from the  
sub-process,  
without blocking the wxWidgets API.

I did think about wrapping wxProcess, but after looking through the  
documentation for it, I felt that it would just be simpler to create the  
API  
myself in another class, then to go through the trouble of trying to  
wrap  
wxProcess.

The simplest way I can explain the API for XProcess, is that you  
initialize  
the class with the executable you wish to run. EG:

xproc = XProcess.new(“ruby”)

Then you use the method execute to pass along command line arguments to  
the  
sub-process, such as -e, -v, or whatever command line arguments your  
sub-process needs.

Then the xproc object has 4 attributes, pid, which is the process id,  
in,  
out, and err, which are the representations of STDIN, STDOUT, and STDERR  
for  
the sub-process.

You can get output from the process by doing:

if xproc.out.is\_data\_avail?  
data = xproc.out.gets  
puts “Received: #{data}”  
end

The method is\_data\_avail? on STDOUT and STDERR will not block the  
process,  
it will either return true, or false.

If you want to see if the process has closed either STDOUT or STDERR,  
you  
simply check by using the eof? method on the STDOUT or STDERR.

if xproc.out.eof?  
puts “The process has closed STDOUT”  
end

XProcess doesn’t close the IO handles for input, output or error, so you  
will need to close them yourself, if you don’t want to use them. And  
if  
you want to kill the process if it’s taking to long, simply call the  
kill  
method.

I will need to write up some documentation for the XProcess library, but  
it’s fairly straight forward, and simple to use. I try to keep things  
simplified, and unified across all platforms.

I have had confirmation that this library works on all 3 platforms, but  
if  
you find any bugs, please feel free to let me know, off list, and I will  
see  
what I can do to fix the problems.

hth,

Mario

---

<div class="post-metadata">

**Author:** ![emmanuel](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/3e96dc/32.png) [@emmanuel](https://www.ruby-forum.com/u/emmanuel)\
**Post date:** [May 20, 2010, 11:47am UTC](https://www.ruby-forum.com/t/need-some-advice-scientific-scripting-language-project/188843/10 "2010-05-20T11:47:41Z")

</div>

ouups :

==\> what about using wxProcess (i.e., creating a process using wxWigets  
API) ??

E.

---

<div class="post-metadata">

**Author:** ![emmanuel](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/3e96dc/32.png) [@emmanuel](https://www.ruby-forum.com/u/emmanuel)\
**Post date:** [May 20, 2010, 9:36pm UTC](https://www.ruby-forum.com/t/need-some-advice-scientific-scripting-language-project/188843/11 "2010-05-20T21:36:54Z")

</div>

Thanks Mario for this long answer !

Few questions

1- Is there also a way to read a maximum number of bytes (non blocking,  
knowing that there are some data available) ?  
(the equivalent of .read\_nonblock()

2- Is it working with ruby 1.9 ?

3- How do I install it, is there a small doc ?

Thanks

Emmanuel

---

<div class="post-metadata">

**Author:** ![emmanuel](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/3e96dc/32.png) [@emmanuel](https://www.ruby-forum.com/u/emmanuel)\
**Post date:** [May 20, 2010, 11:22pm UTC](https://www.ruby-forum.com/t/need-some-advice-scientific-scripting-language-project/188843/12 "2010-05-20T23:22:46Z")

</div>

ouups sorry for this message (again!)

I was wrong in my last post… it is not blocking. Something really  
weired happens.  
As I already explained both processes run a wxMainLoop and both have  
wxTimers that wake up the process regularly.

So when the child process is processing the stdin the parent process  
does not receive any alarm from its timer any more ??!

---

<div class="post-metadata">

**Author:** ![emmanuel](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/3e96dc/32.png) [@emmanuel](https://www.ruby-forum.com/u/emmanuel)\
**Post date:** [May 20, 2010, 10:48pm UTC](https://www.ruby-forum.com/t/need-some-advice-scientific-scripting-language-project/188843/13 "2010-05-20T22:48:18Z")

</div>

Okay Mario,

I did install it, it seems to work … except that, I am doing  
something wrong maybe, but in order to send to the child process some  
strings into the stdin, I do

cmd.in.puts(string)

Now, when the child process finds a string in the stdin, it processes it  
… and really weired… but this cmd.in.puts is blocking as long as  
the child process has finished its processing ???

I really don’t understand what’s hapenning here

Could you help me ?

Emmanuel

---

<div class="post-metadata">

**Author:** ![emmanuel](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/3e96dc/32.png) [@emmanuel](https://www.ruby-forum.com/u/emmanuel)\
**Post date:** [May 21, 2010, 12:01pm UTC](https://www.ruby-forum.com/t/need-some-advice-scientific-scripting-language-project/188843/14 "2010-05-21T12:01:50Z")

</div>

Hello Mario,

I identified the problem (first of all no I am NOT using ruby 1.9… I  
was just asking …) :

When the child process is writing quickly to the stdout then the call  
(from the main process) to

cmd.out.data\_avail?

basically waits for the child process to have finished writing…

* * *

Example :  
parent process runs Idle every 10ms. Idle looks at the stdout of the  
child process and prints on a textCtrl whatever is in it

child process runs 100000.times {|i| puts i}

==\> The whole thing takes 3 seconds to run (macos X) but the flush is  
done just once !  
(Idle is called once and blocks at cmd.out.data\_avail?, wait for child  
process to finish writing and then displays all the numbers at once)

Let’s note that if I put a sleep after the puts then the display is  
updated very often during the loop…

Emmanuel

---

<div class="post-metadata">

**Author:** ![emmanuel](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/3e96dc/32.png) [@emmanuel](https://www.ruby-forum.com/u/emmanuel)\
**Post date:** [May 21, 2010, 2:44pm UTC](https://www.ruby-forum.com/t/need-some-advice-scientific-scripting-language-project/188843/15 "2010-05-21T14:44:51Z")

</div>

hi

Glad you’re making progress; not surprised to hear that Windows is the  
sticking point. Mario knows a lot more than me about using pipes etc  
with wxRuby so just a few observations that might or might not be  
useful:

Emmanuel E. wrote:

> The child process is very simple : a wxMainLoop (again!) and every 10ms  
> an Idle procedure is ran.

Are you sure child process needs to be a wx program at all? I would  
expect it to be a simple Ruby non-GUI program if it just uses STDIN and  
STDOUT to communicate in an efficient format. With that you can test  
different options, eg

- a long-running server to handle many jobs (which starts up and then  
enters an endless “loop do … end”) vs a per-job new process. If it’s  
like R, the overhead of starting a new process per-job is insignificant  
against the overall job computation time. And you can close the write  
pipe on each job and then read data incrementally, rather than the more  
complex popen3 model.

- I wonder if Drb (distributed ruby) might work? I have used it with  
some success in attaching a wxRuby GUI to an application running  
separately. One other bonus of this is it would then be easy to move to  
a client/server model on different pcs if wanted in the future.

> Notice that this process can open windows, draw graphics, handle  
> interactions with these graphics …

Is this in the child process? That seems an unusual arrangement but of  
course there might be very good reasons for this…

> ++++++++++++++
> 
> That’s it.  
> In this way, the display on the terminal is really fast (never got  
> something as fast).

Great. I’m away on hols for a few weeks now so won’t be picking up, but  
best of luck with the project

alex

---

<div class="post-metadata">

**Author:** ![emmanuel](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/3e96dc/32.png) [@emmanuel](https://www.ruby-forum.com/u/emmanuel)\
**Post date:** [May 21, 2010, 1:03am UTC](https://www.ruby-forum.com/t/need-some-advice-scientific-scripting-language-project/188843/16 "2010-05-21T01:03:31Z")

</div>

Hey Emmanuel,

First off, I haven’t really tested Ruby 1.9, so if it’s working for you,  
that’s great.

Second off, once any output to stdin is executed, it should  
automatically  
return, without any delays. I will have to investigate this more,  
unless  
it’s this third thing.

Third, are you using Ruby 1.9, Wx::Timer, and Threads in the setup?  
Cause,  
if your using Ruby 1.9, there is no need to utilize Wx::Timer, to wake  
up  
your application, as Ruby 1.9 has native Thread support, which is  
compatible  
with wxWidgets thread utilization setup. The only problem you would  
have,  
is attempting to update the GUI from any other thread, outside the Main  
Thread. Which is the base level thread, that your app runs in. If you  
attempt to update the GUI from any other thread, wxWidgets will panic,  
and  
seg-fault on you, or lock your program up.

I will be in IRC, if you wish to discuss more about my library. irc://  
[irc.freenode.net:6667/#wxruby](http://irc.freenode.net:6667/#wxruby)

hth,

Mario
