Hi All,
I have a client for whom I developed a small application using Ruby
and Curl to upload files to his file server (to and from Windows
boxes). He now wants that application to have multiple sockets in
order to upload more than one file at a time. I have no idea what
course is the safest here. Do I simply create threads that run the
“curl” command and let that be, or do I try a library such as (http://
www.chilkatsoft.com/upload-ruby.asp) and go from there? I’m honestly
not familiar with threading or chilkat.
Any suggestions would be most appreciated.
Regards.
2008/6/22 gberz3 [email protected]:
I have a client for whom I developed a small application using Ruby
and Curl to upload files to his file server (to and from Windows
boxes). He now wants that application to have multiple sockets in
order to upload more than one file at a time. I have no idea what
course is the safest here. Do I simply create threads that run the
“curl” command and let that be,
You just need to fork multiple curl processes.
or do I try a library such as (http://
www.chilkatsoft.com/upload-ruby.asp) and go from there? I’m honestly
not familiar with threading or chilkat.
I do not know that lib. If it does provide multiple concurrent
uploads that’s probably a good choice.
If you were doing it in pure Ruby (i.e. using net/http) you could use
threads.
Kind regards
robert
you should use DRb with multi-process architecture since ruby uses
green-threads, that should be much faster.
On Tue, Jun 24, 2008 at 11:06 AM, Robert K.
2008/6/24 Zhukov P. [email protected]:
you should use DRb with multi-process architecture since ruby uses
green-threads, that should be much faster.
This is not necessarily true for IO bound tasks. Also, if curl is used
you just need multiple child processes but not multiple threads.
Kind regards
robert
On Jun 24, 7:03 am, Robert K. [email protected] wrote:
This is not necessarily true for IO bound tasks. Also, if curl is used
you just need multiple child processes but not multiple threads.
Kind regards
robert
Alright, I’m currently using Threads and it has sped things up
tremendously. Thank you both for your suggestions.
One problem I’m having now is that sometimes a file will hang cURL,
and while the other subprocesses finish there might be one or two that
hang up. In this case I’d like to have some kind of granular control
over killing individual items without necessarily killing the whole
thread. I’d be afraid to kill the entire thread as it may not be
finished processing legitimate cURL processes when you send the kill
switch.
Thoughts?
On 24.06.2008 22:56, gberz3 wrote:
On Jun 24, 7:03 am, Robert K. [email protected] wrote:
This is not necessarily true for IO bound tasks. Also, if curl is used
you just need multiple child processes but not multiple threads.
Alright, I’m currently using Threads and it has sped things up
tremendously. Thank you both for your suggestions.
But note that the speedup likely comes from using multiple curl
processes - not from using multiple Ruby threads.
One problem I’m having now is that sometimes a file will hang cURL,
and while the other subprocesses finish there might be one or two that
hang up. In this case I’d like to have some kind of granular control
over killing individual items without necessarily killing the whole
thread. I’d be afraid to kill the entire thread as it may not be
finished processing legitimate cURL processes when you send the kill
switch.
Thoughts?
Killing threads does nothing to your external processes. You can either
kill curl and force its termination or you need to investigate whether
curl is built in a way that it responds to certain signals by dropping
the current upload.
Kind regards
robert
Michael Williams wrote:
On Jun 24, 7:03�am, Robert K. [email protected] wrote:
This is not necessarily true for IO bound tasks. Also, if curl is used
you just need multiple child processes but not multiple threads.
Kind regards
robert
Alright, I’m currently using Threads and it has sped things up
tremendously. Thank you both for your suggestions.
One problem I’m having now is that sometimes a file will hang cURL,
and while the other subprocesses finish there might be one or two that
hang up. In this case I’d like to have some kind of granular control
over killing individual items without necessarily killing the whole
thread. I’d be afraid to kill the entire thread as it may not be
finished processing legitimate cURL processes when you send the kill
switch.
Thoughts?
This could be caused by a bad domain name causing a dns to take a long
time… if you install the c-ares library and recompile libcurl you can
at least select and get activity while dns is resolving…
Michael Williams wrote:
Hi All,
I have a client for whom I developed a small application using Ruby
and Curl to upload files to his file server (to and from Windows
boxes). He now wants that application to have multiple sockets in
order to upload more than one file at a time. I have no idea what
course is the safest here. Do I simply create threads that run the
“curl” command and let that be, or do I try a library such as (http://
www.chilkatsoft.com/upload-ruby.asp) and go from there? I’m honestly
not familiar with threading or chilkat.
Any suggestions would be most appreciated.
Regards.
You could try the curb library. This version included support for the
Multi interface, allowing multiple easy handles to perform concurrently.
Check it out: https://github.com/taf2/curb/tree