# Multiple HTTP Sockets or Threads for file uploads **URL:** \ **Category:** Ruby\ **Created:** [June 22, 2008, 4:06am UTC](https://www.ruby-forum.com/t/multiple-http-sockets-or-threads-for-file-uploads/141109 "2008-06-22T04:06:21Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![dezrtdude](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/f4b2a3/32.png) [@dezrtdude](https://www.ruby-forum.com/u/dezrtdude)\ **Post date:** [June 22, 2008, 4:06am UTC](https://www.ruby-forum.com/t/multiple-http-sockets-or-threads-for-file-uploads/141109/1 "2008-06-22T04:06:21Z")
Hi All, I have a client for whom I developed a small application using Ruby and Curl to upload files to his file server (to and from Windows boxes). He now wants that application to have multiple sockets in order to upload more than one file at a time. I have no idea what course is the safest here. Do I simply create threads that run the “curl” command and let that be, or do I try a library such as (http:// [www.chilkatsoft.com/upload-ruby.asp](http://www.chilkatsoft.com/upload-ruby.asp)) and go from there? I’m honestly not familiar with threading or chilkat. Any suggestions would be most appreciated. Regards. ---
**Author:** ![dezrtdude](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/f4b2a3/32.png) [@dezrtdude](https://www.ruby-forum.com/u/dezrtdude)\ **Post date:** [June 24, 2008, 9:09am UTC](https://www.ruby-forum.com/t/multiple-http-sockets-or-threads-for-file-uploads/141109/2 "2008-06-24T09:09:03Z")
2008/6/22 gberz3 [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid): > I have a client for whom I developed a small application using Ruby > and Curl to upload files to his file server (to and from Windows > boxes). He now wants that application to have multiple sockets in > order to upload more than one file at a time. I have no idea what > course is the safest here. Do I simply create threads that run the > “curl” command and let that be, You just need to fork multiple curl processes. > or do I try a library such as (http:// > [www.chilkatsoft.com/upload-ruby.asp](http://www.chilkatsoft.com/upload-ruby.asp)) and go from there? I’m honestly > not familiar with threading or chilkat. I do not know that lib. If it does provide multiple concurrent uploads that’s probably a good choice. If you were doing it in pure Ruby (i.e. using net/http) you could use threads. Kind regards robert ---
**Author:** ![dezrtdude](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/f4b2a3/32.png) [@dezrtdude](https://www.ruby-forum.com/u/dezrtdude)\ **Post date:** [June 24, 2008, 9:14am UTC](https://www.ruby-forum.com/t/multiple-http-sockets-or-threads-for-file-uploads/141109/3 "2008-06-24T09:14:19Z")
you should use DRb with multi-process architecture since ruby uses green-threads, that should be much faster. On Tue, Jun 24, 2008 at 11:06 AM, Robert K. ---
**Author:** ![dezrtdude](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/f4b2a3/32.png) [@dezrtdude](https://www.ruby-forum.com/u/dezrtdude)\ **Post date:** [June 24, 2008, 1:06pm UTC](https://www.ruby-forum.com/t/multiple-http-sockets-or-threads-for-file-uploads/141109/4 "2008-06-24T13:06:44Z")
2008/6/24 Zhukov P. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid): > you should use DRb with multi-process architecture since ruby uses > green-threads, that should be much faster. This is not necessarily true for IO bound tasks. Also, if curl is used you just need multiple child processes but not multiple threads. Kind regards robert ---
**Author:** ![dezrtdude](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/f4b2a3/32.png) [@dezrtdude](https://www.ruby-forum.com/u/dezrtdude)\ **Post date:** [June 24, 2008, 11:01pm UTC](https://www.ruby-forum.com/t/multiple-http-sockets-or-threads-for-file-uploads/141109/5 "2008-06-24T23:01:04Z")
On Jun 24, 7:03 am, Robert K. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > This is not necessarily true for IO bound tasks. Also, if curl is used > you just need multiple child processes but not multiple threads. > > Kind regards > > robert Alright, I’m currently using Threads and it has sped things up tremendously. Thank you both for your suggestions. One problem I’m having now is that sometimes a file will hang cURL, and while the other subprocesses finish there might be one or two that hang up. In this case I’d like to have some kind of granular control over killing individual items without necessarily killing the whole thread. I’d be afraid to kill the entire thread as it may not be finished processing legitimate cURL processes when you send the kill switch. Thoughts? ---
**Author:** ![dezrtdude](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/f4b2a3/32.png) [@dezrtdude](https://www.ruby-forum.com/u/dezrtdude)\ **Post date:** [June 25, 2008, 7:30am UTC](https://www.ruby-forum.com/t/multiple-http-sockets-or-threads-for-file-uploads/141109/6 "2008-06-25T07:30:29Z")
On 24.06.2008 22:56, gberz3 wrote: > On Jun 24, 7:03 am, Robert K. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > > > This is not necessarily true for IO bound tasks. Also, if curl is used > > you just need multiple child processes but not multiple threads. > > Alright, I’m currently using Threads and it has sped things up > tremendously. Thank you both for your suggestions. But note that the speedup likely comes from using multiple curl processes - not from using multiple Ruby threads. > One problem I’m having now is that sometimes a file will hang cURL, > and while the other subprocesses finish there might be one or two that > hang up. In this case I’d like to have some kind of granular control > over killing individual items without necessarily killing the whole > thread. I’d be afraid to kill the entire thread as it may not be > finished processing legitimate cURL processes when you send the kill > switch. > > Thoughts? Killing threads does nothing to your external processes. You can either kill curl and force its termination or you need to investigate whether curl is built in a way that it responds to certain signals by dropping the current upload. Kind regards robert ---
**Author:** ![dezrtdude](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/f4b2a3/32.png) [@dezrtdude](https://www.ruby-forum.com/u/dezrtdude)\ **Post date:** [July 24, 2008, 4:48am UTC](https://www.ruby-forum.com/t/multiple-http-sockets-or-threads-for-file-uploads/141109/7 "2008-07-24T04:48:31Z")
Michael Williams wrote: > On Jun 24, 7:03�am, Robert K. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > > > This is not necessarily true for IO bound tasks. Also, if curl is used > > you just need multiple child processes but not multiple threads. > > > > Kind regards > > > > robert > > Alright, I’m currently using Threads and it has sped things up > tremendously. Thank you both for your suggestions. > > One problem I’m having now is that sometimes a file will hang cURL, > and while the other subprocesses finish there might be one or two that > hang up. In this case I’d like to have some kind of granular control > over killing individual items without necessarily killing the whole > thread. I’d be afraid to kill the entire thread as it may not be > finished processing legitimate cURL processes when you send the kill > switch. > > Thoughts? This could be caused by a bad domain name causing a dns to take a long time… if you install the c-ares library and recompile libcurl you can at least select and get activity while dns is resolving… ---
**Author:** ![dezrtdude](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/f4b2a3/32.png) [@dezrtdude](https://www.ruby-forum.com/u/dezrtdude)\ **Post date:** [July 24, 2008, 4:45am UTC](https://www.ruby-forum.com/t/multiple-http-sockets-or-threads-for-file-uploads/141109/8 "2008-07-24T04:45:39Z")
Michael Williams wrote: > Hi All, > > I have a client for whom I developed a small application using Ruby > and Curl to upload files to his file server (to and from Windows > boxes). He now wants that application to have multiple sockets in > order to upload more than one file at a time. I have no idea what > course is the safest here. Do I simply create threads that run the > “curl” command and let that be, or do I try a library such as (http:// > [www.chilkatsoft.com/upload-ruby.asp](http://www.chilkatsoft.com/upload-ruby.asp)) and go from there? I’m honestly > not familiar with threading or chilkat. > > Any suggestions would be most appreciated. > > Regards. You could try the curb library. This version included support for the Multi interface, allowing multiple easy handles to perform concurrently. Check it out: [https://github.com/taf2/curb/tree](https://github.com/taf2/curb/tree)