= socket_sendfile
Rubyforge Project:
http://rubyforge.org/projects/rctools/
Documentation:
http://dev.robotcoop.com/Libraries/socket_sendfile/
== About
Socket#sendfile implements sendfile(2) for sending files without
copying data
to the process. See the sendfile(2) manual page for more details.
Note that your system must support the sendfile(2) system call the
same way
FreeBSD does for Socket#sendfile to work.
== Installing socket_sendfile
First you need an OS that supports sendfile() the way FreeBSD does.
Then install the gem:
$ sudo gem install socket_sendfile
== Using socket_sendfile
require ‘rubygems’
require ‘socket’
require ‘socket_sendfile’
socket = TCPSocket.open host, port
File.open ‘myfile’ do |fp|
socket.sendfile fp
end
–
Eric H. - [email protected] - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant
http://trackmap.robotcoop.com
[partially answering my own question …]
The reason that builder can’t create xml elements with “-” chars in
the element is because strings with “-” chars are not legal Ruby
symbols. Builder uses method_missing to create the xml element markup
and so it seems Ruby’s parser chokes when it tries to convert the
string into a symbol.
I’ve got a workaround below but it is more general than I want (it
replaces all “_” chars with “-”). I left some code commented out that
I was trying too get working but didn’t. It’s a bit confusing to
debug because the inheritance chain (XmlMarkup < XmlBase <
BlankSlate) end up hiding all the normal methods – I’m a bit
surprised that my code works at all.
Anyway, I’m just learning Ruby and figuring out just what is going on
here is fun. If anybody has suggestions I’d appreciate it.
Thanks
#!/usr/bin/env ruby
file: jnlpmarkup.rb
extends Builder::XmlMarkup for creating XML markup for jnlp documents.
Some jnlp elements have a “-” char in the element name. Builder
doesn’t
accept this character. So this class postprocesses the result and
replaces
proxy jnlp element names such as “<application_desc>” with
“”
because I can’t get what I intended to work right now this just
replaces all “_” chars with ‘-’ chars
require ‘builder/xmlmarkup’
module Builder
class JnlpMarkup < XmlMarkup
@@jnlp_in = %w{application_desc applet_desc component_desc
installer_desc offline_allowed related_content all_permissions
j2ee_application_client_permissions ext_download}
@@jnlp_out = %w{application-desc applet-desc component-desc
installer-desc offline-allowed related-content all-permissions
j2ee-application-client-permissions ext-download}
def method_missing(sym, *args, &block)
super.gsub(/_/, '-')
various things that didn’t work below:
x = super
x = x.gsub(/application_desc/, ‘application-desc’)
x = x.gsub(/applet_desc/, ‘applet_desc’)
x = x.gsub(/#{@@jnlp_in[i]}/, @@jnlp_out[i])
@@jnlp_in.each_index do |i| x = x.gsub(/#{@@jnlp_in[i]}/,
@@jnlp_out[i]) end
end
end
end
On Fri, 24 Mar 2006, Eric H. wrote:
== About
socket = TCPSocket.open host, port
File.open ‘myfile’ do |fp|
socket.sendfile fp
end
i’m just about to write a server which sends 1-3gb images as the
response…
this could be quite useful.
one gripe. it doesn’t work:
/home/ahoward is insecure (40777), needs 0700 for perms. Exiting
i work in a collaborative lab… all our home dirs are group readable by
default. any way to adjust this? an env var perhaps? seems like this
should
just warn.
ok. one more issue:
fortytwo :~/tmp > ruby ../a.rb
/usr/local/ruby-1.8.4/lib/ruby/gems/1.8/gems/RubyInline-3.4.0/./inline.rb:392:
warning: Insecure world writable dir /usr/local, mode 040777
/usr/local/ruby-1.8.4/lib/ruby/gems/1.8/gems/socket_sendfile-1.1.0/lib/socket_sendfile.rb:
In function bsock_sendfile': /usr/local/ruby-1.8.4/lib/ruby/gems/1.8/gems/socket_sendfile-1.1.0/lib/socket_sendfile.rb:46: warning: implicit declaration of functionsendfil
e’
/usr/local/ruby-1.8.4/lib/ruby/gems/1.8/gems/RubyInline-3.4.0/./inline.rb:396:in
build': error executing gcc -shared -Wall -W -Wpointer-arith - Wcast-qual -Wcast-align -Wwrite-strings -Wmissing-noreturn -Werror -g -O2 -I /usr/local/ruby-1.8.4/lib/ruby/1.8/i686-linux -o /home/ahoward/.rub y_inline/Inline_BasicSocket_0a2b.so /home/ahoward/.ruby_inline/Inline_BasicSocket_0a2b.c : 256 (CompilationError) Renamed /home/ahoward/.ruby_inline/Inline_BasicSocket_0a2b.c to /home/ahoward/.ruby_inline/Inline_BasicSocket_0a2b.c.bad from /usr/local/ ruby-1.8.4/lib/ruby/gems/1.8/gems/RubyInline-3.4.0/./inline.rb:591:ininline’
from
/usr/local/ruby-1.8.4/lib/ruby/gems/1.8/gems/socket_sendfile-1.1.0/lib/socket_sendfile.rb:6
from
/usr/local/ruby-1.8.4/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in
`require’
from …/a.rb:3
my system definitely has sendfile
fortytwo :~/tmp > PAGER=cat man sendfile|head -12
SENDFILE(2) Linux Programmer's Manual
SENDFILE(2)
NAME
sendfile - transfer data between file descriptors
SYNOPSIS
#include <sys/sendfile.h>
ssize_t sendfile(int out_fd, int in_fd, off_t *offset,
size_t count);
guess the signature won’t cut it though eh? maybe i’ll just have to use
dl?
regards.
-a
On Mar 23, 2006, at 7:24 PM, [email protected] wrote:
File.open ‘myfile’ do |fp|
i work in a collaborative lab… all our home dirs are group
readable by
default. any way to adjust this? an env var perhaps? seems like
this should
just warn.
That’s RubyInline complaining. Let me check with Ryan D. to see
what can be done.
socket_sendfile-1.1.0/lib/socket_sendfile.rb:46: warning: implicit
Renamed /home/ahoward/.ruby_inline/Inline_BasicSocket_0a2b.c
my system definitely has sendfile
#include <sys/sendfile.h>
ssize_t sendfile(int out_fd, int in_fd, off_t *offset,
size_t count);
guess the signature won’t cut it though eh? maybe i’ll just have
to use dl?
On FreeBSD it looks like this:
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/uio.h>
int
sendfile(int fd, int s, off_t offset, size_t nbytes,
struct sf_hdtr *hdtr, off_t *sbytes, int flags);
And has this to say about non-blocking sockets:
When using a socket marked for non-blocking I/O, sendfile() may
send
fewer bytes than requested. In this case, the number of bytes
success-
fully written is returned in *sbytes (if specified), and the
error EAGAIN
is returned.
Try this patch:
On Mar 23, 2006, at 7:24 PM, [email protected] wrote:
default.
Your home dir is group and world writable. A 755 directory works,
the message is broken. (Well, just fixed in perforce.)
any way to adjust this? an env var perhaps?
The INLINEDIR environment variable controls this, so set it to a
directory that’s got the right permissions (at least 755).
seems like this should just warn.
Other people may be able to inject code into ~/.ruby_inline if the
directory is writeable by other people than you.
That would be Really Bad.
A warning is insufficient. You would be royally screwed by the time
you read it.
–
Eric H. - [email protected] - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant
http://trackmap.robotcoop.com
On Mar 23, 2006, at 7:24 PM, [email protected] wrote:
default. any way to adjust this? an env var perhaps? seems like
this should
just warn.
Your home dir is world and group writable, not just group readable.
There is a big big big difference between those two. There is no way
I’m going to let it just warn in that case. However, you do point out
that the error message poor. I’ve changed the error message to:
“#{path} is insecure (#{’%o’ % mode}). It may not be group or
world writable. Exiting.”
–
_why: zenspider’s most intense moments of solice are immediately
following the slaughter […]
_why: that topknot’s the only thing keeping a lid on the righteous anger
bricolage: yeah, that and his flagrant obsession with dvorak
On Fri, 24 Mar 2006, Ryan D. wrote:
“#{path} is insecure (#{’%o’ % mode}). It may not be group or world
writable. Exiting.”
i totally understand what you are saying. however i dont’ see why a
ruby lib
should be more secure that ruby itself:
fortytwo :/tmp > ls -ltar /|grep tmp
drwxrwxrwt 21 root root 815104 Mar 24 07:41 tmp
fortytwo :/tmp > echo ‘puts 42’ > a.rb
fortytwo :/tmp > chmod 777 a.rb
fortytwo :/tmp > ruby -W2 -e’ $SAFE = 0; load “a.rb”’
42
it sure seems fine loading world writable files from world writable
directories… unless you set $SAFE above the default:
fortytwo :/tmp > ruby -W2 -e’ $SAFE = 1; load “a.rb”’
-e:1: warning: Insecure world writable dir /usr/local, mode 040777
-e:1:in `load’: loading from unsafe path
/usr/local/ruby-1.8.4/lib/ruby/site_ruby/1.8:/usr/local/ruby-1.8.4/lib/ruby/site_ruby/1.8/i686-linux:/usr/local/ruby-1.8.4/lib/ruby/site_ruby:/usr/local/ruby-1.8.4/lib/ruby/1.8:/usr/local/ruby-1.8.4/lib/ruby/1.8/i686-linux:.
(SecurityError)
from -e:1
this seems sufficient and allows stupid people like me to shoot their
own foot
off if they choose. does RubyInline really need more that this : do
nothing
unless not $SAFE.nil? and $SAFE > 0?
regards.
-a
On Fri, 24 Mar 2006, Eric H. wrote:
Your home dir is group and world writable. A 755 directory works, the
message is broken. (Well, just fixed in perforce.)
thanks figured that out later… unfortunately it needs to be as some
webdav
stuff lands there and www has it’s own group which i cannot affect…
any way to adjust this? an env var perhaps?
The INLINEDIR environment variable controls this, so set it to a directory
that’s got the right permissions (at least 755).
this works - thanks!
seems like this should just warn.
Other people may be able to inject code into ~/.ruby_inline if the directory
is writeable by other people than you.
That would be Really Bad.
A warning is insufficient. You would be royally screwed by the time you
read it.
all good but not at $SAFE==0 or $SAFE==nil. after all, $SAFE==nil
really
ought to mean $VERY_DANGEROUS_PLEASE==true! 
regards.
-a
Eric H. wrote:
= socket_sendfile
Rubyforge Project:
http://rubyforge.org/projects/rctools/
Documentation:
http://dev.robotcoop.com/Libraries/socket_sendfile/
Was there something wrong with my existing ruby-sendfile library?
http://rubyforge.org/projects/ruby-sendfile
–
Toby DiPasquale
On Sat, 25 Mar 2006, Toby DiPasquale wrote:
Was there something wrong with my existing ruby-sendfile library?
http://rubyforge.org/projects/ruby-sendfile
ahhhhhh.
-a
unknown wrote:
On Sat, 25 Mar 2006, Toby DiPasquale wrote:
Was there something wrong with my existing ruby-sendfile library?
http://rubyforge.org/projects/ruby-sendfile
ahhhhhh.
Does that work for you, ara? It should work on Linux, Solaris and
FreeBSD out of the box. I didn’t put it in a gem because I wasn’t sure
how to get the GemSpec to correctly identify only the platforms on which
it would build. Let me know.
–
Toby DiPasquale
On Sat, 25 Mar 2006, Toby DiPasquale wrote:
Does that work for you, ara? It should work on Linux, Solaris and
FreeBSD out of the box. I didn’t put it in a gem because I wasn’t sure
how to get the GemSpec to correctly identify only the platforms on which
it would build. Let me know.
it sure as heck does!
harp:~/ruby-sendfile > cat a.rb
require 'sendfile'
open(__FILE__){|f| STDOUT.sendfile f}
harp:~/ruby-sendfile > ruby a.rb
require 'sendfile'
open(__FILE__){|f| STDOUT.sendfile f}
this is fantastic. basically i’ll be responding to certain web requests
with
something like
send_mime_header
sendfile huge_massive_file
and was thinking i’d have to fork a cat to make it fast enough - so this
is
fantastic.
thanks alot.
-a
On Mar 24, 2006, at 8:39 AM, Toby DiPasquale wrote:
Was there something wrong with my existing ruby-sendfile library?
http://rubyforge.org/projects/ruby-sendfile
I didn’t know it existed, but:
It isn’t as easy to install as a gem is.
It doesn’t work well with non-blocking sockets on FreeBSD. Handling
it in Ruby is too annoying to be worthwhile.
Mine is 1.0.
Mine has tests.
–
Eric H. - [email protected] - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant
http://trackmap.robotcoop.com
On Mar 24, 2006, at 9:19 AM, Toby DiPasquale wrote:
Does that work for you, ara? It should work on Linux, Solaris and
FreeBSD out of the box. I didn’t put it in a gem because I wasn’t sure
how to get the GemSpec to correctly identify only the platforms on
which
it would build. Let me know.
I didn’t even bother to mark which platforms it would/wouldn’t work
on. Failing might get people interested enough to send you patches.
–
Eric H. - [email protected] - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant
http://trackmap.robotcoop.com
Eric H. wrote:
It isn’t as easy to install as a gem is.
Given.
It doesn’t work well with non-blocking sockets on FreeBSD. Handling
it in Ruby is too annoying to be worthwhile.
Thanks for the first suggestions for improvements to ruby-sendfile. I
will fix it over the weekend and make a gem out of it. That way Linux
and Solaris users won’t be left out, as I had originally intended they
not be.
P.S. In your test suite for socket_sendfile, you can use the
IO#nonblock= call by requiring ‘io/nonblock’ instead of having to
explicitly call fcntl. Makes the code more readable, IMO.
–
Toby DiPasquale
On Mar 24, 2006, at 11:25 AM, Toby DiPasquale wrote:
P.S. In your test suite for socket_sendfile, you can use the
IO#nonblock= call by requiring ‘io/nonblock’ instead of having to
explicitly call fcntl. Makes the code more readable, IMO.
Thanks, I was wondering where that wandered off to.
–
Eric H. - [email protected] - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant
http://trackmap.robotcoop.com