I was thinking about writing a patch to modify how Marshal handles
floats, right now it dumps them using sprintf(3) and stores the
resulting string in the Marshal stream. I’d like to see it handle
floats the same way that Array#pack does:
Marshal.dump(400.53).length - 3 == 22
(and is slower, to boot)
I want to make sure, though, that this would be an acceptable patch.
I can’t think why it would be OK for Array#pack to work this way and
not Marshal, but is there any particular reason why it can’t be done?
Obviously it would break backwards compatability with older Marshal
dumps, but I don’t think they’re often used for long-term storage,
are they?
Marshal.dump(400.53).length - 3 == 22
(and is slower, to boot)
I want to make sure, though, that this would be an acceptable patch. I can’t
think why it would be OK for Array#pack to work this way and not Marshal, but
is there any particular reason why it can’t be done? Obviously it would break
backwards compatability with older Marshal dumps, but I don’t think they’re
often used for long-term storage, are they?
– Brian P.
i’ve never tried to use marshaled data across a big and little endian
machine
but this would break it. consider drb: if you had a mac and a linux
box
talking on the wire you might see
The ‘g’ and ‘G’ flags for Array#pack/String#unpack are in network
byte order, so they’re in a platform-independent format, as far as I
know. I actually tested this by Packing a couple thousand floats on
my Mac, sending them in UDP packets and unpacking them on my AMD64
desktop, they all came across correctly.
In message “Re: Marshal’s handling of floats”
on Sun, 9 Jul 2006 13:31:59 +0900, Brian P. [email protected] writes:
|I was thinking about writing a patch to modify how Marshal handles
|floats, right now it dumps them using sprintf(3) and stores the
|resulting string in the Marshal stream. I’d like to see it handle
|floats the same way that Array#pack does:
|
|[400.53].pack(‘g’).length == 4
|[400.53].pack(‘G’).length == 8
|
|while
|
|Marshal.dump(400.53).length - 3 == 22
|(and is slower, to boot)
|
|I want to make sure, though, that this would be an acceptable patch.
There are issues:
pack(‘g’) would not work on non-IEEE floating machines.
changing marshal format in incompatible way causes a lot of
troubles, so that it should be avoided if possible.
I think we can merge it for 1.9 (if we address IEEE754 issue).
pack(‘g’) would not work on non-IEEE floating machines.
changing marshal format in incompatible way causes a lot of
troubles, so that it should be avoided if possible.
I think we can merge it for 1.9 (if we address IEEE754 issue).
matz.
Yes, that’s a biggie. I didn’t realize that ruby compiled on non-IEEE
machines, but it makes sense now that I think about it. I think this
is out of my league, I suppose it would require integrating a
floating-point emulation library into ruby on such platforms, and
having that library handle the packing/marshaling, or even using that
library to back all Float objects on such platforms. I think that for
my purposes it makes more sense to just write a separate Marshal-type
extension library, since I only plan to target IA32, IA64 and Apple
G4/G5.
my purposes it makes more sense to just write a separate Marshal-type
Actually, I’m not terribly concerned with precision, but rather with
speed and size. The application I’m building needs to communicate
data over a wireless network every 100 milliseconds, and it needs to
use as little bandwidth as possible. I was considering just using the
Marshal methods, but Marshal’s floating-point representation killed
that idea, though for other Ruby built-in data types it seems to be
quite good, even truncating small integers to bytes and shorts. I’ve
decided that I only need a couple decimal places of accuracy, though,
so I’m just going to multiply each float by 100 and then convert it
to an int.
Out of curiousity, what do you mean by ‘represent floating points in
hexadecimal format’? I’ve never heard of that before.
At Mon, 10 Jul 2006 09:12:32 +0900,
Brian P. wrote in [ruby-talk:201018]:
Yes, that’s a biggie. I didn’t realize that ruby compiled on non-IEEE
machines, but it makes sense now that I think about it. I think this
is out of my league, I suppose it would require integrating a
floating-point emulation library into ruby on such platforms, and
having that library handle the packing/marshaling, or even using that
library to back all Float objects on such platforms. I think that for
my purposes it makes more sense to just write a separate Marshal-type
extension library, since I only plan to target IA32, IA64 and Apple
G4/G5.
What’s the reason of your proposal?
If it is for precision issue, rather I’d suppose to represent
floating points in hexadecimal format.
Actually, I’m not terribly concerned with precision, but rather with
speed and size. The application I’m building needs to communicate data
over a wireless network every 100 milliseconds, and it needs to use as
little bandwidth as possible. I was considering just using the Marshal
methods, but Marshal’s floating-point representation killed that idea,
though for other Ruby built-in data types it seems to be quite good,
even truncating small integers to bytes and shorts. I’ve decided that I
only need a couple decimal places of accuracy, though, so I’m just going
to multiply each float by 100 and then convert it to an int.
If you don’t need to send arbitrary (marshallable) ruby objects, but
only fairly well-defined struct-like packets, the you might find this
helpful. It’s built on top of pack/unpack, but it has a “dsl” flavor and
makes it easier to work with odd-length bit fields, for example:
(I wrote this for purposes like yours: to send data over a constrained
wireless network, and also to communicate easily with non-ruby code, so
Marshal was not an option.)
pack(‘g’) would not work on non-IEEE floating machines.
changing marshal format in incompatible way causes a lot of
troubles, so that it should be avoided if possible.
I think we can merge it for 1.9 (if we address IEEE754 issue).
matz.
A couple of questions:
How does XML handle floats?
How does YAML handle floats?
What is the format of a float when dumped using Marshal?
My recommendation would be to dump floats as the hexadecimal
representation of IEEE 64-bit formatted numbers. This is “almost
universal” and occupies only 16 bytes. The alternative, dumping them in
decimal in some “scientific” notation, takes more bytes and loses small,
but noticeable, accuracy. Moreover, it does not capture IEEE’s “Inf” and
“NaN” values, which are very much part of the semantics and syntax of
modern numeric processing.
Non-IEEE architectures are very much the exception rather than the rule,
and they can be expected to dump IEEE hex and read IEEE hex as a penalty
for not adopting the standard.
The ‘g’ and ‘G’ flags for Array#pack/String#unpack are in network
byte order, so they’re in a platform-independent format, as far as I
know. I actually tested this by Packing a couple thousand floats on
my Mac, sending them in UDP packets and unpacking them on my AMD64
desktop, they all came across correctly.
Yeah, but I still see serious drawbacks of your proposal:
it will break existing code (YAML data)
it’s not human readable which is one of the key advantages (YAML
files
seem to be frequently written by hand as a nice user interface to config
data)
it’s not portable and it might not be compliant with YAML spec
In sum all these are serious showstoppers for a general change of YAML
behavior.
Another solution - if you want to make use of this: AFAIK you can
customize
YAMLfication of classes. If that was true, you can pretty easily
perform
the conversion during to conversion to and from YAML. That would be the
way
to go IMHO.
it’s not human readable which is one of the key advantages (YAML files
seem to be frequently written by hand as a nice user interface to config
data)
it’s not portable and it might not be compliant with YAML spec
In sum all these are serious showstoppers for a general change of YAML
behavior.
I thought the OP wanted to change the behavior of Marshal, not YAML. Or
are you saying that the YAML rep’n would also have to change so that
both formats have the same lossiness?
Hey Robert, I have gone with another solution but just as an aside –
we’re talking about the Marshal class, not YAML. I definitely
wouldn’t suggest storing anything in YAML as binary data
This could be changed in 1.9, maybe keeping some backwards
compatibilty in Marshal.load for the old format.
for not adopting the standard.
I think that’s reasonable. Note that legacy machines don’t need to
implement full IEEE operations, just reading and writing.
-Jürgen
This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.
