Forum: Ruby Unpacking signed shorts and integers with specified endianne

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
06e8ca56db1b93248c94037070833d91?d=identicon&s=25 Gavin K. (phrogz)
on 2007-06-19 00:37
(Received via mailing list)
I'm deciphering a byte-based binary file structure which can be stored
in either big or little endian. (The first byte of the file describes
the format.) Among others, some of the fields are UINT32 (unsigned 4-
byte integers), and some are INT32 (signed 4-byte integers).

I know about BitStruct, but was trying to roll my own solution using
String.unpack. (The file format has nested and repeating sections in
it.) Given that I know exactly how many bytes I want for each field
and whether the file is big or little endian, I thought it would be
easy to pick a specific unpack character for each field. However,
reading through the String.unpack docs, I can't find something that
corresponds to signed 2-byte and 4-byte integers for a given
endianness.

Here's what I see (ASCII art ahead):
      |     signed     |    unsigned    |
bytes |  big  | little |  big  | little |
------+-------+--------+-------+--------+
    1 |       c        |       C        |
    2 |   ?   |    ?   |   n   |    v   |
    4 |   ?   |    ?   |   N   |    V   |

1) I see "l" (lowercase L) which is 4 bytes treated as a signed
integer...but in 'native' endian order. Does String.unpack not provide
a way to unpack a 4-byte signed integer with a specified endianness?

2) I see "s" which is 2 bytes treated as a signed integer...but in
'native' endian order. Does String.unpack not provide a way to unpack
a 2-byte signed integer with a specified endianness?

(The file format doesn't actually use any signed 2-byte integers, but
I wanted to include them for completeness.)
06e8ca56db1b93248c94037070833d91?d=identicon&s=25 Gavin K. (phrogz)
on 2007-06-19 20:11
(Received via mailing list)
On Jun 18, 4:32 pm, Phrogz <g...@refinery.com> wrote:
> 1) I see "l" (lowercase L) which is 4 bytes treated as a signed
> integer...but in 'native' endian order. Does String.unpack not provide
> a way to unpack a 4-byte signed integer with a specified endianness?
>
> 2) I see "s" which is 2 bytes treated as a signed integer...but in
> 'native' endian order. Does String.unpack not provide a way to unpack
> a 2-byte signed integer with a specified endianness?

Alright, let's try asking this question a different way. Given a
binary file spec like the following, where the first byte tells you
whether to interpret the rest of the file as little- or big-endian,
and given the presence of a signed integer in the file, how would you
write a parser for this?

endian     UINT8    (1==little, 0==big)
job_count  UINT8    # of repeated binary sections following this
(jobs)     JOB*

    Each JOB section is:
    foo    UINT8
    bar    UINT16
    jim    UINT32
    jam    INT32
    jill   UINT8

gib_mark   0xdead   # 2 bytes
gib_count  UINT16   # of repeated binary sections following this
(gibs)     GIB+

    Each GIB section is:
    gob    8 chars
    blurb  UINT8


Would you invent a new character for String.unpack, split the string
around it, and use knowledge of the native endianness of the platform
you're running on to decide whether to pull out those 4 bytes
independently, reverse them, and then unpack the result as a signed
integer?

Is there some sweet trick you could do after extracting 4 bytes as an
integer to switch the implied interpreted endianness?

Would you patch String.unpack in C to add options for specific-endian
signed shorts and integers?

Can you easily do the above (including the repeating sub-binary
sections) with BitStruct?
Daniel Berger (Guest)
on 2007-06-19 20:20
(Received via mailing list)
On Jun 18, 4:35 pm, Phrogz <g...@refinery.com> wrote:

<snip>

> 1) I see "l" (lowercase L) which is 4 bytes treated as a signed
> integer...but in 'native' endian order. Does String.unpack not provide
> a way to unpack a 4-byte signed integer with a specified endianness?
>
> 2) I see "s" which is 2 bytes treated as a signed integer...but in
> 'native' endian order. Does String.unpack not provide a way to unpack
> a 2-byte signed integer with a specified endianness?

See the 'N', 'n', 'V' and 'v' directives. There are equivalent
directives for floats as well - 'E', 'e', 'G' and 'g'.

Regards,

Dan
Mark Day (Guest)
on 2007-06-19 20:39
(Received via mailing list)
On Jun 19, 2007, at 11:19 AM, Daniel Berger wrote:

> directives for floats as well - 'E', 'e', 'G' and 'g'.
Those handle endianness, but not signed values.  I suppose you could
unpack as unsigned, then manually test for the sign bit being set and
correct the value.  Even uglier, you could unpack as unsigned with
desired endianness, repack as unsigned in native order, then unpack as
signed in native order.

-Mark
Joel VanderWerf (Guest)
on 2007-06-24 06:43
(Received via mailing list)
Mark Day wrote:
>> See the 'N', 'n', 'V' and 'v' directives. There are equivalent
>> directives for floats as well - 'E', 'e', 'G' and 'g'.
>
> Those handle endianness, but not signed values.  I suppose you could
> unpack as unsigned, then manually test for the sign bit being set and
> correct the value.  Even uglier, you could unpack as unsigned with
> desired endianness, repack as unsigned in native order, then unpack as
> signed in native order.

What bit-struct does in these cases is the first kind of ugly:

   # Let's say we start with a negative number packed in
   # 16 bits, big-endian:
   x = -123
   s = [x].pack("n")

   # Note that the sign is not packed with the number. It packs to the
   # same chars as 2**16 + x

   bits = 16
   max_unsigned = 2 ** bits
   max_signed = 2 ** (bits - 1)
   to_signed = proc { |n| (n >= max_signed) ? n - max_unsigned : n }

   puts to_signed[s.unpack("n").first] # ==> -123

(This has come up a few times on the list -- search for "to_signed", for
example.)

It's still a hack, though, and I'd like to see Gavin's RCR go through,
if the naming issues can be resolved.
This topic is locked and can not be replied to.