Binaryparse: length in bytes of block definition?

Ruby
1

Hi,

is there a way to derive the size in bytes of a custom
BinaryBlocker::Blocker definition?

class Foo < BinaryBlocker::Blocker
has_one :offset, :int32
has_one :length, :int32
end

Deriving the length would yield 8.

I’ve been using bit-struct for similar tasks, but it doesn’t support
nesting class definition. However, bit-struct provides a
round_byte_length() for that purpose.

Is there an equivalent for it in binaryparse?

thanks,

  • Markus
2

Markus F. wrote:

I’ve been using bit-struct for similar tasks, but it doesn’t support
nesting class definition. However, bit-struct provides a
round_byte_length() for that purpose.

Can you describe the nesting you are trying to do? Bit-struct does have
some nesting capability, and my working copy now supports embedded
vectors (fixed length arrays of bit-structs). This is an addition in the
last week or so, but I can make it available.

Nesting example:

class NestedPart < BitStruct
unsigned :x, 5
unsigned :y, 3
char :s, 5*8
end

class Container < BitStruct
nest :n1, NestedPart, “Nest 1”
nest :n2, NestedPart, “Nest 2”
end

The more or less equivalent effect using vectors instead:

class MyEntry < BitStruct
unsigned :x, 5
unsigned :y, 3
char :s, 5*8
end

class Container < BitStruct
vector :n, MyEntry, “some entries”, :length => 2
end

3

Hi,

first, I guess I’ve to apologize in some way, because I was the OP last
week and you kindly, very quickly, posted a suggestion on pastebin and I
was completely ignorant. Actually, I was in limbo because I had so many
questions and didn’t know where to start.

Joel VanderWerf wrote:

Can you describe the nesting you are trying to do?

Currently, I’m writing a reader for the BSP file format, as described
here: Unofficial Quake 3 Map Specs

The week before, I was writing a MD3 model reader, which works already
thanks to bit-struct :slight_smile:

Back to the BSP format: since it uses some fixed arrays, and it was one
of your post which suggested binaryparse, I tried that one and succeeded
so far, except the following:

  1. it was incredible hart to get into binaryparse as you basically
    have to learn from the source, as there’s no real usable rdoc doc

  2. probably due 1), I’ve no idea how to read float values with it. I
    just don’t get it, I guess :frowning: It does not seem to have out of the box
    support, however it seems to provide some kind of support via custom
    packing/unpacking, but I can’t figure out how.

Due 2), I don’t support reading the lumps containing floats ATM. Right
now I don’t need them, but reading all data is the goal.

 char     :s,  5*8
 unsigned :x,    5
 unsigned :y,    3
 char     :s,  5*8

end

class Container < BitStruct
vector :n, MyEntry, “some entries”, :length => 2
end

Now, I wouldn’t mind going back to bit-struct for the BSP reading stuff,
as I find the documentation outstanding well! However, through
binaryparse I discovered one thing I’m really enjoying: it supports
direct reading from an IO-type object. So I can pass either a File IO or
an StringIO object directory, which is really comfortable.

Maybe bit-struct supports that in some way, but I couldn’t find
anything. Basically, with bit-struct I’ve to know how much to pass to a
BitStruct class. Binaryparse, in contrast, uses the given structure
information to know itself how much to consume.

class Header < BinaryBlocker::Blocker
has_one …
has_one …
end

class Datadir < BinaryBlocker::Blocker
…
…
end

io = File.new(“data”)

Header.new io
Datadir.new io

That’s really nice compared to

Header.new io.read(Header.round_byte_length)
Datadir.new io.read(Datadir.round_byte_length)

But I can see that there are different philosophies at work, given that
it’s named round_byte_length, because reading single bits wouldn’t be
really possible that way, therefore the “rest” reader …

About your upcoming addition and your example: how would a simple
integer array look like?

C-struct example:

typedef struct {
float normal[3];
float dist;
} dplane_t;

Ruby (?):

class MyFloat < BitStruct
float :value, 32
end

class Plane < BitStruct
vector :normal, MyFloat, :length => 3
float :dist, 32
end

This wold mean I need to access plane.normal[0].value I guess … having
not to need the additional .value would be nice, e.g.

class Plane < BitStruct
vector :normal, :length => 3 do
float 32
end
float :dist, 32
end

Would multiple dimension be supported, too? Like C-style float[2][2] ?

class Plane < BitStruct
vector :normal, :length => 3 do
vector :length => 3 od
float 32
end
end
float :dist, 32
end

Ok, I think it’s getting out of control now :wink:

thanks

  • Markus
4

Hello Joel,

I’ve now rewritten my code to take advantage of you latest 0.12 changes
and have to say that it was a very pleasing experience. Everything
worked out of the box and the addition of the IO object was a real time
safer in moving from binaryparser to be bit-struct-only.

During this rewrite-session I made some notes how my further bit-struct
experience went. If you’re annoyed by them, simply ignore 'em :wink:

  • Allow block also for “nest” ?

It may sound redundant in the first place, but when it’s just for
providing more structure without requiring an extra class definition, it
could be useful:

class Foo < BitStruct
nest :coord do
signed :x, 8
signed :y, 8
end
end

Foo.new.coord.x

  • Is explicit :length for vector necessary?

    class Tag < BitStruct
    signed :flags, 32
    vector :axis, Vec, :length => 3
    end

compared to

class Tag < BitStruct
signed :flags, 32
vector :axis, Vec, 3
end

Or am I missing any ambiguities because of the simple cases?

  • Allow length not only in bits, but also bytes?

In my cases I’ve never encountered a situation where I needed a member
of a structure to represent anything which is not on a byte boundary.

All my signed and unsigned int have to read “32” (32bit, 4bytes) all
over the place. For strings I’m writing “64*8” for string with 64
characters.

How about a new default_option :granularity which is “1” by default.
When set to “8”, every specified length (except for vector …) is
multiplied by that factor before used?

  • Why “vector” and not just “array”?

I’m getting pretty confused all the time due the term vector :slight_smile: I’m
always thinking about graphical vectors, etc. For, me arrays would also
better describe the environment how the data is used (array of ints …
vector of ints … ?).

Joel VanderWerf wrote:

class MyFloat < BitStruct
Plane.new.normal[2].value
As you said, looks like one layer to much. Thanks for your block-support
to vector, could this maybe be detected for simple cases?

class Plane < BitStruct
vector :normal, :length => 3 do
signed 32
end
end

I.e., having only one element and deliberately skip the naming to be
prepared for

Plane.new.normal[2]

As you can see, the reader method is fine, but the writer has to be
broken down into a sequence of writes to each nested chunk… yuck.

The reason it doesn’t allow simply

md.row[2].col[7].x = 1.23
[…]

Any other ideas…?

Unfortunately not at this time. I’m using bit-struct solely for reading
binary data ATM and not creating them.

thanks,

  • Markus
5

Markus F. wrote:

Now, I wouldn’t mind going back to bit-struct for the BSP reading stuff,
as I find the documentation outstanding well! However, through
binaryparse I discovered one thing I’m really enjoying: it supports
direct reading from an IO-type object. So I can pass either a File IO or
an StringIO object directory, which is really comfortable.

That’s a useful addition. I’ve added that to bit-struct 0.12 now (along
with the vector stuff), you can get it at:

http://redshift.sourceforge.net/bit-struct

Ruby (?):

class MyFloat < BitStruct
float :value, 32
end

class Plane < BitStruct
vector :normal, MyFloat, :length => 3
float :dist, 32
end

and then

Plane.new.normal[2].value

works, but yes it seems like one layer too many.

Would multiple dimension be supported, too? Like C-style float[2][2] ?

class Plane < BitStruct
vector :normal, :length => 3 do
vector :length => 3 od
float 32
end
end
float :dist, 32
end

Yes, that’s supported now:

class MD < BitStruct
vector :row, :length => 3 do
vector :col, :length => 10 do
float :x, 32
end
end
end

md = MD.new
rows = md.row
row = rows[2]
cols = row.col
col = cols[7]
col.x = 1.23
cols[7] = col
row.col = cols
rows[2] = row
md.row = rows

p md
p md.row[2].col[7].x

As you can see, the reader method is fine, but the writer has to be
broken down into a sequence of writes to each nested chunk… yuck.

The reason it doesn’t allow simply

md.row[2].col[7].x = 1.23

is that all those intermediate objects have to be something, either
separate (copied) bit-structs, or proxies that delegate back to the
original. If the latter, there is IMO the possibility of confusion
as these proxies propagate around and the original object changes
unexpectedly.

Any other ideas…?