How can I parse binary files?

fabvit · July 17, 2006, 12:54pm

I’ve the need to parse a binary file with the following structure:
How can I accomplish this in Ruby?

Header (36 bytes):

Version (4 byte unsigned integer) currently 1
UIDValidity (4 byte unsigned integer)
UIDNext (4 byte unsigned integer)
Last Write Counter (4 byte unsigned integer)
the rest unused

Message data (36 bytes per message):

Filename (23 bytes including terminating NUL character)
Flags (1 byte bitmask)
UID (4 byte unsigned integer)
Message size (4 byte unsigned integer)
Date (4 byte time_t value)

Flags mask is 1:Recent, 2:Draft, 4:Deleted, 8:Flagged, 16:Answered,
32:Seen.

fabvit · July 17, 2006, 1:06pm

On 17/07/06, Fabio V. [email protected] wrote:

I’ve the need to parse a binary file with the following structure:
How can I accomplish this in Ruby?

String#unpack.

fabvit · July 17, 2006, 3:14pm

Fabio V. [email protected] writes:

I’ve the need to parse a binary file with the following structure:
How can I accomplish this in Ruby?

In addition to parsing this yourself using ruby’s String#unpack
method, you should also look at the BitStruct extension available at
http://redshift.sourceforge.net/bit-struct/

(And found via http://raa.ruby-lang.org/ by doing a search on
“binary”)

Am I the only one who thinks that ruby-forum.com should include in a
prominent place pointers to standard ruby documentation, and to the
Ruby Application Archive? I don’t object to people posting to the
list via the web form at ruby-forum.com, but I think that a prominent
display of common sources of information would help everyone.

fabvit · July 17, 2006, 6:08pm

On Tue, 18 Jul 2006, Fabio V. wrote:

I’ve found bit-struct very intresting, anyway I cannot figure how to
unsigned :uid_Validity, 4, “UIDValidity”
unsigned :uid_next, 4, “UIDNext”
unsigned :last_write_counter, 4, “LastWriteCounter”
rest :unused, “Unused”
end

mrk = MRK.new

And now: how to populate the mrk instance just created from the imap.mrk
binary file?

without even looking at the docs i’d guess you could do

data = IO.read ‘your.data’

mrk = MRK.new data

and, indeed, this seems to work:

harp:~ > cat a.rb
require ‘bit-struct’

class C < BitStruct
unsigned :a, 16
unsigned :b, 16
unsigned :c, 16
end

c = C.new ‘a’ => 42

p c

buf = c.to_s

p buf

c = C.new buf

p c.a

harp:~ > ruby a.rb
#
“\000*\000\000\000\000”
42

incidentally, you are probably going to want

class MRK < BitStruct
unsigned :version, 32, “Version”
unsigned :uid_Validity, 32, “UIDValidity”
unsigned :uid_next, 32, “UIDNext”
unsigned :last_write_counter, 32, “LastWriteCounter”
rest :unused, “Unused”
end

the field size declares the number of bits not bytes.

bit-struct

regards.

-a

fabvit · July 17, 2006, 5:45pm

Daniel M. wrote:

Fabio V. [email protected] writes:

I’ve the need to parse a binary file with the following structure:
How can I accomplish this in Ruby?

In addition to parsing this yourself using ruby’s String#unpack
method, you should also look at the BitStruct extension available at
BitStruct

I’ve found bit-struct very intresting, anyway I cannot figure how to
load a binary file in a newly created bit-structure.
Any help appreciated.

Say I’ve an imap.mrk binary file,
I’ve defined class MRK as follows:

require ‘bit-struct’

class MRK < BitStruct
unsigned :version, 4, “Version”
unsigned :uid_Validity, 4, “UIDValidity”
unsigned :uid_next, 4, “UIDNext”
unsigned :last_write_counter, 4, “LastWriteCounter”
rest :unused, “Unused”
end

mrk = MRK.new

And now: how to populate the mrk instance just created from the imap.mrk
binary file?

Thank you

fabvit · July 17, 2006, 10:39pm

[email protected] wrote:

class MRK < BitStruct
binary file?
[snip]
This looks like a nice way.
I just wanted to show that in such a simple case unpack isn’t that ugly,
too.

open(‘file.bin’, ‘rb’).do |f|
version, uidValid, uidNext, lwCounter = f.read(36).unpack(‘IIII’)
name, flags, uid, size, date = f.read(36).unpack(‘Z23CIII’)

#do something
end

This is of course untested because i don’t have such a file, but i hope
the idea is clear.

cheers

Simon

fabvit · July 17, 2006, 10:49pm

Fabio V. [email protected] writes:

And now: how to populate the mrk instance just created from the imap.mrk
binary file?

First off, the other message’s advice about your field sizes should be
taken (you want to use “32”, not “4”). Also, you almost certainly
want to add :endian => :native to your structure. Finally, you’ll
want to adjust the bit_length method of your MRKHeader class since it
won’t construct the appropriate length just from the field info.

class MRKHeader < BitStruct
unsigned :version, 32, “Version”, :endian => :native
unsigned :uid_Validity, 32, “UIDValidity”, :endian => :native
unsigned :uid_next, 32, “UIDNext”, :endian => :native
unsigned :last_write_counter, 32, “LastWriteCounter”, :endian =>
:native
rest :unused, “Unused”
def MRKHeader.bit_length
super
36*8
end
end

Okay, now let’s assume that you also define the per-message structure
using BitStruct as MRKMessage. (For the message code, you don’t need
to redefine bit_length since it can be computed straight from the
fields. Do however use the endianness option on all the integers)

Then:

File.open(“imap.mrk”) {|f|
head_string = f.read(MRKHeader.round_byte_length)
raise “No header!” unless head_string
mrk_header = MRKHeader.new(head_string)
puts mrk_header.inspect
while msg_string = f.read(MRKMessage.round_byte_length) do
puts MRKMessage.new(msg_string)
end
}

fabvit · July 17, 2006, 11:08pm

Daniel M. [email protected] writes:

}
I forgot to open the file in binary mode, and forgot an inspect call.
I should have said:

File.open(“imap.mrk”, “rb”) {|f|
head_string = f.read(MRKHeader.round_byte_length)
raise “No header!” unless head_string
mrk_header = MRKHeader.new(head_string)
puts mrk_header.inspect
while msg_string = f.read(MRKMessage.round_byte_length) do
puts MRKMessage.new(msg_string).inspect
end
}

fabvit · July 18, 2006, 8:20am

Daniel M. wrote:

Daniel M. [email protected] writes:

require ‘bit-struct’

class MRKHeader < BitStruct
unsigned :version, 32, “Version”, :endian => :native
unsigned :uid_Validity, 32, “UIDValidity”, :endian => :native
unsigned :uid_next, 32, “UIDNext”, :endian => :native
unsigned :last_write_counter, 32, “LastWriteCounter”, :endian =>
:native
rest :unused, “Unused”
def MRKHeader.bit_length
super
36*8
end
end

File.open(“imap.mrk”, “rb”) {|f|
head_string = f.read(MRKHeader.round_byte_length)
raise “No header!” unless head_string
mrk_header = MRKHeader.new(head_string)
puts mrk_header.inspect
while msg_string = f.read(MRKMessage.round_byte_length) do
puts MRKMessage.new(msg_string).inspect
end
}

Now an error is raised:

ruby b.rb
#
b.rb:19: uninitialized constant MRKMessage (NameError)
from b.rb:14
Exit code: 1

Also the problem is that there is to process the Message data structure:
how can I accomplish this?
Thank you all very much for the help!

fabvit · July 18, 2006, 8:38am

Fabio V. wrote:

This is the structure of class MRKMessage:

Message data (36 bytes per message):

Filename (23 bytes including terminating NUL character)
Flags (1 byte bitmask)
UID (4 byte unsigned integer)
Message size (4 byte unsigned integer)
Date (4 byte time_t value)

Flags mask is 1:Recent, 2:Draft, 4:Deleted, 8:Flagged, 16:Answered,
32:Seen.

Now 3 major questions:

Q 1: what type must I declare for Filename in the class MRKMessage?

Q 2: what type must I declare for Flags in the class MRKMessage?

Q 3: what type must I declare for Date in the class MRKMessage?

…and 2 minor ones :-))

Q 4: How to decode Flags?

Q 5: How to decode Date?

BIG BIG THANKS TO ALL!

require ‘bit-struct’
class MRKHeader < BitStruct
unsigned :version, 32, “Version”, :endian => :native
unsigned :uid_Validity, 32, “UIDValidity”, :endian => :native
unsigned :uid_next, 32, “UIDNext”, :endian => :native
unsigned :last_write_counter, 32, “LastWriteCounter”, :endian =>
:native
rest :unused, “Unused”
def MRKHeader.bit_length
super
36*8
end
end

class MRKMessage < BitStruct
char :filename, 184, “FileName”, :endian => :native
unsigned :flags, 8, “Flags”, :endian => :native
unsigned :uid, 32, “UID”, :endian => :native
unsigned :msg_size, 32, “MsgSize”, :endian => :native
unsigned :date, 32, “Date”, :endian => :native
def MRKMessage.bit_length
super
36*8
end
end

File.open(“imap.mrk”, “rb”) {|f|
head_string = f.read(MRKHeader.round_byte_length)
raise “No header!” unless head_string
mrk_header = MRKHeader.new(head_string)
puts mrk_header.inspect
while msg_string = f.read(MRKMessage.round_byte_length) do
puts MRKMessage.new(msg_string).inspect
end
}

This now generates:

ruby b.rb
#
#<MRKMessage
filename="\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\r\nmd5",
flags=48, uid=808464432, msg_size=942814256, date=1936535094>
#<MRKMessage
filename=“g\000\000\000\000\000\00006\020\000\000\374P\000\000k\353\246Cmd5”,
flags=48, uid=808464432, msg_size=858993712, date=1936535091>
#<MRKMessage filename=“g\000\000\000\000\000\000
e\020\000\000\334\226\003\000X\373\253Cmd5”, flags=48, uid=808464432,
msg_size=858993712, date=1936535092>

fabvit · July 18, 2006, 5:12pm

Fabio V. [email protected] writes:

Now 3 major questions:

Q 1: what type must I declare for Filename in the class MRKMessage?

Okay, first off I apologize but I lead you astray. Apparently it’s
not enough to override bit_length in your subclass. When you read the
file, you’re not getting the stuff lined up properly. Therefore I’ve
decided to make up for it by finishing the rest of your code for you.

Note that now I override round_byte_length instead, and we get:

require ‘bit-struct’
class MRKHeader < BitStruct
unsigned :version, 32, “Version”, :endian => :native
unsigned :uid_Validity, 32, “UIDValidity”, :endian => :native
unsigned :uid_next, 32, “UIDNext”, :endian => :native
unsigned :last_write_counter, 32, “LastWriteCounter”, :endian =>
:native
rest :unused, “Unused”
# Override so that it gets padded properly
def MRKHeader.round_byte_length
super
36
end
end

Ideally, I’d construct some sort of “flags” bit-struct field

Or define a boolean field type and make this a series of boolean

fields.

However, for now we can deal with a series of 0s and 1s

class MRKMessageFlags < BitStruct
unsigned :flagUnused, 2, “Unused”
unsigned :flagSeen, 1, “Seen”
unsigned :flagAnswered, 1, “Answered”
unsigned :flagFlagged, 1, “Flagged”
unsigned :flagDeleted, 1, “Deleted”
unsigned :flagDraft, 1, “Draft”
unsigned :flagRecent, 1, “Recent”
end

class MRKMessage < BitStruct

Note “text” for nul-terminated strings

text :filename, 23*8, “FileName”, :endian => :native
nest :flags, MRKMessageFlags, “Flags”
unsigned :uid, 32, “UID”, :endian => :native
unsigned :msg_size, 32, “MsgSize”, :endian => :native
unsigned :date, 32, “Date”, :endian => :native

Now we futz with the way that date is set and gotten.

we rename the existing date field to __date, and

then we supply our own meaning for “date” that does

translation into and out of seconds-since-1970

Again, the ideal solution would be to define a new bit-struct

field type that did this stuff itself.

alias_method :__date=, :date=
alias_method :__date, :date
def date=(time)
self.__date= time.to_i
end
def date
Time.at(self.__date)
end

we don’t need to override the length computation here

end

File.open(“imap.mrk”, “rb”) {|f|
head_string = f.read(MRKHeader.round_byte_length)
raise “No header!” unless head_string
mrk_header = MRKHeader.new(head_string)
puts mrk_header.inspect
while msg_string = f.read(MRKMessage.round_byte_length) do
puts MRKMessage.new(msg_string).inspect
end
}

END

This produces (on the first bit from your file):

#
#<MRKMessage filename=“md50000004286.msg”, flags=#, uid=4150, msg_size=20732,
date=Mon Dec 19 12:18:35 Eastern Standard Time 2005>

This is more what you expected, right?

fabvit · July 18, 2006, 5:19pm

Fabio V. wrote:
…

flags=48, uid=808464432, msg_size=858993712, date=1936535091>
#<MRKMessage filename=“g\000\000\000\000\000\000
e\020\000\000\334\226\003\000X\373\253Cmd5”, flags=48, uid=808464432,
msg_size=858993712, date=1936535092>
…

Looks like you need to investigate #unpack.

In any case, Ruby F.s has a BinaryReader mixin
(http://facets.rubyforge.org/api/more/classes/BinaryReader.html) that
does the reading and upacking for you. Just mix it into File (or a
subclass of File) and you should be good to go…

Cheers
Chris

fabvit · July 18, 2006, 5:27pm

Daniel M. wrote:
Martin Thank you very much: you solved my problem like a charm!

fabvit · July 18, 2006, 9:26am

Fabio V. wrote:
This is my binary file IMAP.RMK zipped and UUEncoded:

=
= Part 001 of 001 of file IMAP.zip
=

begin 666 IMAP.zip
M4$L#!!0(`)9(\31/<?*PE!8``%!&```(24U!4"Y-4DMTVWEXCM?6
M!O#U2B0Q$P012<Q.8G@S)]14A):JS%T0$TQQ%1*A%)333'55/,8&D3-LW*H MF>.T2A7%,12MK^:AU'<_7UW7N>^T7Z_+.7_X7>^S]UYKK[WV\[Y<9M8UZ6[= M?7YF4RO8W_R7+W?OSE'N__LO,CPVNFKO=W^MW=$&S%\W->MY96>^_)B(B MXK\FNO,A5D>UN;W3#&19+K"=*[BLIN^:\1$D>D&,ZE?01L7)B;23:873/JK
MC;A\7HQ861ZP]3+:S8KF^'Q](797-SL2<Q78F+(#(!)R6TV<LX&-E'\.:DP M'^4R^S%KHQB>UU85Y!9TD$UM,[!:3!S#D?>VP7P^,9!M-P3R$;KB::GS7&
MF1?&W’7\3C%Q9,;#.YK-MYO%YM8GM<<F*]+FOW0<K\8?M9<9^Y?</V)8F)
MHS&[5!X^'I9R6^8;‘DP$S,+]9^^=L(MT<]U4PR[$^17,>$$-Q=Z^&Z5WM;'HVP[F:"1.']3FX30W-W;T&9B;,&SO4\-S7.C'-9U8KF^&8KH.93-OIQI>
MGRR8$/RI>U(-K]73FY@[W;P.<@FC’-^(TPIF’.EU(33LS;!#,MCEA"E)I;,
M#IB;F/NC=^P">=8[()Y/M(L:,PA,3R>W3!5#G=KXB)H/$$‘X:IY&4V-:H
M&([7$9CJ?V,X5X"),LS&8X7L=@$F$6J8ED<P:F7)V"=N+P<3$<T^]A#G9U
M6>QQ,5P3W)=@@E[EL@EE3HCAS[D,XWXOK[4+4</[0K,Q%27^7J?%$/Q"KX*
MDS(IQ>H%BHGF6%R’2:MLMKBW&H[%#2</8\PZ+E7#Z_PSS/TG?M3HGA-;P%
M\C;[.Q269]HGOOM?PW.^-HN#9’VDCOT*$UH(;KF,)X9S]2[,@68N*[%;
MC-2Z%S;#IM%^I]F$\?KG!O/Z5'.[*GW]V)X/'[email protected]?9";**DCN6%24$= M&^.OAN.5#V8']LYOI=1PO/+#O+O!976KJ^%8%((9_9VG74Q0P['PA3GZ;R_[ MOHD:CD5AF)LXOYHGJN&Y%X5)7Y[32B6+X?H37!RFS3+4YQIGQ=!^=Y>B<Z M]^ZIAN/E#[/QLI?5ZZ^&YQ44ZYG@MW*%!/.GU/.64/,??J%<V)X#<O#G*^&
M/B’W#V+X615@>N,\G7Q.C-2ZRC#?H?X\Z7R>313’O3;,0.1\2I&?Q’_5M_) M=WQ.6@TUG*OQ,,D83_^::BA>[H8P7P?DM2O)8J(Y5QO!K,/Y-7>@&EH?]ULP M47-MI]3P^-I#C/S%T_+Z’U9#.=/"YC2,\PV]1’#>]G=$N8^^H1ZP]3P&?<V
M3![T?I5;7A’#9TIK)=P5F;^J(9K^#O.‘BR+/+NJAN>>!+,2>IY5PWG1AN8
M8ZF>=O2!&HY%>Y@HG,OS<^'32SGQGLPK89Z6HM0-9QC[/<0]3(%I,’,^K
M!\P&Y$][GYMB…=[PE3Z/U$334\KUXP.Z?DM!ZCV$1+'_413,-).:W.Z)_% M<&X,ABE6#'7A2S6<&T-@1N1T66BFF##.U5286<.1/]O4<&_S*<S>2#/_K%MB M^%FC8)XD(V:7Q4AO,P9F4S^SBWENB^%8C(7I$>2RAV%JN/Y\!K/8QZS:,S4< MB_$PZZ=Y6,D7:K@^3X1YMK>M'0R^(X9S;#),?N3\R!Q$;R_TF%RXT[T+)OA
M?G4F3&:TV5L]?V’#/9)[(.K^)X3$O<F(ZP:[email protected]<4P*?B<Q^W5
M+R6P/AB#S[N(":<V,I3%HCESWJJ(;O#LM@?FUO%CA7#>?*IC_H%?8->T>
MFV@^O[)@#F-;YY7PW%?#U.[HEG\936\/AMA@K'.+R[>%\/KLPGF=I^\YOL_ M:GA]-L/T0MQSEGS1NKJ%A@OG’&+0M5PSF^%&8_Q),:IX9S?!A.2XFFCJSX4
MP[5W.\R#$CA3LAE>GQTPZXK@.ZLAF.Q"V94>=QC>XB1ONX3$%LRF9C\3P
MYWP#D^&)6-Q[+(9C<<C9IS!=GZGANG$$QJ/GE7]“9LXSL.C,.<^KQ1#:_S M,9B*..,2UCT5P^M\'";J;PRO\PF85G]C.'_^#;/T’/Q/”\OH5ICAK5X0R;
M&.EIOW.>5;&T=%YG8+9BG_Z4S7LSL-L_@8Y=.*Y&*[A%V"&^N->5/AW M,3SFBSK*R9M<5PGQE!:8-^H1[E5^(X?%<@ZE_S<.2WU3#:W@#YH;+[+/N
M8L+9.;4#>1SESYBN#]T/X1I,0(Y/^ZE&#Y/’,P><4VZ2&>Y(G,/GC,;G
M8OC]F/LE3)?YV!NG_Q##Z_/R7GD:O’;:GA_&7K,BYW,KMX5(WUF9@%N.^L MW96C/AGI,PO"Q*//?'11#>=/(9BU'F8>=]7POO"%2<:SEE?P$,,Q]8-Y7-IL M__N>8GB_%_/]\WW"?R:HX9@6AVF.'.LP40WG80F8[OB<BVJD'I:":03S[:2< M8CCN@3=<V^Q&^EJ…Y!,/VH:=$1/+ZUP.IA7.@EW’O<3P.I>'V8^[YY<G
MU7"N5H"9BS&O.J6&QUP)YA;6.>?:C@6(3"W2YF=]O)F([7.#>,=@GPMXB.&
MUS,)A/Y,[29&L[5<)B=EYO8LRYJ>#R1,'?K(::=<Y&)E5H7^SJFW7_)(X;7 M,XF\J]&^M4W87J&U[>0_GG%+P:.#%MD6A/J^03P_.A\G;#W]JJ>%^K!%,
M)<3]Z’MJN]+@)F(WOCD/#6<\XV=/8AX[5JHAO=[$YA-."_?2E&WA\FPC3N
MX&->66HX?YH[STM<YW(+X;GU1+&R;SGAK>RZU@LFYYV0$K((;W:6N8A[A_ M5?]133’-FF&>XI05X%Q?"^:.,\"^M<PEL-Q[0M3SZEHFYU/ZOP]SV?3
M]/!";0^?PA3!/7Y\1@U?’?HN.+6!3\V)=-%'^.\][B$G(UHG(1,9QC*3!Q MZ"6>9S,<B[XPL]/,?DXORH;KLWL03(L\!:S_'#\Q/.8A,(F)B7;@N1K>IZDP M8W.:>38IQD9ZT1$P7^.\V%^_N!C.L4]A4BIC;@U*B.%GC8;I&&'VLJD8J8<3 M8)0HSZ9X"^&[P438=K@7GEIBACIM>;![(DUFY,<2"9.ZL]\F![%7/9@AH^
MWQ?_&MD#FMZ7@VO\Q*8?>7,DO8%B>&Y+X5I_0Q[]8:WE_+8-8$XLZ3S7^ M+W>>A?P9>TH-SWT%S"F<.X->BI&:N1*F8JS+>KQ2PW//<,:#FA#I$RR&<W6- M$R^,N4VH&I[76IA>V(.><6IX7NM@KL7C3IW-\+RR8!(P]YIOBGG/;@1QHU[
M95_OTF(X%EM@\F.?!A54PV/>"M,.9G8VPV/>YN0\UJ=753’RKG(7S"O\F=Q
M#8]Y-_X^$/DSLK4:SK&]3DU/7PZ0@W/ZVN86[BC=9B@AN>U#R:M+,Z#;(;7 M^9\P3U#K?,^*D9IY"*;J;K/KR67$<$TX#'-NB-D/?=7PO([[“AL=G”@&I[7
M49@^Z$7[CE##\SH&4]OYSG>5&I[7"9@9#<UV;Q##[S?<)YU]40?[]:'O.W M,$V1A]O/EA7#/<9F#==+OO78Z-O+NX+,!8T[M4D$,S^LGF!+%<9>;5E$, MW^,NP\37PCY=KX;G?@6F1E/T+AO$R+N+JS"#0\W2MZCAN%^'*8P[XXLG:GA> M-V#V8^]$>502PWOG-LSSQCFLV6TUW/_<@7F,9P4]%R/O)7YSXH[:<JY?J!B. MUSV89^B?*PY1PSEV'V8,8I$U20W'X@&,W\=5;/%4,O@_CKLW<KDI>IO.E<+$ M<+Q\8*ZB/VP;(D:^W\D+L[H2\K5MN!@^!_/!9/I[VD<#U/#ZY(=I@GX^?*0: M7I\",(DPF=D,KT]!F)8ES8I.4L/K4PAF(FI+TF(QW*^ZB\<#T?]71,AAL=3
MN8,ZEBEO6IX//XPCU&C0FI%BN'QE(1YA?Q9UE8,OZ<-#H0I7@+Q.*:&>Y)@ M)Q8P30I$L9%WIV'.>-"W+RISD;>G8;#-/VNQRIAN,>"3/K0[/2;=5PW*-@ M]K0R^VZ!&IY['$R-*F8?)-40PS6A!LQ3]+V=9XJ1]PEOP&Q";OPX2PV/IW9A MY[VHRYY<5\/Q>A-F"7J2.Z?>8".]7\O"?^[!V)3:8CB?WX8IA7G98#6</ZUA M.J(^7TI3P^-Y!Z8=^HV3B\5P#^G^$.;7JIC[XOIDPJ2'[RS(=’#VBQ1PS6S
M&\QEY.KA;6IX7MUA9A_!O6FG&JXM/0O_><8-+]FC=2$?C"-$--'E>/%\+/Z MP^1#S>Q=50VOST>OU^?#1FHXQP;"G,V!'D"-?'>3"M,0]\$MD]7PYWP"TQ[S M&MFUH1@^+T;5,/^^C%9#>?S2)B=>UU6IY\8V>^?PLS7MXZ3@SW$NX9,"N" MO"S#KQD;^?YB/LR5HF;K@]3PO!;-/9RSE0U?!8LA’D7L2@>IH;GM0CF/(Y MUWMB>)^Z%\.X@SQM>[(:OG<O@7&^2VO=1PW':RE,D_?-7./4<"Q6P?@$F4TJ MWEP,[YW5,G(P[$EQ$B_\27,]CH9?W4!TD$Z96#-9HE!K>7VN=,6.=&\U0
MPSF_KK#SNP.8>6JX;F0Y,?T;PW%?#[,’/5+516IX[AMA0M"S33DE1MX?;H9
M[O0NY]5PO#;#C$/<,X);B.%X;8%9CWXC7WDUO#Y;85)P)*LH8;79]OKN;O>
M4,/SV@U3J&D.2[TN1KYSV0MSJWP.6Q#8DDRXW)=/P%Q’3:C@DR2&U_DDS"34
MA(IJY#OH\S#-GK:TZO;L)'W8Q=@=N%S.FQ7PY]SS:D;I<SN'6K+1NK&#9A[ MWF8#+JKAN=^$^6GL7PW/_0[,)]COIZ/;L9&[U0.83XNX;%S>]F+X;O409DXP M>HYB:O@]]B.83.R+J_YJN$8]@6FW&74\3W7GV<P[[^)#^PH1MY]O8297<QE
MY0Z*X;G@JFZSP/ZW5’C-0Q3]RG.R%7WRG@1A>YYPP:Q"OQLW5’N)W#K M4.N>[._ANN&NQ!,W%YO]/,=Q?#<B%D(%Y+‘W4B$$]@+LR3"S&$5T=S$
MYBHP/^#<N3A<#8^G&LSS7KGLR!PUO$=SG=6@>AO,M3P.H?!5,7^BENGAFMX
M),QGV.M;XJ1]S\Q,’[HP<,2Q;#]2<6IAWV5]AP-9P;<3!O8PWKCU##9VY-
MF)K[O.ST,C4<B]HPVS&ODGX]V,A^KP/3#"8@F^%8U(7IAO’$AJKA-6PLR@- M?:U_3S%<H^)AAO;-94T#Q,B[G;=@?FF)GNU$+S$\KU8P,1CSM;3>;.2W^F_# M)%5R6>M/U/"\6L/DJVPV=8(:CM<[,._AW.FZ4@V/.0EF=46<80?%R'[O"(.I MV[>[^HGA&M4)[email protected],@[ZMXP6_$YH1?[LY'?2PR.>3ELM-%!HCAO3,0
M9EJ@R[J%B)$S;@1,HVHYS6?01V)X/"-ALKS-1@P6(]]38"YA+TS>/M,?PY M$V%\T/_TWZ&&]VZC’-W.K!;#>?&%)C/<V4W^GO^J3"AR).>]3P&DY[:PN
MV0R/9P[,HW>;VOXO!XGA<R%F>@1;UO6B)%WRU3*]N5B9+#8]G'HSS^]W1 ME]1PKLZ'.5C@KX;7>1',!<S]4<3'8OB,6P(3#U,C38S<4U;"S'>^$S\^6SG
M?9,>=2-]G.&B.'QK(;Q=<YWOU0Q/)Y,F,5/_6W,$34<"^>]N_.=B\]#,?); M_0TP3W&^U_I]J!@>ST:841A/88\T,5S'-L$,#,#^R69XS)MA9N(L^#E(#8]Y M"\S=NLBQ8#%2Q[;!%,]+K2V&A[S;I@,/[/4,</$\)CWP,RK9+8_G4V4?’_J MC?O91G_T4M.‘LY&XYX;9A36<L$.,_,:C%,RF@#SVL.IH,7Q^E85)P?X:5&V, M&.XSR\$$HFYTZZ2&QUP>YG3-OQAY1UT!I@AJ0KLN:GA>E6!J._W&9#5<,_\! MD[>UV?ET-1R+4)BGN,L,7*2&>ZW*,<0]O9#,^K&LQ#C.=&AABY5;"!"‘N
MA>Z,92/WN%HP_1JB’^LZ3@SG:MVB?]YECKH_8R-WJP8P[`DME?#<T^.8=U M#L@W7@S/O3',+\B?7*7%\%G@;@739+/+$LI,$L/KDP2S&N.Y4G<RF6CI,]O
MC,0:;NNOAO/PYB7<69+!J2+X77N-,F"OMYH!H>3T<G#U$W?>)D;ZN"TQ2 MCGCK=58-KT]WF!FH\_.NJN$Q)\,XW\&/+3>%C;PW[@538Y39-_O5\/O#%)B^ M5=#SYYLJAL?<!^8MB[<[Y=5PCO5U]D59LQ85Q<@[X<$P79WO9NI.$\,U*@WF M!YP[TZ(D9YM),P=]$BW.DP7PV,>Y>SEW\S^R%+#Y)XF([%79;H_IR-[-.I
M,’.\S=:VFR6&:]1TF&(P5X:IX6?-<.:.^KMFG!BYQWT.,QN?TW>J&E[#63/ MD/-KOE##-6HV3,]\V#]KQ<B_2UH($]@9]_/O9HOA-5SBY'-1EQWVF".&Z\8* M9\SX\[B!&/EMZE?X^T$8\^F$N6SDMT_;8&;AC/-N\848WH,[G+CC/"W9B4V, MO#?>"W,0?_:-$2/O:8_C[W_".O]Q4@WOY5/.^E3$'3]C'AO>%^YO82+1:]6[ MK8;KX1F8Z;A?)+Q4P^/Y'J8,:N:#///%\#J??6V>J)'W+>=A^F$\K2N(D5[B M&LRA4L[=<X$8?M8=F"-8GRF?+Q3#,?T5IDE^L\8Y%K&1W^W<A_''>&:-4\/K M\P#FH@]ZI"EJ>'T>.GL9]7#@0C%R[A3"&9BNK%M]!(V_/MGMQ^,?WFSW.ZE
M8KAO093#6?WI+9JY-^1^3G_X[)I:KA^NS\6]B(8)>5G:V&\SD0QOD-Q_F%
M:GB=G;/=U_E>>#<;;U?:9AHG’'5AB\3PS&M"!."'NG>[>5LYX2I.!-4R.
M7LE&SI1(F7OFI5^O$H,KT\TS-;3S:U*X&HQO$]C8-*1S_.#U?!=.!;F\P"7 M+:KZOUW<STO381P'\*<.;1T\98,V-NDR04$F#(**(35J6X$*ZH*$J971=DBH MA1H8%$4:[2((HH@'(\KY(^9<>%ETZ%!_@$A>1G0(LMR(B,&B]\,ZO-_>7VS/ M]WD^S^?S>;Y[F!K.SZ=A?J-VOPFHX7D^"W/[3L2T710C=TXZ8#Z@#IYQO1;# M\7S.QD^O,4>B8F1_16#>85^$[B^+X1B+P@R6W*:YJ(;'$X,YBGWJV%?#<W@) MIK+=:LH5-3SFRS"^II/FZR\U'!N=,%7TX<&763$\AUTP<9@+!PS':C?,$YB8 M&NFQ>V"ZVW%V**KAO-H+8^](%3^IX=CH@WGAP[GRLQKN(>,PSE/&Y$MJ.'ZN MP#Q$?GGU10W7@JLP,0]ZDN]BI';WP_Q!C<O4U/!S)6"\R#^CQU?$\'H-P*P% MC4EXU/"8AV#\F.?WCE4QG'^NV77'^?3C<-K>AUF%S%VWJF&U_0&S",\5[I1 MC/02PS"/\3GN(36\7DF8F]CO,T_7R<A]O`4SE\E_]9CHWDJ(R=GY\-)C2O MAFO!-$S9WG/(J^$UM??"CB'73?[88"/O1>U_'DS@N?8J:CCWSL+<2J$_K*GA M?3H'8WN@I#<OAGO(15?]?>:W!3&2HY9LK!XVYL%?-;Q>RS#/85K2FVPD1V5A M[.^1X7$UG!-RKGI=WAHIB.'XV8!)X[P3OZN&QY.'V3F$L^X],7+'8_._<8RI MX;[W+<PT^N=4M1#^!U!+0(4!0````()9(\31/<?*PE!8``%!&(```` L````````(`````````!)34%0+DU22U!+!08``````0`!`#8"Z%@``````
`
end

fabvit · July 18, 2006, 7:14pm

“ChrisH” [email protected] writes:

Looks like you need to investigate #unpack.

In any case, Ruby F.s has a BinaryReader mixin
(http://facets.rubyforge.org/api/more/classes/BinaryReader.html) that
does the reading and upacking for you. Just mix it into File (or a
subclass of File) and you should be good to go…

That’s fine if you want to pull out each field in succession yourself,
but BitStruct provides much more than that, by providing a DSL for
packed-bit structures. Also, if you see my reply, you’ll notice that
he was in fact very close to getting what he wanted.

Actually, going through this exercise has pointed up some features
that I would like to add to BitStruct, since in many cases it almost
but not quite completely was exactly what the poster wanted. It would
be nice to have an easy, obvious, and supported way to define extra
padding in a structure (as we needed to here). It would be nice to
have an easier, supported syntax for reading a structure from a binary
file. Finally, it would be nice to allow an easy way to define data
wrappers, as was done with the date property.

fabvit · July 18, 2006, 8:04pm

Daniel M. wrote:
…

Actually, going through this exercise has pointed up some features
that I would like to add to BitStruct, since in many cases it almost
but not quite completely was exactly what the poster wanted. It would
be nice to have an easy, obvious, and supported way to define extra
padding in a structure (as we needed to here). It would be nice to
have an easier, supported syntax for reading a structure from a binary
file. Finally, it would be nice to allow an easy way to define data
wrappers, as was done with the date property.

My ref to BinaryReader wasn’t a slight against BitStruct, I haven’t
used either
so can’t comment. Just found it and figured I’d throw it into the mix.

Just occurred to me that combining BitStruct with BinaryReader and
maybe StringIO could produce a nice BinaryIO class/module?

BTW, I noticed that all the fields in the BitStruct had endianess
specified.
Is there a way to set the endianess for the whole structure? Would you

have a strucutre with mixed endianess?

Nice work
Chris

fabvit · July 24, 2006, 4:06pm

This time I’m trying to write a binary file.

Q1: why does the structure MRKMessageFlags does not get the apropriate
values?

Q2: how do I convert a date to seconds-since-1970?

require ‘bit-struct’
class MRKHeader < BitStruct
unsigned :version, 32, “Version”, :endian =>
:native
unsigned :uid_Validity, 32, “UIDValidity”, :endian =>
:native
unsigned :uid_next, 32, “UIDNext”, :endian =>
:native
unsigned :last_write_counter, 32, “LastWriteCounter”, :endian =>
:native
rest :unused, “Unused”
# Override so that it gets padded properly
def MRKHeader.round_byte_length
super
36
end
end

Ideally, I’d construct some sort of “flags” bit-struct field

Or define a boolean field type and make this a series of boolean

fields.

However, for now we can deal with a series of 0s and 1s

class MRKMessageFlags < BitStruct
unsigned :flagUnused, 2, “Unused”
unsigned :flagSeen, 1, “Seen”
unsigned :flagAnswered, 1, “Answered”
unsigned :flagFlagged, 1, “Flagged”
unsigned :flagDeleted, 1, “Deleted”
unsigned :flagDraft, 1, “Draft”
unsigned :flagRecent, 1, “Recent”
end

class MRKMessage < BitStruct

Note “text” for nul-terminated strings

text :filename, 23*8, “FileName”, :endian => :native
nest :flags, MRKMessageFlags, “Flags”
unsigned :uid, 32, “UID”, :endian => :native
unsigned :msg_size, 32, “MsgSize”, :endian => :native
unsigned :date, 32, “Date”, :endian => :native

Now we futz with the way that date is set and gotten.

we rename the existing date field to __date, and

then we supply our own meaning for “date” that does

translation into and out of seconds-since-1970

Again, the ideal solution would be to define a new bit-struct

field type that did this stuff itself.

alias_method :__date=, :date=
alias_method :__date, :date
def date=(time)
self.__date= time.to_i
end
def date
Time.at(self.__date)
end

we don’t need to override the length computation here

end

File.open(“imap2.mrk”, “wb”) {|f|
#

mrk_header = MRKHeader.new()
mrk_header.version = 1
mrk_header.uid_Validity = 1106138982
mrk_header.uid_next = 5887
mrk_header.last_write_counter = 9962
mrk_header.unused =
“\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\r\n”
puts mrk_header.inspect

msg = MRKMessage.new()
msg.filename = “md50000006021.msg”

msg.flags.flagSeen = 1
msg.flags.flagAnswered = 0
msg.flags.flagFlagged = 1
msg.flags.flagDeleted = 0
msg.flags.flagDraft = 0
msg.flags.flagRecent = 0
puts msg.flags.inspect

msg.uid = 5885
msg.msg_size = 4184
msg.date = “Mon Jul 24 12:34:04 2006”

puts MRKMessage.new(msg).inspect
}

fabvit · July 24, 2006, 6:00pm

Fabio V. [email protected] writes:

This time I’m trying to write a binary file.

Q1: why does the structure MRKMessageFlags does not get the apropriate
values?

I’m not sure. It appears to be a bug in bit-struct. Here’s a
workaround; in your code replace the bit that sets the flag bits with:

work around bit-struct nested types bug

flags = msg.flags
flags.flagSeen = 1
flags.flagFlagged = 1
msg.flags = flags

Q2: how do I convert a date to seconds-since-1970?

Don’t assign date a string, assign it a Time object. The easiest way
to get one of those is with Time.parse:

msg.date = Time.parse(“Mon Jul 24 12:34:04 2006”)

Your file loop now looks like this: (I also added calls to write out
the data to the file)

File.open(“imap2.mrk”, “wb”) {|f|
#<MRKHeader version=1, uid_Validity=1106138982,

uid_next=5887, last_write_counter=9962,

unused=“\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\r\n”>

mrk_header = MRKHeader.new()
mrk_header.version = 1
mrk_header.uid_Validity = 1106138982
mrk_header.uid_next = 5887
mrk_header.last_write_counter = 9962

I omitted modifying unused since in theory we don’t need to

change the last two bytes to “\r\n” - it is unused, after all

puts mrk_header.inspect
f.write(mrk_header)

msg = MRKMessage.new()
msg.filename = “md50000006021.msg”

work around bit-struct nested types bug

flags = msg.flags
flags.flagSeen = 1
flags.flagFlagged = 1
msg.flags = flags

msg.uid = 5885
msg.msg_size = 4184
msg.date = Time.parse(“Mon Jul 24 12:34:04 2006”)

puts msg.inspect
f.write(msg)
}

fabvit · July 24, 2006, 8:11pm

Thank you all!
I’ll investigate on the bit-struct nested types bug.
Anyway, with Daniel’s workarund it seems to work.

fabvit · July 24, 2006, 6:03pm

Q2: have look at Time.parse or ParseDate in stdlib. Keep in mind that
it is reeeeeally slow, it does all sorts of computation (gcd for
example), so if you need it, it helps to cache it’s results, or
compute offsets from a known date

J.