# Customized serialization unreliable?

**URL:** <https://www.ruby-forum.com/t/customized-serialization-unreliable/138406>\
**Category:** Ruby\
**Created:** [May 22, 2008, 6:16pm UTC](https://www.ruby-forum.com/t/customized-serialization-unreliable/138406 "2008-05-22T18:16:20Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![sayoyo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/ba8739/32.png) [@sayoyo](https://www.ruby-forum.com/u/sayoyo)\
**Post date:** [May 22, 2008, 6:16pm UTC](https://www.ruby-forum.com/t/customized-serialization-unreliable/138406/1 "2008-05-22T18:16:20Z")

</div>

Hi,

I have a strange problem with the customized serialization…  
I have a large object which is built by several classes and each class  
has its own marshal\_dump and marshal\_load, like:

class YYY  
attr\_reader :data, :version

def initialize  
@data = “different objects”  
@version = 1  
end

def marshal\_dump()  
return [@version,@data]  
end

def marshal\_load(var)  
@version = var[0]  
case @version  
when 1  
@data = var[1]  
else  
#do something else  
end  
end  
end

The problem I find out is the serialization works randomly, sometime the  
object is serialized correctly and sometime not.(since I’m not able to  
deserialize the object, some of the attributes of the objects become  
nil) also I find out the size of serialized file is different each time.  
and I try to put the serialized data in memory, the size of the memory  
is also different each time. like this:

```
              dumpStr = Marshal.dump(mainResults)
              puts dumpStr.length.to_s()
              mainResults2 = Marshal.load(dumpStr)

```

Does someone know anything about the customized serialization? There is  
not a lot of doc about this…

Thanks you very much

Sayoyo

---

<div class="post-metadata">

**Author:** ![sayoyo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/ba8739/32.png) [@sayoyo](https://www.ruby-forum.com/u/sayoyo)\
**Post date:** [May 22, 2008, 6:34pm UTC](https://www.ruby-forum.com/t/customized-serialization-unreliable/138406/2 "2008-05-22T18:34:49Z")

</div>

sayoyo Sayoyo wrote:

> Hi,
> 
> I have a strange problem with the customized serialization…  
> I have a large object which is built by several classes and each class  
> has its own marshal\_dump and marshal\_load, like:

Another thing to try: call marshal\_dump on your top-level object. You  
should get a nested structure of arrays and so on. Does this look ok?

Now, feed that data structure back into your top-level marshal\_load  
method. Does this cause the problem too?

---

<div class="post-metadata">

**Author:** ![sayoyo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/ba8739/32.png) [@sayoyo](https://www.ruby-forum.com/u/sayoyo)\
**Post date:** [May 22, 2008, 6:48pm UTC](https://www.ruby-forum.com/t/customized-serialization-unreliable/138406/3 "2008-05-22T18:48:54Z")

</div>

Hi,

First, Thanks you very much for helping me, Yes, I have tried it, and  
the serialization still works randomly. no idea why…

I suspect the there is a problem in memory allocation somewhere when  
data is written to the “file”, since it is an random effect, I can  
hardly put a hand on it.

do you know who is the responsible of this part of ruby?

Sayoyo

---

<div class="post-metadata">

**Author:** ![sayoyo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/ba8739/32.png) [@sayoyo](https://www.ruby-forum.com/u/sayoyo)\
**Post date:** [May 22, 2008, 9:41pm UTC](https://www.ruby-forum.com/t/customized-serialization-unreliable/138406/4 "2008-05-22T21:41:29Z")

</div>

On 22.05.2008 18:48, sayoyo Sayoyo wrote:

> First, Thanks you very much for helping me, Yes, I have tried it, and  
> the serialization still works randomly. no idea why…
> 
> I suspect the there is a problem in memory allocation somewhere when  
> data is written to the “file”, since it is an random effect, I can  
> hardly put a hand on it.

My gut feeling rather points to an effect caused by different ordering  
of objects in a Hash. Or you have an issue caused by a loop in your  
object graph. As far as I can see custom serialization works ok - at  
least for non complex structures:

irb(main):016:0\> F = Struct.new :a, :b do  
irb(main):017:1\* def marshal\_dump  
irb(main):018:2\> [a,b]  
irb(main):019:2\> end  
irb(main):020:1\>  
irb(main):021:1\* def marshal\_load(x)  
irb(main):022:2\> self.a = x[0]  
irb(main):023:2\> self.b = x[1]  
irb(main):024:2\> end  
irb(main):025:1\> end  
=\> F  
irb(main):026:0\> x = F.new 1,2  
=\> #  
irb(main):027:0\> s = Marshal.load(Marshal.dump(x))  
=\> #  
irb(main):028:0\>

I am not sure why you need custom serialization. But here is an  
alternative approach: create a method that returns a data structure  
which you then serialize and add a class method that constructs your in  
memory structure from that state. E.g.

class Foo  
attr\_accessor :name, :size

def to\_serial  
[name, size]  
end

def self.from\_serial(obj)  
f = new  
f.name = obj[0]  
f.size = obj[1]  
f  
end  
end

Of course, this only works if you know what you are deserializing.

Another alternative is to separate “configuration state” (which is  
serializable, e.g. file name) from “operation state” (which is not  
serializable, e.g. file descriptor) and serialize only the configuration  
state. This is probably the cleanest approach.

> do you know who is the responsible of this part of ruby?

You probably can find out by looking at the sources.

Kind regards

robert

---

<div class="post-metadata">

**Author:** ![sayoyo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/ba8739/32.png) [@sayoyo](https://www.ruby-forum.com/u/sayoyo)\
**Post date:** [May 23, 2008, 10:33am UTC](https://www.ruby-forum.com/t/customized-serialization-unreliable/138406/5 "2008-05-23T10:33:28Z")

</div>

> def marshal\_dump()  
> return [@version,@data]  
> end

If there are actually classes built on instance methods marshal\_dump and  
marshal\_load, then Marshal would be completely ignoring those methods.  
Assuming you meant “\_dump” which is the instance method to overload for  
customized serialization, and either “self.\_load” or klassname.\_load  
which is the class method for the same, then this would be happening:

Array.to\_s will be called in Marshal.dump because it expects a string.  
For example, with the YYY class:  
@version = 1  
@data = [“e”, “i”]  
\_dump returns [1, [“e”, “i”]]  
The Marshaling process will form “1ei”

Your Loading process with “1ei” will give  
@version = 1  
@data = “e” # instead of [“e”, “i”]

If for whatever reason you are Marshaling the results of your  
marshal\_dump methods – Marshal.dump(my\_yyy.marshal\_dump) --, then you  
have neglected to properly handle object graph cycles, and shared  
references.

Further, from the ruby-doc Marshal class:  
“Some objects cannot be dumped: if the objects to be dumped include  
bindings, procedure or method objects, instances of class IO, or  
singleton objects, a TypeError will be raised.”

---

<div class="post-metadata">

**Author:** ![sayoyo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/ba8739/32.png) [@sayoyo](https://www.ruby-forum.com/u/sayoyo)\
**Post date:** [May 23, 2008, 11:30am UTC](https://www.ruby-forum.com/t/customized-serialization-unreliable/138406/6 "2008-05-23T11:30:51Z")

</div>

On 23.05.2008 02:29, Stefan R. wrote:

> > def initialize  
> > @data = “different objects”  
> > @version = 1  
> > end
> > 
> > def marshal\_dump()  
> > return [@version,@data]  
> > end
> 
> Your problems most likely are caused by you returning an array instead  
> of a String. I’m surprised Marshal doesn’t complain.

I don’t think a String must be returned? Apparently the standard lib  
also does not think so:

irb(main):009:0\> require ‘ostruct’  
=\> true  
irb(main):010:0\> o=OpenStruct.new  
=\> #  
irb(main):011:0\> o.foo=123  
=\> 123  
irb(main):012:0\> o.marshal\_dump  
=\> {:foo=\>123}  
irb(main):013:0\> o.marshal\_dump.class  
=\> Hash  
irb(main):014:0\>

> This works here quite well without any random outtakes. At least I  
> haven’t yet hit one.

I’ll show you one - but it’s not random. 🙂 Your approach with the  
String works only well for simple cases. But the downside is that it  
does not handle loops in object graphs properly:

robert@fussel /cygdrive/c/Temp  
$ cat marsh.rb  
F = Struct.new :x, :y  
a = F.new  
b = F.new  
c = F.new a,b  
a.x = c  
b.x = c  
t1 = Marshal.load(Marshal.dump©)  
p t1.equal?(t1.x.x)  
class F  
def marshal\_dump  
[x,y]  
end

def marshal\_load(dat)  
self.x, self.y = dat  
end  
end  
t2 = Marshal.load(Marshal.dump©)

robert@fussel /cygdrive/c/Temp  
$ cat marsh.rb  
F = Struct.new :x, :y  
a = F.new  
b = F.new  
c = F.new a,b  
a.x = c  
b.x = c

t1 = Marshal.load(Marshal.dump©)  
p t1.equal?(t1.x.x)

class F  
def marshal\_dump  
[x,y]  
end

def marshal\_load(dat)  
self.x, self.y = dat  
end  
end

t2 = Marshal.load(Marshal.dump©)  
p t2.equal?(t2.x.x)

class F  
def marshal\_dump  
Marshal.dump([x,y])  
end

def marshal\_load(dat)  
self.x, self.y = Marshal.load(dat)  
end  
end

t3 = Marshal.load(Marshal.dump©)  
p t3.equal?(t3.x.x)

robert@fussel /cygdrive/c/Temp  
$ ruby marsh.rb  
true  
true  
marsh.rb:26:in `marshal_dump': stack level too deep (SystemStackError) from marsh.rb:26:in`dump’  
from marsh.rb:26:in `marshal_dump' from marsh.rb:26:in`dump’  
from marsh.rb:26:in `marshal_dump' from marsh.rb:26:in`dump’  
from marsh.rb:26:in `marshal_dump' from marsh.rb:26:in`dump’  
from marsh.rb:26:in `marshal_dump' ... 15600 levels... from marsh.rb:26:in`dump’  
from marsh.rb:26:in `marshal_dump' from marsh.rb:34:in`dump’  
from marsh.rb:34

robert@fussel /cygdrive/c/Temp  
$

Kind regards

robert

---

<div class="post-metadata">

**Author:** ![sayoyo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/ba8739/32.png) [@sayoyo](https://www.ruby-forum.com/u/sayoyo)\
**Post date:** [May 23, 2008, 11:40am UTC](https://www.ruby-forum.com/t/customized-serialization-unreliable/138406/7 "2008-05-23T11:40:49Z")

</div>

On 23.05.2008 10:33, Andrew M. wrote:

> > def marshal\_dump()  
> > return [@version,@data]  
> > end
> 
> If there are actually classes built on instance methods marshal\_dump and  
> marshal\_load, then Marshal would be completely ignoring those methods.

I don’t think so, at least not in 1.8.6:

robert@fussel /cygdrive/c/Temp  
$ irb  
irb(main):001:0\> class F  
irb(main):002:1\> def marshal\_dump  
irb(main):003:2\> puts “dump”  
irb(main):004:2\> “123”  
irb(main):005:2\> end  
irb(main):006:1\> def marshal\_load(x)  
irb(main):007:2\> puts “load #{x}”  
irb(main):008:2\> end  
irb(main):009:1\> end  
=\> nil  
irb(main):010:0\> Marshal.load(Marshal.dump(F.new))  
dump  
load 123  
=\> #\<F:0x7ff7be4c\>  
irb(main):011:0\>

> Assuming you meant “\_dump” which is the instance method to overload for  
> customized serialization, and either “self.\_load” or klassname.\_load  
> which is the class method for the same, then this would be happening:

I believe this is yet another mechanism to control custom marshalling.

> Array.to\_s will be called in Marshal.dump because it expects a string.

I don’t think so. The mechanism is different. It would be too fragile  
to depend on #to\_s to return something that can be used to deserialize.

> If for whatever reason you are Marshaling the results of your  
> marshal\_dump methods – Marshal.dump(my\_yyy.marshal\_dump) --, then you  
> have neglected to properly handle object graph cycles, and shared  
> references.

Correct. See my other posting for a nice example. 🙂

All in all I believe, if one wants to exclude some fields from  
serialization (like with “transient” in Java) the best way is to  
implement #marshal\_dump to just return an array of the fields that need  
to be serialized and deserialized and implement #marshal\_load(ar)  
accordingly. That way Marshal can properly handle loops in object  
graphs etc.

Kind regards

robert

---

<div class="post-metadata">

**Author:** ![sayoyo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/ba8739/32.png) [@sayoyo](https://www.ruby-forum.com/u/sayoyo)\
**Post date:** [May 23, 2008, 2:29am UTC](https://www.ruby-forum.com/t/customized-serialization-unreliable/138406/8 "2008-05-23T02:29:19Z")

</div>

sayoyo Sayoyo wrote:

> Hi,
> 
> I have a strange problem with the customized serialization…  
> I have a large object which is built by several classes and each class  
> has its own marshal\_dump and marshal\_load, like:
> 
> class YYY  
> attr\_reader :data, :version
> 
> def initialize  
> @data = “different objects”  
> @version = 1  
> end
> 
> def marshal\_dump()  
> return [@version,@data]  
> end

Your problems most likely are caused by you returning an array instead  
of a String. I’m surprised Marshal doesn’t complain.

class Foo  
def marshal\_dump  
Marshal.dump([@version, @data])  
end  
def marshal\_load(data)  
@version, @data = \*Marshal.load(data)  
end  
…  
end

This works here quite well without any random outtakes. At least I  
haven’t yet hit one.

Regards  
Stefan