# Nonlinear scaling - could symbols help solve this?

**URL:** <https://www.ruby-forum.com/t/nonlinear-scaling-could-symbols-help-solve-this/60639>\
**Category:** Ruby\
**Created:** [May 16, 2006, 1:37pm UTC](https://www.ruby-forum.com/t/nonlinear-scaling-could-symbols-help-solve-this/60639 "2006-05-16T13:37:53Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![jeremyheiler](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/34f0e0/32.png) [@jeremyheiler](https://www.ruby-forum.com/u/jeremyheiler)\
**Post date:** [May 16, 2006, 1:37pm UTC](https://www.ruby-forum.com/t/nonlinear-scaling-could-symbols-help-solve-this/60639/1 "2006-05-16T13:37:53Z")

</div>

Hello List,

I have really just started dabbling in Ruby and have found my first  
little task that I can use to try out some of what I’ve read.

Basically I’m reading in a CSV file and converting it into a proprietry  
XML format. To facilitate in the XML generation, I store the items in a  
Hash of Hashes of Arrays. Where the first level of hashing is based on  
the type of data item, the second hash is based on the item’s parent,  
and the Array carries all the items for a specific parent.  
eg. hash=Hash[“class1” =\> {“parent1” =\>  
[class1(“a”),class1(“b”),class1(“c”)], “parent2” =\>  
[class1(“e”),class1(“f”),class1(“g”)], …}, “class2” =\> […], …]

The code below does this in a somewhat clumsy way using Ruby. I find  
that when running this on 10 000 lines -\> execution time is 4.5 seconds,  
however if I increase to 300 000 lines (30 times the data) -\> execution  
time is 14 min 21 sec (168 times the execution time). Is this nonlinear  
scaling due to allocating memory to hold the data? Could I improve this  
using symbols? Any other general remarks would be appreciated.

csv\_file.each\_line do |line| #FORMAT:  
PARENT1,PARENT2,PARENT3,ITEM,ITEMR,ITEMTYPE,SUBITEM,ITEM\_LAYER,PNAME,PVALUE,LEVEL  
sl=line.split(",")  
parent1=sl[0]; parent2=sl[1]; parent3=sl[2]; item=sl[3];  
itemr=sl[4]; itemtype=sl[5]; subitem=sl[6]; item\_layer=sl[7];  
pname=sl[8]; pvalue=sl[9]; level=sl[10]  
if parent1 == “NONE” && parent2 != “NONE”  
parent\_type=“PARENT2”  
node=parent2  
elsif parent1 != “NONE” && parent2 == “NONE”  
parent\_type=“PARENT1”  
node=parent1  
end  
i = case level  
when “ITEM” then Item.new(node,item)  
when “SUBITEM” then Subitem.new(node,item,subitem)  
when “ITEM\_LAYER” then  
Item\_layer.new(node,item,subitem,item\_layer)  
when “RELATION” then Relation.new(node,item,itemr)  
when “PARENT3” then Parent3.new(node,parent3)  
when “PARENT2” then Parent2.new(parent2)  
when “PARENT1” then Parent1.new(parent1)  
end  
if ! i.nil?  
hash[(i.class)]=Hash.new if hash[(i.class)].nil?  
hash[(i.class)][(i.parent)]=Array.new if  
hash[(i.class)][(i.parent)].nil?  
ind = hash[(i.class)][(i.parent)].index(i)  
if ind.nil?  
i.addSetting( Setting.new( pname, pvalue ) )  
hash[(i.class)][(i.parent)].push(i)  
else  
(hash[(i.class)][(i.parent)])[ind].addSetting(  
Setting.new( pname, pvalue ) )  
end  
end  
end

Thanks in advance,  
Jeremy

---

<div class="post-metadata">

**Author:** ![jeremyheiler](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/34f0e0/32.png) [@jeremyheiler](https://www.ruby-forum.com/u/jeremyheiler)\
**Post date:** [May 16, 2006, 3:05pm UTC](https://www.ruby-forum.com/t/nonlinear-scaling-could-symbols-help-solve-this/60639/2 "2006-05-16T15:05:12Z")

</div>

A CSV file is (usually) very much like a database table. And an XML  
document is (usually) a tree. A hash of hashes of arrays is an  
inefficient way to store a tree.

You need an algorithm and a data structure tuned to the algorithm and  
the data. I would recommend starting from the required output and  
working backwards to the input data. Have a look at the Ruby libraries  
devoted to dealing with XML documents, and the Ruby libraries devoted to  
dealing with CSV files. Let the XML libraries build the data structure,  
rather than creating your own.

Jeremy H. wrote:

> csv\_file.each\_line do |line| #FORMAT: PARENT1,PARENT2,PARENT3,ITEM,ITEMR,ITEMTYPE,SUBITEM,ITEM\_LAYER,PNAME,PVALUE,LEVEL  
> when “ITEM” then Item.new(node,item)  
> ind = hash[(i.class)][(i.parent)].index(i)  
> Jeremy

–  
M. Edward (Ed) Borasky

> **[世界杯竞猜彩票-世界杯官网竞猜（集团）有限公司](http://www.linuxcapacityplanning.com/)**
>
> 世界杯竞猜彩票一直以来遵守商业道德，世界杯官网竞猜坚持诚信为本，保证诚实经营。世界杯竞猜彩票先后成为安徽省电子商务协会理事单位、安徽省网商协会会员单位、合肥市电子商务协会理事单位、合肥市服务外包行业协会会员单位、安徽省云计算产业促进会会员单位。世界杯官网竞猜一直保持良好的技术及服务态度，更好地为客户服务。

---

<div class="post-metadata">

**Author:** ![jeremyheiler](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/34f0e0/32.png) [@jeremyheiler](https://www.ruby-forum.com/u/jeremyheiler)\
**Post date:** [May 16, 2006, 4:52pm UTC](https://www.ruby-forum.com/t/nonlinear-scaling-could-symbols-help-solve-this/60639/3 "2006-05-16T16:52:03Z")

</div>

2006/5/16, Jeremy H. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid):

> The code below does this in a somewhat clumsy way using Ruby. I find that when running this on 10 000 lines → execution time is 4.5 seconds, however if I increase to 300 000 lines (30 times the data) → execution time is 14 min 21 sec (168 times the execution time). Is this nonlinear scaling due to allocating memory to hold the data? Could I improve this using symbols? Any other general remarks would be appreciated.

You can verify that yourself. Use your operating system’s tools to  
monitor memory usage and page faults of the Ruby process and you’ll  
soon know whether it’s slow due to paging or not.

Symbols can help with repeated content but you could as well feed  
strings you want to store through a hash like this in order to prevent  
duplicates:

strings = Hash.new {|h,k| k=k.freeze; h[k]=k}

str = strings[str]

Kind regards

robert