# Efficient access to a large int32 array **URL:** \ **Category:** Ruby\ **Created:** [January 16, 2007, 3:11pm UTC](https://www.ruby-forum.com/t/efficient-access-to-a-large-int32-array/81957 "2007-01-16T15:11:50Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Martin\_DeMello](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/f17d59/32.png) [@Martin\_DeMello](https://www.ruby-forum.com/u/Martin_DeMello)\ **Post date:** [January 16, 2007, 3:11pm UTC](https://www.ruby-forum.com/t/efficient-access-to-a-large-int32-array/81957/1 "2007-01-16T15:11:50Z")
I have a file that is a dump of a large array of 32-bit unsigned ints (bitvectors actually). All I need is to load this into memory and have fast, efficient access to the array members, and a few bitmasking operations - the array will never be written to, and ideally should not need to be bounds-checked. Should I be looking at something like mmap or narray to do this, or would I be better served by a custom C extension? martin ---
**Author:** ![Martin\_DeMello](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/f17d59/32.png) [@Martin\_DeMello](https://www.ruby-forum.com/u/Martin_DeMello)\ **Post date:** [January 16, 2007, 3:16pm UTC](https://www.ruby-forum.com/t/efficient-access-to-a-large-int32-array/81957/2 "2007-01-16T15:16:53Z")
On 16.01.2007 15:10, Martin DeMello wrote: > I have a file that is a dump of a large array of 32-bit unsigned ints > (bitvectors actually). All I need is to load this into memory and have > fast, efficient access to the array members, and a few bitmasking > operations - the array will never be written to, and ideally should > not need to be bounds-checked. Should I be looking at something like > mmap or narray to do this, or would I be better served by a custom C > extension? How large is “large”? robert ---
**Author:** ![Martin\_DeMello](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/f17d59/32.png) [@Martin\_DeMello](https://www.ruby-forum.com/u/Martin_DeMello)\ **Post date:** [January 16, 2007, 3:17pm UTC](https://www.ruby-forum.com/t/efficient-access-to-a-large-int32-array/81957/3 "2007-01-16T15:17:53Z")
Martin DeMello wrote: > I have a file that is a dump of a large array of 32-bit unsigned ints > (bitvectors actually). All I need is to load this into memory and have > fast, efficient access to the array members, and a few bitmasking > operations - the array will never be written to, and ideally should > not need to be bounds-checked. Should I be looking at something like > mmap or narray to do this, or would I be better served by a custom C > extension? I would write my own extension and use mmap (if you’re under linux) to save trouble and memory. That’s just a question of a few hours, and you’ll get high performance. Cheers, Vince, in a posting mood today… ---
**Author:** ![Martin\_DeMello](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/f17d59/32.png) [@Martin\_DeMello](https://www.ruby-forum.com/u/Martin_DeMello)\ **Post date:** [January 16, 2007, 3:30pm UTC](https://www.ruby-forum.com/t/efficient-access-to-a-large-int32-array/81957/4 "2007-01-16T15:30:04Z")
On 1/16/07, Robert K. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > On 16.01.2007 15:10, Martin DeMello wrote: > > > I have a file that is a dump of a large array of 32-bit unsigned ints > > (bitvectors actually). All I need is to load this into memory and have > > fast, efficient access to the array members, and a few bitmasking > > operations - the array will never be written to, and ideally should > > not need to be bounds-checked. Should I be looking at something like > > mmap or narray to do this, or would I be better served by a custom C > > extension? > > How large is “large”? Order of 1 MB - not gigantic, but not trivial either. martin ---
**Author:** ![Martin\_DeMello](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/f17d59/32.png) [@Martin\_DeMello](https://www.ruby-forum.com/u/Martin_DeMello)\ **Post date:** [January 16, 2007, 4:16pm UTC](https://www.ruby-forum.com/t/efficient-access-to-a-large-int32-array/81957/5 "2007-01-16T16:16:18Z")
On 16.01.2007 15:29, Martin DeMello wrote: > > How large is “large”? > > Order of 1 MB - not gigantic, but not trivial either. Um, I would not say 1MB is large. What accesses do you do? Do you just access based on index? Do you often need to load this data? etc. In your case I’d start out writing a Ruby version (either using a single String or an Array with integers) and change that into an extension if performance is insufficient. My 0.02 EUR… Kind regards robert ---
**Author:** ![Martin\_DeMello](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/f17d59/32.png) [@Martin\_DeMello](https://www.ruby-forum.com/u/Martin_DeMello)\ **Post date:** [January 16, 2007, 8:25pm UTC](https://www.ruby-forum.com/t/efficient-access-to-a-large-int32-array/81957/6 "2007-01-16T20:25:23Z")
“Martin DeMello” [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) writes: > I have a file that is a dump of a large array of 32-bit unsigned ints > (bitvectors actually). All I need is to load this into memory and have > fast, efficient access to the array members, and a few bitmasking > operations - the array will never be written to, and ideally should > not need to be bounds-checked. Should I be looking at something like > mmap or narray to do this, or would I be better served by a custom C > extension? I think Ara has been hacking some package to use Narray on mmap’ed files, which works fast and is persistent. Search the archives. ---
**Author:** ![Martin\_DeMello](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/f17d59/32.png) [@Martin\_DeMello](https://www.ruby-forum.com/u/Martin_DeMello)\ **Post date:** [January 16, 2007, 8:52pm UTC](https://www.ruby-forum.com/t/efficient-access-to-a-large-int32-array/81957/7 "2007-01-16T20:52:26Z")
On 1/16/07, Christian N. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > I think Ara has been hacking some package to use Narray on mmap’ed > files, which works fast and is persistent. Search the archives. [http://codeforpeople.com/lib/ruby/nmap/](http://codeforpeople.com/lib/ruby/nmap/) You’ll need to install Guy’s mmap package … [http://moulon.inra.fr/ruby/mmap.html](http://moulon.inra.fr/ruby/mmap.html) Blessings, TwP ---
**Author:** ![Martin\_DeMello](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/f17d59/32.png) [@Martin\_DeMello](https://www.ruby-forum.com/u/Martin_DeMello)\ **Post date:** [January 16, 2007, 4:22pm UTC](https://www.ruby-forum.com/t/efficient-access-to-a-large-int32-array/81957/8 "2007-01-16T16:22:19Z")
On 1/16/07, Robert K. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > Um, I would not say 1MB is large. What accesses do you do? Do you just > access based on index? Do you often need to load this data? etc. Pure array indexing, just a lot of it. > In your case I’d start out writing a Ruby version (either using a single > String or an Array with integers) and change that into an extension if > performance is insufficient. My 0.02 EUR… Good point - it’d be the quickest thing to get up and running, if nothing else 🙂 martin