# The Gems Repository Stats

**URL:** <https://www.ruby-forum.com/t/the-gems-repository-stats/139115>\
**Category:** Ruby\
**Created:** [May 30, 2008, 11:19pm UTC](https://www.ruby-forum.com/t/the-gems-repository-stats/139115 "2008-05-30T23:19:39Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![leslieviljoen](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/3d9bf3/32.png) [@leslieviljoen](https://www.ruby-forum.com/u/leslieviljoen)\
**Post date:** [May 30, 2008, 11:19pm UTC](https://www.ruby-forum.com/t/the-gems-repository-stats/139115/1 "2008-05-30T23:19:39Z")

</div>

Wondering why the endless “bulk updating” takes so long I looked that  
the source a bit.  
There seems that if there are more than 50 gems missing from the quick  
list, a bulk update  
is done:

source\_index.rb:  
use\_incremental = missing\_gems.size \<= INCREMENTAL\_THRESHHOLD

(INCREMENTAL\_THRESHOLD is 50)

…yet it seems the quick list is always more than 50 gems out of date  
(for the  
bulk update seems to always be done)

When downloading and extracting both lists, it look like there are 13464  
gems  
listed in the bulk file and 13432 in the quick list.

(For the curious, they are these files, compressed with zlib:  
[RubyGems.org | your community gem host](http://gems.rubyforge.org/yaml.Z)  
[RubyGems.org | your community gem host](http://gems.rubyforge.org/quick/index.rz))

This is a difference of 32 gems, but then perhaps some on the quick list  
are out  
of date? Or perhaps I got the numbers wrong?

## So to make things faster:

1. Since the bulk index is 854k and expands to 20MB, perhaps there’s a  
way to  
keep that quick index more up-to-date?

2. Only 3134 of the 13432 gems are unique gems - 10298 are older  
versions  
of these gems. I think that people rarely search or install old gems, so  
perhaps  
the list can be split into a file for latest versions versus old  
versions.

3. I often search for gems repeatedly, and the bulk index gets pulled  
down  
repeatedly - why not save this file locally for at least a few hours?  
(probably try to implement this myself just now)

4. Perhaps if the server is taking strain, a mirror or two could be  
set up? I doubt  
many people would care about such relatively small files on their  
servers -  
I’d be willing to ask some people if they’d do a ZA mirror.

Any comments?

Les

---

<div class="post-metadata">

**Author:** ![leslieviljoen](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/3d9bf3/32.png) [@leslieviljoen](https://www.ruby-forum.com/u/leslieviljoen)\
**Post date:** [June 5, 2008, 6:35am UTC](https://www.ruby-forum.com/t/the-gems-repository-stats/139115/2 "2008-06-05T06:35:02Z")

</div>

On May 30, 2008, at 14:18 PM, Leslie V. wrote:

> …yet it seems the quick list is always more than 50 gems out of  
> date (for the  
> bulk update seems to always be done)

You are looking at RubyGems 0.9.4 or older, so this code is at least  
one year old. RubyGems 0.9.5 has a configurable threshold. The next  
release of RubyGems has a new metadata fetcher (SpecFetcher) and will  
not need to perform bulk updates at all (which is 2-4 weeks from  
release).

> When downloading and extracting both lists, it look like there are  
> 13464 gems  
> listed in the bulk file and 13432 in the quick list.

I’m not sure why that is, I’ll have to update my local mirror and see  
if there is a bug in the latest indexer code. Can you get a list of  
the missing gems and file a bug on rubyforge to remind me?

[http://rubyforge.org/tracker/?atid=575&group\_id=126&func=browse](http://rubyforge.org/tracker/?atid=575&group_id=126&func=browse)

> * * *
> 
> 1. Since the bulk index is 854k and expands to 20MB, perhaps there’s  
> a way to  
> keep that quick index more up-to-date?

All indexes are regenerated at the same time in a directory in /tmp  
then moved into place when the index is complete. .gem files may be  
missing because it takes time for the mirrors to sync up to the  
master. The mailing list on the link at the bottom of this email  
would be the best place to take this up.

> 1. Only 3134 of the 13432 gems are unique gems - 10298 are older  
> versions  
> of these gems. I think that people rarely search or install old  
> gems, so perhaps  
> the list can be split into a file for latest versions versus old  
> versions.

This was done in 1.1.1, IIRC, but hasn’t been very helpful.  
SpecFetcher also makes this distinction, but it is written in less of  
a band-aid fashion and works out quite cleanly. It was probably a  
mistake to add a latest index to 1.1.1, but without it I would not  
have arrived at the SpecFetcher so quickly.

> 1. I often search for gems repeatedly, and the bulk index gets  
> pulled down  
> repeatedly - why not save this file locally for at least a few hours?  
> (probably try to implement this myself just now)

This is a bug that will not be fixed. SpecFetcher has a much simpler  
API that doesn’t have the big pile of band-aids I applied to the old  
one in an attempt to get it to scale to the current size. The  
original design held up surprisingly well over the years, but it’s  
finally time to see it go.

SpecFetcher will call back into the old bulk index code when a  
repository doesn’t have the indexes it needs, and will spit out a  
warning that hopefully can help the affected users get the indexer  
updated.

> 1. Perhaps if the server is taking strain, a mirror or two could be  
> set up? I doubt many people would care about such relatively small  
> files on their servers - I’d be willing to ask some people if they’d  
> do a ZA mirror.

CPU-wise there isn’t any strain on the mirrors that I know of, since  
they only serve up static content. If you would like to set up a  
mirror, there’s information here:

[http://rubyforge.org/docman/view.php/5/231/mirror\_setup.html](http://rubyforge.org/docman/view.php/5/231/mirror_setup.html)

> Any comments?

This is a better topic for the rubygems-developers list, but I’ve  
solved the problem (hopefully for at least the next five years)  
already. Compare lib/rubygems/spec\_fetcher.rb in trunk to the old code.

---

<div class="post-metadata">

**Author:** ![leslieviljoen](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/3d9bf3/32.png) [@leslieviljoen](https://www.ruby-forum.com/u/leslieviljoen)\
**Post date:** [June 5, 2008, 11:02am UTC](https://www.ruby-forum.com/t/the-gems-repository-stats/139115/3 "2008-06-05T11:02:55Z")

</div>

On Thu, Jun 5, 2008 at 6:32 AM, Eric H. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> > 
> 
> > When downloading and extracting both lists, it look like there are 13464  
> > gems  
> > listed in the bulk file and 13432 in the quick list.

Probably my regular expressions letting me down, I thought this was  
normal and the cause  
of the bulk-updating but have misunderstood.

> > 1. Only 3134 of the 13432 gems are unique gems - 10298 are older versions
> > 2. I often search for gems repeatedly, and the bulk index gets pulled down  
> > hopefully can help the affected users get the indexer updated.  
> > Is specfetcher part of an upcoming version of gems?

> > 1. Perhaps if the server is taking strain, a mirror or two could be  
> > set up? I doubt many people would care about such relatively small files  
> > on their servers - I’d be willing to ask some people if they’d do a ZA  
> > mirror.
> 
> CPU-wise there isn’t any strain on the mirrors that I know of, since they  
> only serve up static content. If you would like to set up a mirror, there’s  
> information here:
> 
> [http://rubyforge.org/docman/view.php/5/231/mirror\_setup.html](http://rubyforge.org/docman/view.php/5/231/mirror_setup.html)

Thanks for your great work!

Les

---

<div class="post-metadata">

**Author:** ![leslieviljoen](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/3d9bf3/32.png) [@leslieviljoen](https://www.ruby-forum.com/u/leslieviljoen)\
**Post date:** [June 5, 2008, 12:49pm UTC](https://www.ruby-forum.com/t/the-gems-repository-stats/139115/4 "2008-06-05T12:49:16Z")

</div>

On Jun 5, 2008, at 01:57 AM, Leslie V. wrote:

> > attempt to get it to scale to the current size. The original  
> > design held up  
> > surprisingly well over the years, but it’s finally time to see it go.
> > 
> > SpecFetcher will call back into the old bulk index code when a  
> > repository  
> > doesn’t have the indexes it needs, and will spit out a warning that  
> > hopefully can help the affected users get the indexer updated.
> 
> Is specfetcher part of an upcoming version of gems?

Yes, 2-4 weeks out. I have to go through the rest of the tracker  
fixing bugs, wait for some feedback from apple and jruby, and give it  
a week or so to settle for testing.