Gems not working

jim-harris · May 29, 2008, 8:55pm

I am currently not able to install any gems. I was using Gem 1.1.1 but
kept receiving the following errors:

ERROR: While executing gem … (Gem::RemoteSourceException)
Error fetching remote gem cache: Errno::EPIPE: Broken pipe reading
http://gems.rubyonrails.org/yaml

That seemed to be the same problem discussed here:
http://www.ruby-forum.com/topic/149595

That solution does me no good since I am using 32bit and running FreeBSD
not Linux.

I decided to install Rubygem 1.1.0 and tried to install rails and got
this:

gem install rails
Bulk updating Gem source index for: http://gems.rubyonrails.org/
Bulk updating Gem source index for: http://gems.rubyforge.org/
ERROR: While executing gem … (Gem::RemoteFetcher::FetchError)
Errno::ECONNRESET: Connection reset by peer reading
RubyGems.org | your community gem host

Additionally this is stopping me from using the Freebsd rubygem-* ports
due to the same error.

my gem env is:

RubyGems Environment:

RUBYGEMS VERSION: 1.1.0 (1.1.0)
RUBY VERSION: 1.8.6 (2007-09-24 patchlevel 111) [i386-freebsd6]
INSTALLATION DIRECTORY: /usr/local/lib/ruby/gems/1.8
RUBY EXECUTABLE: /usr/local/bin/ruby18
RUBYGEMS PLATFORMS:
- ruby
- x86-freebsd-6
GEM PATHS:
- /usr/local/lib/ruby/gems/1.8
GEM CONFIGURATION:
- :update_sources => true
- :verbose => true
- :benchmark => false
- :backtrace => false
- :bulk_threshold => 1000
- :sources => [“http://gems.rubyonrails.org”,
  “http://gems.rubyforge.org”]
REMOTE SOURCES:
- http://gems.rubyonrails.org
- http://gems.rubyforge.org

Any Ideas? I am at the end of my rope on this.

jim-harris · May 29, 2008, 10:10pm

On Thu, May 29, 2008 at 8:55 PM, Jim H. [email protected]
wrote:

That solution does me no good since I am using 32bit and running FreeBSD
RubyGems.org | your community gem host

INSTALLATION DIRECTORY: /usr/local/lib/ruby/gems/1.8

:backtrace => false

:bulk_threshold => 1000

:sources => [“http://gems.rubyonrails.org”,
“http://gems.rubyforge.org”]

REMOTE SOURCES:

http://gems.rubyonrails.org

http://gems.rubyforge.org

Any Ideas? I am at the end of my rope on this.

Have you tried a couple of times? The errors just look like a slow
line that is disconnecting.

I have had strange proxy issues with Rubyforge before, but that
was always when trying to log in, never with getting gems.

Leslie

jim-harris · May 30, 2008, 5:17pm

On Fri, May 30, 2008 at 7:11 AM, Jim H. [email protected]
wrote:

A day later, still the same results, even after running a barrage of
network diagnostics and tests.

Are you running IPV6 on this system? If so, I’d try disabling that – no
good reason, just a WAG

If that doesn’t help, use a packet sniffer to examine the wire traffic.

Good luck,

jim-harris · May 30, 2008, 4:11pm

Leslie V. wrote:

On Thu, May 29, 2008 at 8:55 PM, Jim H. [email protected]
wrote:

That solution does me no good since I am using 32bit and running FreeBSD
RubyGems.org | your community gem host

INSTALLATION DIRECTORY: /usr/local/lib/ruby/gems/1.8

:backtrace => false

:bulk_threshold => 1000

:sources => [“http://gems.rubyonrails.org”,
“http://gems.rubyforge.org”]

REMOTE SOURCES:

http://gems.rubyonrails.org

http://gems.rubyforge.org

Any Ideas? I am at the end of my rope on this.

Have you tried a couple of times? The errors just look like a slow
line that is disconnecting.

I have had strange proxy issues with Rubyforge before, but that
was always when trying to log in, never with getting gems.

Leslie

That’s what I thought, but other win32 machines on the same subnet work
just fine. It seems contained to the one FreeBSD Box.

A day later, still the same results, even after running a barrage of
network diagnostics and tests.

jim-harris · June 1, 2008, 2:21am

On Fri, 2008-05-30 at 03:55 +0900, Jim H. wrote:

I am currently not able to install any gems. I was using Gem 1.1.1 but
kept receiving the following errors:

ERROR: While executing gem … (Gem::RemoteSourceException)
Error fetching remote gem cache: Errno::EPIPE: Broken pipe reading
http://gems.rubyonrails.org/yaml

Hm, that “yaml” file is 20 MB now… is your client timing out, or maybe
refusing to fetch that much data?

Yours,

Tom

jim-harris · June 1, 2008, 4:55pm

On Sun, 2008-06-01 at 22:53 +0900, Michael T. Richter wrote:

Hm, that “yaml” file is 20 MB now… is your client timing out, or maybe
local source cache from the remote site, but whether or not I specify
–no-update-sources the source cache is downloaded at tremendous
expense. Sometimes it’s downloaded multiple times in a single
invocation (if I install a lot of gems at the command line at once).

Since I’ve had to reinstall all my gems after the upgrade to 1.8.7,
here’s an idea of what’s happening with the gem servers right now:

$ for a in actionmailer RedCloth rspec rubygems-update rails fxruby
bindata actionpack gem_plugin rush ruby-debug-ide activeresource opod ;
do echo $a ; until gem install $a ; do echo Retrying $a… ; done ; done
actionmailer
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyonrails.org/
ERROR: While executing gem … (Gem::RemoteSourceException)
Error fetching remote gem cache: Errno::EPIPE: Broken pipe reading
http://gems.rubyonrails.org/yaml
Retrying actionmailer…
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyonrails.org/
ERROR: While executing gem … (Gem::RemoteSourceException)
Error fetching remote gem cache: Errno::EPIPE: Broken pipe reading
http://gems.rubyonrails.org/yaml
Retrying actionmailer…
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyonrails.org/
ERROR: While executing gem … (Gem::RemoteSourceException)
Error fetching remote gem cache: Errno::EPIPE: Broken pipe reading
http://gems.rubyonrails.org/yaml
Retrying actionmailer…
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyonrails.org/
ERROR: While executing gem … (Gem::RemoteSourceException)
Error fetching remote gem cache: Errno::EPIPE: Broken pipe reading
http://gems.rubyonrails.org/yaml
Retrying actionmailer…
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyonrails.org/
ERROR: While executing gem … (Gem::RemoteSourceException)
Error fetching remote gem cache: Errno::EPIPE: Broken pipe reading
http://gems.rubyonrails.org/yaml
Retrying actionmailer…
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyonrails.org/
ERROR: While executing gem … (Gem::RemoteSourceException)
Error fetching remote gem cache: Errno::EPIPE: Broken pipe reading
http://gems.rubyonrails.org/yaml
Retrying actionmailer…
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyonrails.org/
ERROR: While executing gem … (Gem::RemoteSourceException)
Error fetching remote gem cache: Errno::EPIPE: Broken pipe reading
http://gems.rubyonrails.org/yaml
Retrying actionmailer…
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyforge.org/

Note that the “bulk updating” thing happens twice for
gems.rubyforge.org. Double the opportunity for failure. Notice how the
gems.rubyonrails.org site always fails while downloading that massive
YAML file.

Wouldn’t it be nice if there were some kind of, oh I don’t know,
distributed mechanism for gem retrieval?

jim-harris · June 1, 2008, 3:56pm

On Sun, 2008-06-01 at 09:20 +0900, Tom C. wrote:

On Fri, 2008-05-30 at 03:55 +0900, Jim H. wrote:

I am currently not able to install any gems. I was using Gem 1.1.1 but
kept receiving the following errors:

ERROR: While executing gem … (Gem::RemoteSourceException)
Error fetching remote gem cache: Errno::EPIPE: Broken pipe reading
http://gems.rubyonrails.org/yaml

Hm, that “yaml” file is 20 MB now… is your client timing out, or maybe
refusing to fetch that much data?

The gem servers at rubyforge and whoever hosts for ruby on rails seem to
be badly overloaded. I get these broken pipe errors all the time to the
point that I wrap all uses of gem in “unless” blocks in bash to make
sure things actually go through.

What would help is if the --no-update-sources flag actually did
something. Half the time the point of failure lies in updating the
local source cache from the remote site, but whether or not I specify
–no-update-sources the source cache is downloaded at tremendous
expense. Sometimes it’s downloaded multiple times in a single
invocation (if I install a lot of gems at the command line at once).

jim-harris · June 2, 2008, 1:59am

On Mon, 2008-06-02 at 04:07 +0900, Tom C. wrote:

$ for a in actionmailer RedCloth rspec rubygems-update rails fxruby
bindata actionpack gem_plugin rush ruby-debug-ide activeresource
opod ; do echo $a ; until gem install $a ; do echo Retrying $a… ;
done ; done

I think an easier way to do this might be:

sudo gem install actionmailer RedCloth [… etc …]

The problem with this is that if – scratch that, when – it fails, I
either have to go into the backscroll and figure out which pieces
installed and which didn’t, or I wrap that all up in an until and if
(when) any single gem fails, the whole thing gets restarted.

My way, while less efficient overall, at least doesn’t reinstall gems a
dozen times over because the next-to-last one failed to install. Were
the gems servers actually reliable, I wouldn’t have to do even the until
wrap, not to mention the grotesquely inefficient one-at-a-time approach
I took above.

jim-harris · June 1, 2008, 9:10pm

On Sun, 2008-06-01 at 23:53 +0900, Michael T. Richter wrote:

Since I’ve had to reinstall all my gems after the upgrade to 1.8.7,
here’s an idea of what’s happening with the gem servers right now:

$ for a in actionmailer RedCloth rspec rubygems-update rails fxruby
bindata actionpack gem_plugin rush ruby-debug-ide activeresource
opod ; do echo $a ; until gem install $a ; do echo Retrying $a… ;
done ; done

I think an easier way to do this might be:

sudo gem install actionmailer RedCloth [… etc …]

actionmailer
Bulk updating Gem source index for: http://gems.rubyforge.org/
Bulk updating Gem source index for: http://gems.rubyforge.org/

I’ve noticed this double bulk update sometimes as well. You might want
to post a note to the rubygems-developers list about it.

Yours,

tom

jim-harris · June 2, 2008, 8:13am

On Mon, 2008-06-02 at 10:31 +0900, Tom C. wrote:

What surprises me is seeing the gem client fetch the uncompressed index
(that is, the yaml file) rather than fetching the compressed index. The
compressed version is only 1MB, and seems like that would be much less
likely to time out. It might be worth posting to the
rubygems-developers list and seeing if they could help you hunt down
this problem.

There’s a lot that surprises me in the recent gem releases (1.0+). The
fact that there seems to be absolutely no caching of metadata is the
first point of surprise. Even if I tell it NOT to go looking for the
metadata it goes looking for the metadata which indicates to me that
it’s not caching locally at all and has to. Then the fact, as you point
out, that it needs to download the whole YAML file instead of the
compressed version is suspicious. I’m seriously unimpressed with the
1.0+ versions of gem, frankly.

And yet another mailing list to sign up for? God DAMN! How about just
a bug tracker somewhere? Is there an URL for that? I really don’t want
to increase my incoming mail any more than I already have it.

jim-harris · June 2, 2008, 3:33am

On Mon, 2008-06-02 at 08:58 +0900, Michael T. Richter wrote:

The problem with this is that if – scratch that, when – it fails, I
either have to go into the backscroll and figure out which pieces
installed and which didn’t, or I wrap that all up in an until and if
(when) any single gem fails, the whole thing gets restarted.

Ah right, because the gem client is timing out, gotcha.

My way, while less efficient overall, at least doesn’t reinstall gems
a dozen times over because the next-to-last one failed to install.
Were the gems servers actually reliable, I wouldn’t have to do even
the until wrap, not to mention the grotesquely inefficient
one-at-a-time approach I took above.

What surprises me is seeing the gem client fetch the uncompressed index
(that is, the yaml file) rather than fetching the compressed index. The
compressed version is only 1MB, and seems like that would be much less
likely to time out. It might be worth posting to the
rubygems-developers list and seeing if they could help you hunt down
this problem.

Yours,

Tom

jim-harris · June 2, 2008, 4:07pm

On Mon, 2008-06-02 at 22:01 +0900, Tom C. wrote:

And yet another mailing list to sign up for? God DAMN! How about
just a bug tracker somewhere? Is there an URL for that? I really
don’t want to increase my incoming mail any more than I already have
it.

Yup, there’s the usual complement of RubyForge trackers and such here:

http://rubyforge.org/projects/rubygems/

Thanks, Tom. Lack of sleep made me not even think about looking at
RubyForge.

jim-harris · June 2, 2008, 3:02pm

On Mon, 2008-06-02 at 15:12 +0900, Michael T. Richter wrote:

just a bug tracker somewhere? Is there an URL for that? I really
don’t want to increase my incoming mail any more than I already have
it.

Yup, there’s the usual complement of RubyForge trackers and such here:

http://rubyforge.org/projects/rubygems/

Yours,

Tom

jim-harris · June 5, 2008, 6:58am

On Jun 1, 2008, at 06:53 AM, Michael T. Richter wrote:

The gem servers at rubyforge and whoever hosts for ruby on rails
seem to be badly overloaded. I get these broken pipe errors all the
time to the point that I wrap all uses of gem in “unless” blocks in
bash to make sure things actually go through.

I do not have this problem, and my usage of RubyGems with remote
sources is excessive. Today I installed 500 gems over the internet,
55MB of gems and 1000 files during the afternoon Pacific Time, which
should be some of the peak usage times for the RubyGems mirrors.

Even on crappy coffee-shop internet, or 40K/s send-my-packets-across-
the-country EVDO internet-on-a-stick I never get Errno::EPIPE, so I
think the problem is on your end of the internet, and not on the
RubyForge or gem mirror end.

What would help is if the --no-update-sources flag actually did
something. Half the time the point of failure lies in updating the
local source cache from the remote site, but whether or not I
specify --no-update-sources the source cache is downloaded at
tremendous expense. Sometimes it’s downloaded multiple times in a
single invocation (if I install a lot of gems at the command line at
once).

This code has been replaced in trunk, trunk no longer performs updates
of specs it doesn’t need.

jim-harris · June 5, 2008, 7:02am

On Jun 1, 2008, at 07:53 AM, Michael T. Richter wrote:

Bulk updating Gem source index for: http://gems.rubyonrails.org/
ERROR: While executing gem … (Gem::RemoteSourceException)
Error fetching remote gem cache: Errno::EPIPE: Broken pipe
readinghttp://gems.rubyonrails.org/yaml

No, there is no problem with the rubyforge mirrors.

gem sources -r http://gems.rubyonrails.org/

Will probably fix your problem.

jim-harris · June 2, 2008, 10:29pm

On Mon, Jun 2, 2008 at 4:05 PM, Michael T. Richter
[email protected] wrote:

Thanks, Tom. Lack of sleep made me not even think about looking at
RubyForge.

Sorry to repost this, but it should probably have been attached to
this thread…

Wondering why the endless “bulk updating” takes so long I looked that
the source a bit.
There seems that if there are more than 50 gems missing from the quick
list, a bulk update
is done:

source_index.rb:
use_incremental = missing_gems.size <= INCREMENTAL_THRESHHOLD

(INCREMENTAL_THRESHOLD is 50)

…yet it seems the quick list is always more than 50 gems out of date
(for the
bulk update seems to always be done)

When downloading and extracting both lists, it look like there are 13464
gems
listed in the bulk file and 13432 in the quick list.

(For the curious, they are these files, compressed with zlib:
RubyGems.org | your community gem host
RubyGems.org | your community gem host)

This is a difference of 32 gems, but then perhaps some on the quick list
are out
of date? Or perhaps I got the numbers wrong?

So to make things faster:

Since the bulk index is 854k and expands to 20MB, perhaps there’s a
way to
keep that quick index more up-to-date?
Only 3134 of the 13432 gems are unique gems - 10298 are older
versions
of these gems. I think that people rarely search or install old gems, so
perhaps
the list can be split into a file for latest versions versus old
versions.
I often search for gems repeatedly, and the bulk index gets pulled
down
repeatedly - why not save this file locally for at least a few hours?
(probably try to implement this myself just now)
– did this by now, but there seems to be an attempt at this already
built-in.
Perhaps if the server is taking strain, a mirror or two could be
set up? I doubt
many people would care about such relatively small files on their
servers -
I’d be willing to ask some people if they’d do a ZA mirror.

Any comments?

Les

jim-harris · September 14, 2008, 6:53am

Eric H. wrote:

On Jun 1, 2008, at 07:53 AM, Michael T. Richter wrote:

Bulk updating Gem source index for: http://gems.rubyonrails.org/
ERROR: While executing gem … (Gem::RemoteSourceException)
Error fetching remote gem cache: Errno::EPIPE: Broken pipe
readinghttp://gems.rubyonrails.org/yaml

No, there is no problem with the rubyforge mirrors.

gem sources -r http://gems.rubyonrails.org/

Will probably fix your problem.

appears to be a 403 if i go directly to

http://gems.rubyonrails.org