Forum: Ruby rubygems-update 1.6.0 Released

rubygems-update version 1.6.0 has been released!

* <http://rubygems.org>
* <http://docs.rubygems.org>
* <http://help.rubygems.org>
* <http://github.com/rubygems>
* <http://rubyforge.org/projects/rubygems>

RubyGems is a package management framework for Ruby.

This gem is an update for the RubyGems software. You must have an
installation of RubyGems before this update can be applied.

See Gem for information on RubyGems (or `ri Gem`)

To upgrade to the latest RubyGems, run:

  $ gem update --system  # you might need to be an administrator or root

See UPGRADING.rdoc for more details and alternative instructions.

-----

If you don't have any RubyGems install, there is still the pre-gem 
approach to
getting software, doing it manually:

* Download from: https://rubygems.org/pages/download
* Unpack into a directory and cd there
* Install with: ruby setup.rb  # you may need admin/root privilege

For more details and other options, see:

  ruby setup.rb --help

Changes:

### 1.6.0 / 2011-02-29

4 Deprecations:

* RubyGems no longer requires 'thread'.  Rails < 3 will need to add 
require
  'thread' to their applications.
* Gem.cache is deprecated.  Use Gem.source_index.
* RbConfig.datadir is deprecated.  Use Gem.datadir.
* Gem::LoadError#version_requirements has been removed.  Use
  Gem::LoadError#requirement.

2 Major Enhancements:

* Rewrote how Gem::activate (gem and require) resolves dependencies.
* Gem::LoadError#version_requirement has been removed. Use
  Gem::LoadError#requirement.

17 Minor Enhancments:

* Added --key to `gem push` for setting alternate API keys.
* Added --format-executable support to gem uninstall.
* Added Gem::DependencyList#clear.
* Added Gem::DependencyList#remove_specs_unsatisfied_by
* Added Gem.latest_spec_for, latest_version_for, and 
latest_rubygems_version.
* Added Gem::Dependency#merge which merges requirements for two
  dependencies.
* Added Gem::TestCase#util_spec for faster tests.
* Added Gem::Specification#dependent_specs.
* Added Gem::TestCase#new_spec and Gem::TestCase#install_specs.
* Added flag to include prerelease gems in 
Gem::SourceIndex#latest_specs.
* Gem.cache_dir always references the proper cache dir.
  Pass true to support a user path.
* Gem.cache_gem, given a filename always references the cache gem.
  Pass true to support a user path.
* Added Gem::Specification#conflicts
* Removed rdoc gem/require from test_case.rb.
* Rubygems will no longer let you push if you're using beta or 
unreleased
  rubygems.
* Save RAM / GC churn by removing spec.files and rdoc options from
  locally cached gem specifications.
* SpecFetcher.fetch_spec can now take a string source_uri.

10 Bug Fixes:

* Added missing require of Gem::RemoteFetcher to the unpack command.
* RubyGems now completely removes a previous install when reinstalling.
* Fixed Gem::Installer#generate_bin to only chmod files that exist.
* Fixed handling of Windows style file:/// uris.
* Fixed requires in tests. (shota)
* Fixed script generation on Windows.
* Fixed test issues if you have older rubygems installed.
* Gem::DependencyInstaller tests use Gem::Security, add the missing 
require.
* Gem::Security used FileUtils but didn't require it.  Reported by Elia 
Schito.
* Gem::Uninstaller now respects --format-executable.

Can anyone suggest where I should put

require 'thread'

in my application?

I have tried environment.rb but this does not seem to have the desired 
effect....

Put it in config/boot.rb and it works :)

> 2 Major Enhancements:

Cool thanks for the release.

If you ever get the time for it, the following are a couple ideas that 
might help speed up rubygems on windows.  I had some success with them 
with my faster_rubygems project, but haven't kept up with the latest 
updates.  Some of these may already be included in rubygems I don't 
know...

1) use marshal instead of yaml for the specs (this helps jruby boot more 
quickly since yaml is a heavier dependency on that VM).
2) combine all the gemspec files into one "cached" gemspec file, per gem 
directory.
3) also cache somehow a list of each "requireable" file within each gem 
(for instance, within the file for 2).  Then for 
Gem.searcher.find_in_unresolved you just iterate over that list instead 
of actually hitting the file system.

The combination of these makes for lightning fast rubygems in windows, 
though I'm not sure which one has most effect.

Anyway thanks for your work on this.
-roger

On Mar 21, 2011, at 6:54 AM, Roger Pack wrote:
> 1) use marshal instead of yaml for the specs (this helps jruby boot more
> quickly since yaml is a heavier dependency on that VM).

The on-disk gemspecs are not in yaml format, they're in ruby format.

> 2) combine all the gemspec files into one "cached" gemspec file, per gem
> directory.

I haven't been willing to sacrifice the simplicity for doing something 
better in this area, there are many other places in RubyGems that need 
to be improved.  Feel free to take a crack at it, though.

> 3) also cache somehow a list of each "requireable" file within each gem
> (for instance, within the file for 2).  Then for
> Gem.searcher.find_in_unresolved you just iterate over that list instead
> of actually hitting the file system.

I tried this once before and found it to be significantly slower than 
Dir[].  I believe I was working on OS X at the time.

Note: RubyGems no longer places bin/ from gems with executables on 
$LOAD_PATH anymore, so the amount of globs required has been cut in (up 
to) half.

> The combination of these makes for lightning fast rubygems in windows,
> though I'm not sure which one has most effect.

Benchmarks would be required, of course.

>> 2) combine all the gemspec files into one "cached" gemspec file, per gem
>> directory.
>
> I haven't been willing to sacrifice the simplicity for doing something
> better in this area, there are many other places in RubyGems that need
> to be improved.  Feel free to take a crack at it, though.

Yeah with some benchmarking this doesn't seem to be the general problem, 
either.


>> 3) also cache somehow a list of each "requireable" file within each gem
>> (for instance, within the file for 2).  Then for
>> Gem.searcher.find_in_unresolved you just iterate over that list instead
>> of actually hitting the file system.
>
> I tried this once before and found it to be significantly slower than
> Dir[].  I believe I was working on OS X at the time.
> Benchmarks would be required, of course.


Interesting.  On my box loading RB itself takes in full about 0.5s 
(which is non awful),

Here's a ruby-prof of a script that requires rubygems and a few gems 
(1.9.2 windows 7):

Total: 2.410000

 %self     total     self     wait    child    calls  name
 83.78      2.12     2.02     0.00     1.89      106 
Kernel#gem_original_require
  2.57      0.15     0.06     0.00     0.08      171  Kernel#eval
  2.32      0.06     0.06     0.00     0.00      102  <Class::IO>#read

so the bottleneck currently appears to be in the require call itself 
(though RG adds about 0.3s that might not be a big problem), and not in 
the Gem.searcher.

So the question is how to speed up those calls themselves, I suppose 
(since 2s to load like 3 gems causes us windows users some grief).


I'm not entirely sure if it's file stat's that slow it down or the 
parsing is slow because it has to look for an encoding string at the top 
(I can't imagine that's it, but...)
My guess is it's the file stats.

As a sidenote, if I include the faster_require gem (which caches the 
location of requires to load them directly the second or third time the 
script is run)
those 2.02 seconds go down to about 0.4,
which I assume is "about as fast as it can ever get" (since it's no 
longer doing any extra file stat's, and also skips looking for encoding 
strings).

faster_require by default does
eval(File.read(previously_cached_require_location))

If I have faster_require use
load(previously_cached_require_location)
instead, you'd think lightning fast, since it avoids lots of file stats, 
but it takes 1.7s--almost as long as normal gem_original_require does,
so I'm not sure if I have a handle on where the real bottleneck is.

Thoughts?
-roger