Status of rubygems unit tests

Ruby IronRuby
1

I have been working on making the rubygem unit tests (from
git://github.com/vvs/rubygems.git) pass with IronRuby. When I started,
the results looked like this.

707 tests, 1170 assertions, 9 failures, 352 errors, 1 skips

Half the tests (9+352/707) were failing (or half were passing depending
on how you want to look at it). The main culprits were Zlib::GzipWriter,
YAML.quick_emit and Zlib::Deflate. After fixing these issues, the
results look like this:

707 tests, 1881 assertions, 40 failures, 53 errors, 4 skips

It’s a good improvement. Errors (where the tests cannot even run
properly) account for 8% (53/707), and failures (functionality defects)
can now show through - in 5% (40/707) of the tests, for a pass rate of
87%. The errors are because of the following exceptions in descending
order of occurrence. The number at the start is the count of how often
the error happened.

9: dump format error(9)
8: ERROR: Failed to build gem native extension.
5: dump format error(8)
5: marshal data too short
4: uninitialized constant OpenSSL::X509
2: uninitialized constant OpenSSL::PKey
2: trying to seek on a non-existent stream?
2: ERROR: Failed to build gem native extension.
2: SSL is not installed on this system
2: SocketError: An established connection was aborted by the software in
your host machine

I will keep pushing on the tests. The failures will be a bit more work
to narrow down as some expectation fails, possibly much after a bug
introduced a bad result in the system. If anyone wants to help dig into
the failures, let me know, and I can help you get set up.

Thanks,
Shri

2

Hello Shri,
I would like to help on this.

please include my email/IM:

Thank you,
-Jirapong

3

Hi Jirapong, It will be great to pair up with you to push on getting
RubyGems tests working with IronRuby. I have written up the instructions
for running the tests at
http://www.ironruby.net/index.php?title=Documentation/How_do_I_install_additional_gems%3F.
Can you build IronRuby (http://wiki.github.com/ironruby/ironruby has
instructions, and we are trying to improve it, so do let us know if
there are any rough edges), and then walk through the steps to get to
the same number of failures. Once both of us have the same setup, we can
split up the failures for investigation. Let me know if you run into
problems and I can help you through it.

Thanks,
Shri

From: [email protected]
[mailto:[email protected]] On Behalf Of
[email protected]
Sent: Tuesday, March 17, 2009 12:13 AM
To: [email protected]
Subject: Re: [Ironruby-core] Status of rubygems unit tests

Hello Shri,
I would like to help on this.

please include my email/IM:

Thank you,
-Jirapong

On Mar 17, 2009, at 1:10 PM, Shri B. wrote:

I have been working on making the rubygem unit tests (from
git://github.com/vvs/rubygems.git) pass with IronRuby. When I started,
the results looked like this.

707 tests, 1170 assertions, 9 failures, 352 errors, 1 skips

Half the tests (9+352/707) were failing (or half were passing depending
on how you want to look at it). The main culprits were Zlib::GzipWriter,
YAML.quick_emit and Zlib::Deflate. After fixing these issues, the
results look like this:

707 tests, 1881 assertions, 40 failures, 53 errors, 4 skips

It’s a good improvement. Errors (where the tests cannot even run
properly) account for 8% (53/707), and failures (functionality defects)
can now show through - in 5% (40/707) of the tests, for a pass rate of
87%. The errors are because of the following exceptions in descending
order of occurrence. The number at the start is the count of how often
the error happened.

9: dump format error(9)
8: ERROR: Failed to build gem native extension.
5: dump format error(8)
5: marshal data too short
4: uninitialized constant OpenSSL::X509
2: uninitialized constant OpenSSL::PKey
2: trying to seek on a non-existent stream?
2: ERROR: Failed to build gem native extension.
2: SSL is not installed on this system
2: SocketError: An established connection was aborted by the software in
your host machine

I will keep pushing on the tests. The failures will be a bit more work
to narrow down as some expectation fails, possibly much after a bug
introduced a bad result in the system. If anyone wants to help dig into
the failures, let me know, and I can help you get set up.

Thanks,
Shri

4

Here are the latest results:

728 tests, 2021 assertions, 26 failures, 35 errors, 4 skips

Latests results will be available at
http://www.ironruby.net/Documentation/How_do_I_install_additional_gems%3F.
The total number of tests increased because we moved from RubyGems
version 0.9 something to the trunk of 1.3.

Jirapong has been tagging with me to help push on this which has been
very useful.

The issues we have fixed include:

  •     GzipWriter.finish should close the GzipWriter object but not

the underlying IO object

  •     File.chmod was a nop before

  •     If a gem supports multiple platforms, the output is

non-deterministic since the code iterates a Hash. The keys of a hash are
not required to be in any order. Jirapong is going to submit a patch to
RubyGems.

  •     Jirapong is working on a fix where File.open(File::EXCL)

should not keep the file locked. This could fix 8 errors.

Thanks,
Shri

From: [email protected]
[mailto:[email protected]] On Behalf Of Shri B.
Sent: Monday, March 16, 2009 11:10 PM
To: ‘[email protected]’
Subject: [Ironruby-core] Status of rubygems unit tests

I have been working on making the rubygem unit tests (from
git://github.com/vvs/rubygems.git) pass with IronRuby. When I started,
the results looked like this.

707 tests, 1170 assertions, 9 failures, 352 errors, 1 skips

Half the tests (9+352/707) were failing (or half were passing depending
on how you want to look at it). The main culprits were Zlib::GzipWriter,
YAML.quick_emit and Zlib::Deflate. After fixing these issues, the
results look like this:

707 tests, 1881 assertions, 40 failures, 53 errors, 4 skips

It’s a good improvement. Errors (where the tests cannot even run
properly) account for 8% (53/707), and failures (functionality defects)
can now show through - in 5% (40/707) of the tests, for a pass rate of
87%. The errors are because of the following exceptions in descending
order of occurrence. The number at the start is the count of how often
the error happened.

9: dump format error(9)
8: ERROR: Failed to build gem native extension.
5: dump format error(8)
5: marshal data too short
4: uninitialized constant OpenSSL::X509
2: uninitialized constant OpenSSL::PKey
2: trying to seek on a non-existent stream?
2: ERROR: Failed to build gem native extension.
2: SSL is not installed on this system
2: SocketError: An established connection was aborted by the software in
your host machine

I will keep pushing on the tests. The failures will be a bit more work
to narrow down as some expectation fails, possibly much after a bug
introduced a bad result in the system. If anyone wants to help dig into
the failures, let me know, and I can help you get set up.

Thanks,
Shri

5

Thanks for the update. I can add this to the runs of
http://ironruby.info so we have up-to-date stats about RubyGems.

From: [email protected]
[mailto:[email protected]] On Behalf Of Shri B.
Sent: Friday, April 03, 2009 2:15 PM
To: [email protected]
Subject: Re: [Ironruby-core] Status of rubygems unit tests

Here are the latest results:

728 tests, 2021 assertions, 26 failures, 35 errors, 4 skips

Latests results will be available at
http://www.ironruby.net/Documentation/How_do_I_install_additional_gems%3F.
The total number of tests increased because we moved from RubyGems
version 0.9 something to the trunk of 1.3.

Jirapong has been tagging with me to help push on this which has been
very useful.

The issues we have fixed include:

  •     GzipWriter.finish should close the GzipWriter object but not

the underlying IO object

  •     File.chmod was a nop before

  •     If a gem supports multiple platforms, the output is

non-deterministic since the code iterates a Hash. The keys of a hash are
not required to be in any order. Jirapong is going to submit a patch to
RubyGems.

  •     Jirapong is working on a fix where File.open(File::EXCL)

should not keep the file locked. This could fix 8 errors.

Thanks,
Shri

From: [email protected]
[mailto:[email protected]] On Behalf Of Shri B.
Sent: Monday, March 16, 2009 11:10 PM
To: ‘[email protected]’
Subject: [Ironruby-core] Status of rubygems unit tests

I have been working on making the rubygem unit tests (from
git://github.com/vvs/rubygems.git) pass with IronRuby. When I started,
the results looked like this.

707 tests, 1170 assertions, 9 failures, 352 errors, 1 skips

Half the tests (9+352/707) were failing (or half were passing depending
on how you want to look at it). The main culprits were Zlib::GzipWriter,
YAML.quick_emit and Zlib::Deflate. After fixing these issues, the
results look like this:

707 tests, 1881 assertions, 40 failures, 53 errors, 4 skips

It’s a good improvement. Errors (where the tests cannot even run
properly) account for 8% (53/707), and failures (functionality defects)
can now show through - in 5% (40/707) of the tests, for a pass rate of
87%. The errors are because of the following exceptions in descending
order of occurrence. The number at the start is the count of how often
the error happened.

9: dump format error(9)
8: ERROR: Failed to build gem native extension.
5: dump format error(8)
5: marshal data too short
4: uninitialized constant OpenSSL::X509
2: uninitialized constant OpenSSL::PKey
2: trying to seek on a non-existent stream?
2: ERROR: Failed to build gem native extension.
2: SSL is not installed on this system
2: SocketError: An established connection was aborted by the software in
your host machine

I will keep pushing on the tests. The failures will be a bit more work
to narrow down as some expectation fails, possibly much after a bug
introduced a bad result in the system. If anyone wants to help dig into
the failures, let me know, and I can help you get set up.

Thanks,
Shri

6

Here are the latest results:

755 tests, 2422 assertions, 8 failures, 14 errors, 4 skips

The total number of tests increased after I did “gem i builder” since
some of the tests get skipped if that gem is not present. The log is at
http://gist.github.com/92738. As you can see, the main remaining issue
is that "igem cert"http://rubygems.org/read/chapter/10#page93 does not
work as OpenSSL is not implemented. Jirapong is going to take a look at
implementing it. However, most gems are unsigned, and so this is not a
big issue. If anyone knows of important gems that are signed, let us
know.

Btw, “igem i rails” works without any issue!

Also, I am working on getting the RubyGems tests included in the repo so
that they can be run as part of continuous integration.

Thanks,
Shri

From: Shri B.
Sent: Friday, April 03, 2009 2:15 PM
To: ‘[email protected]’
Subject: RE: Status of rubygems unit tests

Here are the latest results:

728 tests, 2021 assertions, 26 failures, 35 errors, 4 skips

Latests results will be available at
http://www.ironruby.net/Documentation/How_do_I_install_additional_gems%3F.
The total number of tests increased because we moved from RubyGems
version 0.9 something to the trunk of 1.3.

Jirapong has been tagging with me to help push on this which has been
very useful.

The issues we have fixed include:

  •     GzipWriter.finish should close the GzipWriter object but not

the underlying IO object

  •     File.chmod was a nop before

  •     If a gem supports multiple platforms, the output is

non-deterministic since the code iterates a Hash. The keys of a hash are
not required to be in any order. Jirapong is going to submit a patch to
RubyGems.

  •     Jirapong is working on a fix where File.open(File::EXCL)

should not keep the file locked. This could fix 8 errors.

Thanks,
Shri

From: [email protected]
[mailto:[email protected]] On Behalf Of Shri B.
Sent: Monday, March 16, 2009 11:10 PM
To: ‘[email protected]’
Subject: [Ironruby-core] Status of rubygems unit tests

I have been working on making the rubygem unit tests (from
git://github.com/vvs/rubygems.git) pass with IronRuby. When I started,
the results looked like this.

707 tests, 1170 assertions, 9 failures, 352 errors, 1 skips

Half the tests (9+352/707) were failing (or half were passing depending
on how you want to look at it). The main culprits were Zlib::GzipWriter,
YAML.quick_emit and Zlib::Deflate. After fixing these issues, the
results look like this:

707 tests, 1881 assertions, 40 failures, 53 errors, 4 skips

It’s a good improvement. Errors (where the tests cannot even run
properly) account for 8% (53/707), and failures (functionality defects)
can now show through - in 5% (40/707) of the tests, for a pass rate of
87%. The errors are because of the following exceptions in descending
order of occurrence. The number at the start is the count of how often
the error happened.

9: dump format error(9)
8: ERROR: Failed to build gem native extension.
5: dump format error(8)
5: marshal data too short
4: uninitialized constant OpenSSL::X509
2: uninitialized constant OpenSSL::PKey
2: trying to seek on a non-existent stream?
2: ERROR: Failed to build gem native extension.
2: SSL is not installed on this system
2: SocketError: An established connection was aborted by the software in
your host machine

I will keep pushing on the tests. The failures will be a bit more work
to narrow down as some expectation fails, possibly much after a bug
introduced a bad result in the system. If anyone wants to help dig into
the failures, let me know, and I can help you get set up.

Thanks,
Shri

7

Btw, “igem i rails” works without any issue!

Awesome, thanks =)

From: [email protected]
[mailto:[email protected]] On Behalf Of Shri B.
Sent: Wednesday, April 15, 2009 5:00 PM
To: [email protected]
Subject: Re: [Ironruby-core] Status of rubygems unit tests

Here are the latest results:

755 tests, 2422 assertions, 8 failures, 14 errors, 4 skips

The total number of tests increased after I did “gem i builder” since
some of the tests get skipped if that gem is not present. The log is at
http://gist.github.com/92738. As you can see, the main remaining issue
is that "igem cert"http://rubygems.org/read/chapter/10#page93 does not
work as OpenSSL is not implemented. Jirapong is going to take a look at
implementing it. However, most gems are unsigned, and so this is not a
big issue. If anyone knows of important gems that are signed, let us
know.

Btw, “igem i rails” works without any issue!

Also, I am working on getting the RubyGems tests included in the repo so
that they can be run as part of continuous integration.

Thanks,
Shri

From: Shri B.
Sent: Friday, April 03, 2009 2:15 PM
To: ‘[email protected]’
Subject: RE: Status of rubygems unit tests

Here are the latest results:

728 tests, 2021 assertions, 26 failures, 35 errors, 4 skips

Latests results will be available at
http://www.ironruby.net/Documentation/How_do_I_install_additional_gems%3F.
The total number of tests increased because we moved from RubyGems
version 0.9 something to the trunk of 1.3.

Jirapong has been tagging with me to help push on this which has been
very useful.

The issues we have fixed include:

  •     GzipWriter.finish should close the GzipWriter object but not

the underlying IO object

  •     File.chmod was a nop before

  •     If a gem supports multiple platforms, the output is

non-deterministic since the code iterates a Hash. The keys of a hash are
not required to be in any order. Jirapong is going to submit a patch to
RubyGems.

  •     Jirapong is working on a fix where File.open(File::EXCL)

should not keep the file locked. This could fix 8 errors.

Thanks,
Shri

From: [email protected]
[mailto:[email protected]] On Behalf Of Shri B.
Sent: Monday, March 16, 2009 11:10 PM
To: ‘[email protected]’
Subject: [Ironruby-core] Status of rubygems unit tests

I have been working on making the rubygem unit tests (from
git://github.com/vvs/rubygems.git) pass with IronRuby. When I started,
the results looked like this.

707 tests, 1170 assertions, 9 failures, 352 errors, 1 skips

Half the tests (9+352/707) were failing (or half were passing depending
on how you want to look at it). The main culprits were Zlib::GzipWriter,
YAML.quick_emit and Zlib::Deflate. After fixing these issues, the
results look like this:

707 tests, 1881 assertions, 40 failures, 53 errors, 4 skips

It’s a good improvement. Errors (where the tests cannot even run
properly) account for 8% (53/707), and failures (functionality defects)
can now show through - in 5% (40/707) of the tests, for a pass rate of
87%. The errors are because of the following exceptions in descending
order of occurrence. The number at the start is the count of how often
the error happened.

9: dump format error(9)
8: ERROR: Failed to build gem native extension.
5: dump format error(8)
5: marshal data too short
4: uninitialized constant OpenSSL::X509
2: uninitialized constant OpenSSL::PKey
2: trying to seek on a non-existent stream?
2: ERROR: Failed to build gem native extension.
2: SSL is not installed on this system
2: SocketError: An established connection was aborted by the software in
your host machine

I will keep pushing on the tests. The failures will be a bit more work
to narrow down as some expectation fails, possibly much after a bug
introduced a bad result in the system. If anyone wants to help dig into
the failures, let me know, and I can help you get set up.

Thanks,
Shri

8

Hello Shri,
Following is a result from my laptop.

  1. Getting latest source and build with rake successfully (I did
    clone)
  2. try first test with MRI
    C:\rubygems>ruby.exe c:\rubygems\test\test_gem.rb
    Loaded suite c:/rubygems/test/test_gem
    Started
    …
    Finished in 15.828000 seconds.

51 tests, 79 assertions, 0 failures, 0 errors, 0 skips

  1. I have to change c: to C: in gem_test.rb because error

c:/ruby/lib/ruby/1.8/pathname.rb:709:in relative_path_from': different prefix: "c:/" and "C:/" (ArgumentError) from c:/ruby/lib/ruby/gems/1.8/gems/minitest-1.3.1/lib/ minitest/unit.rb:17 from c:/ruby/lib/ruby/site_ruby/1.8/rubygems/ custom_require.rb:31:in gem_original_require’
from c:/ruby/lib/ruby/site_ruby/1.8/rubygems/
custom_require.rb:31:in `require’
from C:/rubygems/gem_tests.rb:3

  1. run ruby c:\rubygems\gem_tests.rb get result: 728 tests, 1714
    assertions, 13 failures, 229 errors, 4 skips
  2. SET GEM_PATH=C:\ruby\lib\ruby\gems\1.8
  3. edit unit.rb line 47 and location e method
  4. run ir.exe gem_tests.rb result: 728 tests, 1130 assertions, 4
    failures, 415 errors, 4 skips

my gem env
RubyGems Environment:

  • RUBYGEMS VERSION: 1.3.1
  • RUBY VERSION: 1.8.6 (2007-09-24 patchlevel 111) [i386-mswin32]
  • INSTALLATION DIRECTORY: c:/ruby/lib/ruby/gems/1.8
  • RUBY EXECUTABLE: c:/ruby/bin/ruby.exe
  • EXECUTABLE DIRECTORY: c:/ruby/bin
  • RUBYGEMS PLATFORMS:
    • ruby
    • x86-mswin32-60
  • GEM PATHS:
    • c:/ruby/lib/ruby/gems/1.8
    • C:/ruby/lib/ruby/gems/1.8
  • GEM CONFIGURATION:
    • :update_sources => true
    • :verbose => true
    • :benchmark => false
    • :backtrace => false
    • :bulk_threshold => 1000
  • REMOTE SOURCES:

Thank you,
-Jirapong