Some feedback on the compile/test process


#1

Some notes as I go though the contributing process, in hope that will
be useful to make it smoother:

  • after a clean checkout and launching dev.bat, brbd would not compile
    (today). I believe it reported a missing microsoft.scripting.dll
    issue.
  • opening the solution in VS2008 shows 14 warnings about referenced
    components that cannot be found (System, mscorlib, System.Net), not
    sure if that’s an issue or not
  • compiling through VS2008 makes the compile using brbd works
    afterwards.
  • first pass at launching irtests opens a bunch of windows - most of
    them fail with failure to load default.mspec (see [1]) - after
    searching, the message was caused by an old .mspecrc that was sitting
    in my home folder. Removing it solved the issue.
  • I noticed a runfirst.cmd, should I actually run it or is it obsolete
    ? It seems to have an error in it (references default.mspec.rb which
    has been renamed to default.mspec since).
  • on a french machine, the MiscTests/NumericLiterals1 fails see [2] -
    sounds like a missing InvariantCulture in a float ToString call. I’ll
    look into it unless someone fixes it in between.

On irtests execution: the end of irtests launches irtest.bat, run.bat
and 4 mspec in parallels (“start” calls). Is there a need for that (ie
one process that monitors the others) or could it be run sequentially
if it’s more comfortable for me ? The load is putting my machine down
on its knees.

cheers,

– Thibaut

===== [1] mspec error =====

IronRuby 0.3 0.3.0.0 on .NET 2.0.0.0
:0:in `load’: no such file to load –
C:\git\ironruby\Merlin\Main…\External\Languages\IronRuby\mspec\default.mspec
(LoadError)

===== [2] NumericLiterals1 failure =====

  1. NumericLiterals1
    c:\git\ironruby\Merlin\Main\Languages\Ruby\IronRuby.Tests\Runtime\MiscTests.cs
    : 411
    Unexpected output:

‘1\r\n-1\r\n1\r\n1,1\r\n-1,1\r\n1,1\r\n2.0\r\n2.0\r\n2,156\r\n’.

First difference (8):
actual = ‘,’
expected = ‘.’

Repro: C:\git\ironruby\Merlin\Main\bin\debug\IronRuby.Tests.exe
NumericLiterals1


#2

(today). I believe it reported a missing microsoft.scripting.dll
issue.

Please remove the ClrAssembly.csproj file from the solution … then all
will build fine. That will be removed shortly.

  • opening the solution in VS2008 shows 14 warnings about referenced
    components that cannot be found (System, mscorlib, System.Net), not
    sure if that’s an issue or not

Hmm, weird. The build configuration was definitely Debug, not
“Silverlight Debug” or something like that?

  • compiling through VS2008 makes the compile using brbd works
    afterwards.
  • first pass at launching irtests opens a bunch of windows - most of
    them fail with failure to load default.mspec (see [1]) - after
    searching, the message was caused by an old .mspecrc that was sitting
    in my home folder. Removing it solved the issue.

If you run dev.bat first, this problem should go away. Irtest opens a
bunch of windows, one for each test runner, simply for parallelism.

  • I noticed a runfirst.cmd, should I actually run it or is it obsolete
    ? It seems to have an error in it (references default.mspec.rb which
    has been renamed to default.mspec since).

Runfirst.cmd should be removed, dev.bat is what you want

  • on a french machine, the MiscTests/NumericLiterals1 fails see [2] -
    sounds like a missing InvariantCulture in a float ToString call. I’ll
    look into it unless someone fixes it in between.

Please track this with a bug on codeplex, someone else might have
context.

On irtests execution: the end of irtests launches irtest.bat, run.bat
and 4 mspec in parallels (“start” calls). Is there a need for that (ie
one process that monitors the others) or could it be run sequentially
if it’s more comfortable for me ? The load is putting my machine down
on its knees.

You could make a version of irtest that does them sequentially


#3

A couple of steps later: I meet the following failures while running
irtests on an unmodified fork (by window title, see matching commands
at the end of the mail):

  • smoke tests => 1 failure
  • library rubyspec tests => 4 failures, 15 errors in various places
  • core rubyspec tests => 1 failure, 1 error

I suspect some of these are linked to the fact that I use a french
culture on my machine.

Just do get a point of comparison and know what needs fixing, are all
these supposed to pass, or are some known to fail and that would be
normal ?

cheers,

– Thibaut

C:\Git\ironruby\Merlin\Main\Languages\Ruby>start “Smoke Tests”
C:\git\ironruby\Merlin\Main\Languages\Ruby\Tests\Scripts\irtest.bat
C:\Git\ironruby\Merlin\Main\Languages\Ruby>start “Legacy Tests”
C:\git\ironruby\Merlin\Main\Languages\Ruby\Tests\run.bat
C:\Git\ironruby\Merlin\Main\Languages\Ruby>start “Core RubySpec tests”
mspec ci -fd -V :core
C:\Git\ironruby\Merlin\Main\Languages\Ruby>start “Language RubySpec
tests” mspec ci -fd -V :lang
C:\Git\ironruby\Merlin\Main\Languages\Ruby>start “Library RubySpec
tests” mspec ci -fd -V :lib
C:\Git\ironruby\Merlin\Main\Languages\Ruby>start “Command Line
RubySpec tests” mspec ci -fd -V :cli :netinterop


#4

Yes, these are all suppose to pass … they run “mspec ci” which is the
continuous integration mode … which omits “critical” or “fails” tagged
tests.

It’d be good to know where IronRuby falls over in different cultures, so
feel free to report bugs on codeplex about these things.


#5

Yes, these are all suppose to pass … they run “mspec ci” which is the continuous integration mode … which omits “critical” or “fails” tagged tests.

It’d be good to know where IronRuby falls over in different cultures, so feel free to report bugs on codeplex about these things.

thanks - I added the first one here:
http://ironruby.codeplex.com/WorkItem/View.aspx?WorkItemId=597

and will add or fix more as I go.

– Thibaut


#6

Hi Jimmy,

thanks for the feedback - comments inlined below:

Please remove the ClrAssembly.csproj file from the solution … then all will build fine. That will be removed shortly.

thanks.

  • opening the solution in VS2008 shows 14 warnings about referenced
    components that cannot be found (System, mscorlib, System.Net), not
    sure if that’s an issue or not

Hmm, weird. The build configuration was definitely Debug, not “Silverlight Debug” or something like that?

It is Debug, not Silverlight Debug. I looked at the .csproj and saw
conditional references to these dll when SilverlightDebug is set.
It seems that the condition is not evaluated properly. Not a big deal
though, it seems to work, but could lead to weird things I guess.

  • I noticed a runfirst.cmd, should I actually run it or is it obsolete
    ? It seems to have an error in it (references default.mspec.rb which
    has been renamed to default.mspec since).

Runfirst.cmd should be removed, dev.bat is what you want

ok, thanks.

  • on a french machine, the MiscTests/NumericLiterals1 fails see [2] -
    sounds like a missing InvariantCulture in a float ToString call. I’ll
    look into it unless someone fixes it in between.

Please track this with a bug on codeplex, someone else might have context.

Done: http://ironruby.codeplex.com/WorkItem/View.aspx?WorkItemId=597

On irtests execution: the end of irtests launches irtest.bat, run.bat
and 4 mspec in parallels (“start” calls). Is there a need for that (ie
one process that monitors the others) or could it be run sequentially
if it’s more comfortable for me ? The load is putting my machine down
on its knees.

You could make a version of irtest that does them sequentially

Thanks - much faster.

– Thibaut


#7

Haya,

The problem is in IoOps.ToPrintedString, where there is indeed an
obj.ToString() for doubles. Converting it with Convert.ToString(obj,
System.Globalization.CultureInfo.InvariantCulture) seems to fix the
issue, but I have just tried it in the REPL without running the test
suite yet. If you want, I can commit this fix in my repository if it
passes the tests.

puts 1.1 gives 1,1 here too.

Please commit the fix, this way I’ll be able to see if this solves
other failing tests in french too (I suppose it will).

I’ll have to figure out how to cherry pick one commit from your
branch, the git way :slight_smile:

– Thibaut


#8

Hi,

The problem is in IoOps.ToPrintedString, where there is indeed an
obj.ToString() for doubles. Converting it with Convert.ToString(obj,
System.Globalization.CultureInfo.InvariantCulture) seems to fix the
issue, but I have just tried it in the REPL without running the test
suite yet. If you want, I can commit this fix in my repository if it
passes the tests.

I ran the whole irtests suite after applying the attached patch, based
on what Daniel described.

The good news is that the smoke tests now pass on my french machine.
Library / core errors remain unchanged (I included a log of the
remaining core errors in case someone has more time for this today).

cheers,

– Thibaut


#9

I confirm that I can reproduce this bug on my system too (it-IT). See
below for a simpler test case:

puts 1.1
1,1
=> nil

p 1.1
1.1
=> nil

The problem is in IoOps.ToPrintedString, where there is indeed an
obj.ToString() for doubles. Converting it with Convert.ToString(obj,
System.Globalization.CultureInfo.InvariantCulture) seems to fix the
issue, but I have just tried it in the REPL without running the test
suite yet. If you want, I can commit this fix in my repository if it
passes the tests.

On Thu, Apr 16, 2009 at 09:58, Thibaut Barrère
removed_email_address@domain.invalid wrote:


Ironruby-core mailing list
removed_email_address@domain.invalid
http://rubyforge.org/mailman/listinfo/ironruby-core


Daniele A.
http://www.clorophilla.net/blog/
http://twitter.com/JoL1hAHN


#10

On Thu, Apr 16, 2009 at 12:35, Thibaut Barrère
removed_email_address@domain.invalid wrote:

I ran the whole irtests suite after applying the attached patch, based
on what Daniel described.

You can find a slightly modified patch attached to this mail, this is
what I’m going to push on my repository. I think this one is better
than the first one as it removes the overhead of Convert.ToString and
it reduces the number of unboxing operations from object to double.


#11

I got 14 warning as well, My workaround was to change a Target
Framework to .NET 3.5

Thanks,
-Jirapong


#12

Thibaut, I have a change to irtests.bat which will give an option to run
the tests in series or in parallel. I will push this through in my next
commit so folks can chose how they want to run the tests…

The one issue with running in series is that if a task fails, you want
to run the remaining tasks as well so that you can have the full list of
failures. However, if you just keep running all tasks in series, the
output of the failing task can scroll out, and then you lose the
information of which test failed. The right solution is to redirect the
output of all the tasks so that you can inspect the log files to see the
details of the failure.

Setting up the output redirection, etc, is more work. A rake task would
be nice so that all the logic can be written in Ruby rather than in
irtests.bat. I will leave this for another day. Let’s see how far
irtests.bat can scale…

Thanks,
Shri


#13

Hi Shri,

Thibaut, I have a change to irtests.bat which will give an option to run the tests in series or in parallel. I will push this through in my next commit so folks can chose how they want to run the tests…

The one issue with running in series is that if a task fails, you want to run the remaining tasks as well so that you can have the full list of failures. However, if you just keep running all tasks in series, the output of the failing task can scroll out, and then you lose the information of which test failed. The right solution is to redirect the output of all the tasks so that you can inspect the log files to see the details of the failure.

Setting up the output redirection, etc, is more work. A rake task would be nice so that all the logic can be written in Ruby rather than in irtests.bat. I will leave this for another day. Let’s see how far irtests.bat can scale…

thanks for the change! And yeah, let’s see how the batch can scale. If
it gets boring, I’ll try to take a stab at redirecting output.

cheers

– Thibaut