Forum: Ruby testy.rb - ruby testing that's mad at the world

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
1bac2e65d64faf472cf2ebc94f0f5ee0?d=identicon&s=25 Ara Howard (ahoward)
on 2009-03-29 00:01
(Received via mailing list)
NAME
   testy.rb

DESCRIPTION
   a BDD testing framework for ruby that's mad at the world and plans
to kick
   it's ass in 80 freakin lines of code

SYNOPSIS
   Testy.testing 'your code' do
     test 'some behaviour' do |result|
       ultimate = Ultimate.new
       result.check :name, :expect => 42, :actual => ultimate.answer
     end
   end

PRINCIPLES AND GOALS
   . it.should.not.merely.be.a.unit.testing.with.a.clever.dsl

   . testing should not require learning a framework.  ruby is a great
   framework so testy uses it instead, requiring programmers learn
exactly 2
   new method calls

   . testing loc should not dwarf those of the application

   . testing framework loc should not dwarf those of the application

   . testing frameworks should *never* alter ruby built-ins nor add
methods to
   Object, Kernel, .et al

   . the output of tests should be machine parsable for reporting and
ci tools
   to easily integrate with

   . the output of tests should be beautiful so that humans can read it

   . the shape of the test file should not insult the programmer so
that tests
   can double as sample code

   . the testing framework should never alter exception semantics

   . hi-jacking at_exit sucks ass

   . the exit status of running a test suite should indicate the
degree of it's
   failure state: the more failures the higher the exit status

   . sample code should easily be able to double as a test suite,
including
   it's output

   . testing should improve your code and help your users, not make
you want to
   kill yourself

   . using a format that aligns in terminal is sanity saving when
comparing
   output

   . testing frameworks should provide as few shortcuts for making
brittle
   tightly coupled tests as possible

   . test suites should be able to be created, manipulated, have their
output
   streamed to different ports, and even tested themselves - they
should be
   plain ol objects under the hood

SAMPLES

   <========< samples/a.rb >========>

   ~ > cat samples/a.rb

     # simple use of testy involves simply writing code, and recording
the result
     # you expect against the actual result
     #
     # notice that the output is very pretty and that the exitstatus
is 0 when all
     # tests pass
     #
       require 'testy'

       Testy.testing 'the kick-ass-ed-ness of testy' do

         test 'the ultimate answer to life' do |result|
           list = []

           list << 42
           result.check :a, :expect => 42, :actual => list.first

           list << 42.0
           result.check :b, 42.0, list.last
         end

       end

   ~ > ruby samples/a.rb #=> exitstatus=0

     ---
     the kick-ass-ed-ness of testy:
       the ultimate answer to life:
         success:
           a: 42
           b: 42.0


   <========< samples/b.rb >========>

   ~ > cat samples/b.rb

     # testy will handle unexpected results and exceptions thrown in
your code in
     # exactly the same way - by reporting on them in a beautiful
fashion and
     # continuing to run other tests.  notice, however, that an
unexpected result
     # or raised exception will cause a non-zero exitstatus (equalling
the number
     # of failed tests) for the suite as a whole.  also note that
previously
     # accumulate expect/actual pairs are still reported on in the
error report.
     #
       require 'testy'

       Testy.testing 'the exception handling of testy' do

         test 'raising an exception' do |result|
           list = []

           list << 42
           result.check :a, :expect => 42, :actual => list.first

           list.method_that_does_not_exist
         end

         test 'returning unexpected results' do |result|
           result.check 'a', 42, 42
           result.check :b, :expect => 'forty-two', :actual => 42.0
         end

       end

   ~ > ruby samples/b.rb #=> exitstatus=2

     ---
     the exception handling of testy:
       raising an exception:
         failure:
           error:
             class: NoMethodError
             message: undefined method `method_that_does_not_exist'
for [42]:Array
             backtrace:
             - samples/b.rb:18
             - ./lib/testy.rb:65:in `call'
             - ./lib/testy.rb:65:in `run'
             - /opt/local/lib/ruby/site_ruby/1.8/orderedhash.rb:65:in
`each'
             - /opt/local/lib/ruby/site_ruby/1.8/orderedhash.rb:65:in
`each'
             - ./lib/testy.rb:61:in `run'
             - ./lib/testy.rb:89:in `testing'
             - samples/b.rb:10
           expect:
             a: 42
           actual:
             a: 42
       returning unexpected results:
         failure:
           expect:
             a: 42
             b: forty-two
           actual:
             a: 42
             b: 42.0




a @ http://codeforpeople.com/
B1b1d33e0655e841d4fd8467359c58d0?d=identicon&s=25 Yossef Mendelssohn (Guest)
on 2009-03-29 00:34
(Received via mailing list)
On Mar 28, 6:01 pm, "ara.t.howard" <ara.t.how...@gmail.com> wrote:
>        returning unexpected results:
>          failure:
>            expect:
>              a: 42
>              b: forty-two
>            actual:
>              a: 42
>              b: 42.0

You call this beautiful, but I don't understand it. This says that 'a'
is okay and 'b' isn't, right? Maybe it's not so much that I don't
understand it as I don't really like it.

Frankly, I find it rather ironic that you're writing a testing
framework and seemingly advocating BDD. Maybe things have changed
mightily in these heady recent times.

I like some of what you have as points, like the output should be
readable ("beautiful" is a little subjective), and of course that
tests should improve your code. The framework points, about the
framework not being huge and not contributing to brittle tests are
good, and the exit status is interesting. Personally, I live with a
couple of methods (as few as possible) on Object and Kernel so writing
the tests doesn't make me want to kill myself.

I used RSpec for a long time, and still do with some projects. I've
switched bacon for my personal projects, and I love it. As for
mocking, which is necessary in some cases if you want to test without
turning into a serial killer, mocha with RSpec, facon with bacon.
0814ff69e309b033508b3c7f363236fd?d=identicon&s=25 Ian Trudel (backorder)
on 2009-03-29 00:45
Yossef Mendelssohn wrote:
> You call this beautiful, but I don't understand it. This says that 'a'
> is okay and 'b' isn't, right? Maybe it's not so much that I don't
> understand it as I don't really like it.

Actually, it seems to be YAML format. It's readable and can be parsed.

Regards,
Ian
Aafa8848c4b764f080b1b31a51eab73d?d=identicon&s=25 Phlip (Guest)
on 2009-03-29 01:00
(Received via mailing list)
ara.t.howard wrote:

>    . it.should.not.merely.be.a.unit.testing.with.a.clever.dsl

How about you simply let the programmer write anything they want, and
then if it
returns false or nil you rip their entire expression and report the name
and
value of every variable within it?

   assert{ foo > 41.8 and foo < 42.1 } => false

      foo => 40.9

Notice we didn't need to tell the assertion the variable's name was
'foo'. This
rewards programmers for writing readable code. You get the maximum
reflection
for the leanest statements.

Honestly I think it's _lack_ of a nauseatingly kewt DSL that inhibits
adoption
of assert{ 2.0 }...
A87f7a014c624587fab0d3d78c5b9c18?d=identicon&s=25 Bil Kleb (Guest)
on 2009-03-29 05:30
(Received via mailing list)
ara.t.howard wrote:
> NAME
>    testy.rb
>
> DESCRIPTION
>    a BDD testing framework for ruby that's mad at the world and plans
> to kick it's ass in 80 freakin lines of code

It's nice to see you finally riffing on testing.

Later,
1bac2e65d64faf472cf2ebc94f0f5ee0?d=identicon&s=25 Ara Howard (ahoward)
on 2009-03-29 06:28
(Received via mailing list)
On Mar 28, 2009, at 5:45 PM, Ian Trudel wrote:


> Actually, it seems to be YAML format. It's readable and can be parsed.


bingo.  emphasis on the latter.  think unix pipes.


a @ http://codeforpeople.com/
1bac2e65d64faf472cf2ebc94f0f5ee0?d=identicon&s=25 Ara Howard (ahoward)
on 2009-03-29 06:29
(Received via mailing list)
On Mar 28, 2009, at 5:33 PM, Yossef Mendelssohn wrote:

> You call this beautiful, but I don't understand it. This says that 'a'
> is okay and 'b' isn't, right? Maybe it's not so much that I don't
> understand it as I don't really like it.


it's a valid complaint.  but compare it to what you'll get in most
frameworks and consider that, by beautiful, i mean that a YAML.load
can slurp the entire set of expect vs actual.  i considered a delta
style format:

   diff
     a:
       expect: 42
       actual: 42.0
     b:
       expect: 43
       actual: 43.0

but it seems very hard to reconstruct for downstream filters.  i'm
open to suggestion on format though.  requirements are

  . readable by humans
         . easily parsed by computers

basically that means some yaml format.  honestly open to suggestion
here...


>
>
> Frankly, I find it rather ironic that you're writing a testing
> framework and seemingly advocating BDD. Maybe things have changed
> mightily in these heady recent times.
>


i personally don't think so, i think the community took a wrong turn,
from wikipedia (http://en.wikipedia.org/wiki/
Behavior_Driven_Development)


"
The practices of BDD include:

     * Involving stakeholders in the process through outside-in
software development

     * Using examples to describe the behavior of the application, or
of units of code
     * Automating those examples to provide quick feedback and
regression testing

     * In software tests, using 'should' to help clarify
responsibility and allow the software's functionality to be questioned
     * Test use 'ensure' to differentiate outcomes in the scope of the
code in question from side-effects of other elements of code.
     * Using mocks to stand-in for modules of code which have not yet
been written


i have major issues with points two and three wrst to most ruby
testing frameworks.  one of the main points of testy is to combine
examples with testing.  rspec and all the others do not serve as
examples unless you are a ruby master.  that is to say they introduce
too many additions to the code that's supposed to be an example to
really preserve it's 'exampleness'.  and of course the output is
utterly useless to normal humans.  if a framework provides 1000
asset_xxxxxxxx methods ad nausea then the point of the code - it's
level of example-good-ness - is lost to mere mortals


> mocking, which is necessary in some cases if you want to test without
> turning into a serial killer, mocha with RSpec, facon with bacon.



this will summarize where my thoughts are on that


cfp:~/redfission > find vendor/gems/{faker,mocha,thoughtbot}* -type f|
xargs -n1 cat|wc -l
    24255

cfp:~/redfission > find app -type f|xargs -n1 cat|wc -l
     1828

rspec and co might be fine but seriously, the above is insane right?


kind regards.


a @ http://codeforpeople.com/
1bac2e65d64faf472cf2ebc94f0f5ee0?d=identicon&s=25 Ara Howard (ahoward)
on 2009-03-29 06:31
(Received via mailing list)
On Mar 28, 2009, at 6:00 PM, Phlip wrote:

>     foo => 40.9
>
> Notice we didn't need to tell the assertion the variable's name was
> 'foo'. This rewards programmers for writing readable code. You get
> the maximum reflection for the leanest statements.
>
> Honestly I think it's _lack_ of a nauseatingly kewt DSL that
> inhibits adoption of assert{ 2.0 }...
>

that's interesting indeed.  one of my goals with testy is that output
is meaningful both for computers and humans and, for that, yaml is tops.

still - reporting on the context the errors was thown raised from is
quite interesting.  you are basically saying report binding not
stacktrace right?


a @ http://codeforpeople.com/
1bac2e65d64faf472cf2ebc94f0f5ee0?d=identicon&s=25 Ara Howard (ahoward)
on 2009-03-29 06:32
(Received via mailing list)
On Mar 28, 2009, at 9:30 PM, Bil Kleb wrote:

> It's nice to see you finally riffing on testing.



;-) more to come


a @ http://codeforpeople.com/
Ae36591847393e58ff189704f5eb18f2?d=identicon&s=25 Jeremy Hinegardner (Guest)
on 2009-03-29 20:16
(Received via mailing list)
On Sun, Mar 29, 2009 at 08:01:18AM +0900, ara.t.howard wrote:
>
>   . the exit status of running a test suite should indicate the degree of
>     it's failure state: the more failures the higher the exit status
>

Up to a limit of course.  how about exiting with the percentage ?  Exit
status
is limited to 256 values, so you can make it exit 0 with lots of
failures:

    http://gist.github.com/87480

enjoy,

-jeremy
Aafa8848c4b764f080b1b31a51eab73d?d=identicon&s=25 Phlip (Guest)
on 2009-03-29 20:40
(Received via mailing list)
Jeremy Hinegardner wrote:
> On Sun, Mar 29, 2009 at 08:01:18AM +0900, ara.t.howard wrote:
>>   . the exit status of running a test suite should indicate the degree of
>>     it's failure state: the more failures the higher the exit status
>>
>
> Up to a limit of course.  how about exiting with the percentage ?  Exit status
> is limited to 256 values, so you can make it exit 0 with lots of failures:

Again: If you have any reason to _count_ the errors, you are already
absolutely
screwed anyway...
4299e35bacef054df40583da2d51edea?d=identicon&s=25 James Gray (bbazzarrakk)
on 2009-03-29 21:21
(Received via mailing list)
On Mar 29, 2009, at 1:39 PM, Phlip wrote:

>
> Again: If you have any reason to _count_ the errors, you are already
> absolutely screwed anyway...

I really feel counting the errors and reading the output are both
things better handled by defining a good interface for the results
writer.  If I could just define some trivial class with methods like
test_passed(), test_failed(), test_errored_out(), and tests_finished()
then just plug that in, I could easily do anything I want.

James Edward Gray II
Aafa8848c4b764f080b1b31a51eab73d?d=identicon&s=25 Phlip (Guest)
on 2009-03-29 21:30
(Received via mailing list)
James Gray wrote:

> I really feel counting the errors and reading the output are both
> things better handled by defining a good interface for the results
> writer.  If I could just define some trivial class with methods like
> test_passed(), test_failed(), test_errored_out(), and tests_finished()
> then just plug that in, I could easily do anything I want.

What, besides instantly fix it (or revert) do you want to do with an
error
message from a broken test?
4299e35bacef054df40583da2d51edea?d=identicon&s=25 James Gray (bbazzarrakk)
on 2009-03-29 21:50
(Received via mailing list)
On Mar 29, 2009, at 2:29 PM, Phlip wrote:

> error message from a broken test?
The default writer wants to write them out to the console for the user
to see.

In TextMate, I would override that behavior to color the error
messages red in our command output window and hyperlink the stack
trace back into TextMate documents.

James Edward Gray II
753dcb78b3a3651127665da4bed3c782?d=identicon&s=25 Brian Candler (candlerb)
on 2009-03-29 21:51
Ara Howard wrote:
>        result.check :name, :expect => 42, :actual => ultimate.answer

I'm afraid I'm missing something. Why is this better than

  assert_equal 42, ultimate.answer, "name"
?

>    . testing should improve your code and help your users, not make
> you want to
>    kill yourself

Hear hear to that!

> requiring programmers learn exactly 2 new method calls

Well, it would be nice to know what those 2 methods calls were, and
their semantics, without reverse-engineering the code. Are these
Testy.testing and Result#check ?

I did find the gem, and installed it, but the generated rdoc is entirely
free of comments or explanation.

The two samples don't really help me understand why testy is good or how
to use it effectively, since they are both more verbose that what I
would have written in test/unit to get the same result.

How would I do something equivalent to these?

   assert_raises(RuntimeError) { somecode }

   assert_match /error/i, response.body

I think I'd also miss the ability to have setup and teardown before each
test (something which 'shoulda' makes very simple and effective).

Please don't get me wrong - I'm absolutely interested in something which
will make testing simpler and easier, if I can understand how to use it
effectively.

Regards,

Brian.
Aafa8848c4b764f080b1b31a51eab73d?d=identicon&s=25 Phlip (Guest)
on 2009-03-29 22:10
(Received via mailing list)
Brian Candler wrote:
> Ara Howard wrote:
>>        result.check :name, :expect => 42, :actual => ultimate.answer
>
> I'm afraid I'm missing something. Why is this better than
>
>   assert_equal 42, ultimate.answer, "name"
> ?

Or even less:

   name = 42
   assert{ name == ultimate.answer }
87939140bb4c3ec804b13005165fd7df?d=identicon&s=25 Lake Denman (ldenman)
on 2009-03-29 22:44
Ara Howard wrote:
> NAME
>    testy.rb
>
> DESCRIPTION
>    a BDD testing framework for ruby that's mad at the world and plans
> to kick
>    it's ass in 80 freakin lines of code
>
> SYNOPSIS
>    Testy.testing 'your code' do
>      test 'some behaviour' do |result|
>        ultimate = Ultimate.new
>        result.check :name, :expect => 42, :actual => ultimate.answer
>      end
>    end
>

Hey Ara,

Interesting post and project. Nice work! I just tried to test testy.rb
out and, maybe i'm overlooking something obvious, but when i run:

 ruby testy_test.rb (http://pastie.org/430735)

 Desktop$: ruby testy_testing.rb
 ---
 naming a student:
   retrieving a student's name:
     failure:
       expect:
         name:
       actual:
         name: Lake
   giving a student a name:
     failure:
       expect:
         name:
      actual:
        name: Jake


You see, the output does not contain the second Testy.testing()
results...

What gives?


Thanks,

Lake
1bac2e65d64faf472cf2ebc94f0f5ee0?d=identicon&s=25 Ara Howard (ahoward)
on 2009-03-29 23:29
(Received via mailing list)
On Mar 29, 2009, at 12:16 PM, Jeremy Hinegardner wrote:

> Up to a limit of course.  how about exiting with the percentage ?
> Exit status
> is limited to 256 values, so you can make it exit 0 with lots of
> failures:
>
>    http://gist.github.com/87480
>
> enjoy,
>
> -jeremy

good catch - that was uh, tired of me ;-)

i'll add precent now.

a @ http://codeforpeople.com/
1bac2e65d64faf472cf2ebc94f0f5ee0?d=identicon&s=25 Ara Howard (ahoward)
on 2009-03-29 23:29
(Received via mailing list)
On Mar 29, 2009, at 12:39 PM, Phlip wrote:

> Again: If you have any reason to _count_ the errors, you are already
> absolutely screwed anyway...



indeed.  i was vaugely thinking of a status report with failure
reported by severity.  really just an idea at this point.


a @ http://codeforpeople.com/
1bac2e65d64faf472cf2ebc94f0f5ee0?d=identicon&s=25 Ara Howard (ahoward)
on 2009-03-29 23:31
(Received via mailing list)
On Mar 29, 2009, at 1:29 PM, Phlip wrote:

> What, besides instantly fix it (or revert) do you want to do with an
> error message from a broken test?


report it in your ci tool


a @ http://codeforpeople.com/
1bac2e65d64faf472cf2ebc94f0f5ee0?d=identicon&s=25 Ara Howard (ahoward)
on 2009-03-30 00:23
(Received via mailing list)
On Mar 29, 2009, at 1:51 PM, Brian Candler wrote:

> Ara Howard wrote:
>>       result.check :name, :expect => 42, :actual => ultimate.answer
>
> I'm afraid I'm missing something. Why is this better than
>
>  assert_equal 42, ultimate.answer, "name"


you can basically do that too, but i continually forget which is
expected and which is actual and, as you know, that's a slippery error
to track down at times.


>>   . testing should improve your code and help your users, not make
>> you want to
>>   kill yourself
>
> Hear hear to that!
>

;-)

>> requiring programmers learn exactly 2 new method calls
>
> Well, it would be nice to know what those 2 methods calls were, and
> their semantics, without reverse-engineering the code. Are these
> Testy.testing and Result#check ?
>

yes.  that is it.

> I did find the gem, and installed it, but the generated rdoc is
> entirely
> free of comments or explanation.
>


what are these comments you speak of?

seriously, this is an experiment at this point but they will be
forthcoming if it sprouts wings



>
at this point my feeling is that breaks ths concept of BDD

"Using examples to describe the behavior of the application, or of
units of code"

because littering the example code with esoteric testing framework
voodoo turns it into code in the testing language that does not
resemble how people might actually use the code - at least not without
mentally unraveling considerable indirection.  i always end up writing
both samples and tests - one of the goals of testy is that, by having
a really simple interface and really simple human friendly output we
can just write examples that double as tests.  in the latest testy you
can do this



cfp:~/src/git/testy > cat a.rb
require 'testy'

Testy.testing 'my lib' do

   test 'foo' do |result|
     list = [42]
     result.check :fooness, :expect => 42, :actual => list.last
   end

   test 'bar' do |result|
     list = [42.0]
     result.check :barness, :expect => 42.0, :actual => list.last
   end

end


get a listing of which tests/examples i can run

cfp:~/src/git/testy > ruby -I lib a.rb --list
---
- foo
- bar


run one of them (actually regex matching so you can select more that
one)


cfp:~/src/git/testy > ruby -I lib a.rb bar
---
my lib:
   bar:
     success:
       barness: 42.0


you can also do something like this (experimental) to just make a
simple example

cfp:~/src/git/testy > cat a.rb
require 'testy'

Testy.testing 'my lib' do

   test 'just an example of summing an array using inject' do
     a = 1,2
     a.push 3
     sum = a.inject(0){|n,i| n += i}
   end

end



cfp:~/src/git/testy > ruby -I lib a.rb 'just an example'
---
my lib:
   just an example of summing an array using inject:
     success: 6


testy will just display the return value if no results are explicitly
checked but, of course, exceptions are still reported and cause a
failed test in the normal way.


so the goal is making it even easier to have a user play with your
tests/examples to see how they work, and even to allow simple examples
to be integrated with your test suite so you make sure you samples
still run without error too.  of course you can do this with test/unit
or rspec but the output isnt' friendly in the least - not from the
perspective of a user trying to learn a library, nor is it useful to
computers because it cannot be parsed - basically it's just vomiting
stats and backtraces to the console that are hard for people to read
and hard for computers to read.  surely i am not the only one that
sometimes resorts to  factoring out a failing test in a separate
program because test/unit and rspec output is too messy to play nice
with instrumenting code?  and that's not even getting to what they do
with at_exit exception raising...



> I think I'd also miss the ability to have setup and teardown before
> each
> test (something which 'shoulda' makes very simple and effective).


yeah that's on deck for sure.  i *do* really like contexts with
shoulda.  but still


cfp:/opt/local/lib/ruby/gems/1.8/gems/thoughtbot-shoulda-2.9.1 > find
lib/ -type f|xargs -n1 cat|wc -l
     3910


if we accept the research and assume that bugs scale linerarly with
the # of lines of code this is not good for robustness.  this is one
of my main gripes with current ruby testing - my current rails app has
about 1000 lines of code and 25,000 lines of testing framework!


>
>
> Please don't get me wrong - I'm absolutely interested in something
> which
> will make testing simpler and easier, if I can understand how to use
> it
> effectively.


feedback is awesome - this is an new idea i'm just fleshing out so i
really appreciate it.

cheers.

a @ http://codeforpeople.com/
1bac2e65d64faf472cf2ebc94f0f5ee0?d=identicon&s=25 Ara Howard (ahoward)
on 2009-03-30 00:34
(Received via mailing list)
On Mar 29, 2009, at 2:09 PM, Phlip wrote:

> Or even less:
>
>  name = 42
>  assert{ name == ultimate.answer }


well, for one that won't run ;-) assert takes and arg and not a
block... but that aside

cfp:~ > cat a.rb
require 'test/unit'

class TC < Test::Unit::TestCase
   def test_with_shitty_error_reporting
     name = 42
     assert name == 'forty-two'
   end
end




cfp:~ > ruby a.rb
Loaded suite a
Started
F
Finished in 0.007649 seconds.

   1) Failure:
test_with_shitty_error_reporting(TC) [a.rb:6]:
<false> is not true.

1 tests, 1 assertions, 1 failures, 0 errors


the error message '<false> is not true', on 8th line of output (out of
possibly thousands) makes me want to hunt the programmer down that
wrote that club him to death with a 2x4.  the assert api facilities
insanity for the code maintainer

now, in testy

cfp:~/src/git/testy > ruby -I lib a.rb
---
my lib:
   name compare:
     failure:
       expect:
         name: forty-two
       actual:
         name: 42


my thinking, currently, is that the test writer should be forced to

   . name the test suite
     . name the test
       . name the check

because doing these three things allows for informative error
reporting.  testing just shouldn't facilitate obfusicating what went
wrong - imho.

cheers.

a @ http://codeforpeople.com/
1bac2e65d64faf472cf2ebc94f0f5ee0?d=identicon&s=25 Ara Howard (ahoward)
on 2009-03-30 00:36
(Received via mailing list)
On Mar 29, 2009, at 2:44 PM, Lake Denman wrote:

> You see, the output does not contain the second Testy.testing()

my stupidity.  i hadn't considered having more that one in a file ;-)
actually a bit tricky with the exit status issue - but i'll roll out a
fix.

/me hangs head

a @ http://codeforpeople.com/
E1d641bfe4071a5413bac781f06d3fd1?d=identicon&s=25 Sean O'halpin (sean)
on 2009-03-30 01:04
(Received via mailing list)
On Sun, Mar 29, 2009 at 11:33 PM, ara.t.howard <ara.t.howard@gmail.com>
wrote:
> that aside
Phlip is referring to his own assert2, not the Test::Unit one. You
should check it out - it's really a great idea.
Aafa8848c4b764f080b1b31a51eab73d?d=identicon&s=25 Phlip (Guest)
on 2009-03-30 01:15
(Received via mailing list)
ara.t.howard wrote:
> On Mar 29, 2009, at 1:29 PM, Phlip wrote:
>
>> What, besides instantly fix it (or revert) do you want to do with an
>> error message from a broken test?
>
>
> report it in your ci tool

Don't integrate a broken build!

Are you implying that an incremental build server, such as
CruiseControl, should
have enough errors it needs to count and report them?
Aafa8848c4b764f080b1b31a51eab73d?d=identicon&s=25 Phlip (Guest)
on 2009-03-30 01:21
(Received via mailing list)
Sean O'Halpin wrote:

> Phlip is referring to his own assert2, not the Test::Unit one. You
> should check it out - it's really a great idea.

Tx, but it sucks! Here's the ideal test case (regardless of its .should
or it{}
syntax:

   test_activate
      x = assemble()
      g = x.activate()
      g == 42
   end

The test should simply reflect the variables and values of everything
after the
activate line. Anything less is overhead. I _only_ want to spend my time
setting
up situations and equating their results. No DSLs or reflection or
anything!

Now, why can't our language just /do/ that for us? It knows everything
it needs...
1bac2e65d64faf472cf2ebc94f0f5ee0?d=identicon&s=25 Ara Howard (ahoward)
on 2009-03-30 01:41
(Received via mailing list)
On Mar 29, 2009, at 5:01 PM, Sean O'Halpin wrote:

> Phlip is referring to his own assert2, not the Test::Unit one. You
> should check it out - it's really a great idea.

ah - reports via the binding i assume - that *is* a nice idea.

a @ http://codeforpeople.com/
Ae36591847393e58ff189704f5eb18f2?d=identicon&s=25 Jeremy Hinegardner (Guest)
on 2009-03-30 01:48
(Received via mailing list)
On Mon, Mar 30, 2009 at 07:22:35AM +0900, ara.t.howard wrote:
> if we accept the research and assume that bugs scale linerarly with the #
> of lines of code this is not good for robustness.  this is one of my main
> gripes with current ruby testing - my current rails app has about 1000
> lines of code and 25,000 lines of testing framework!

This, of course, just means you are not writing big enough rails apps
:-).
Aafa8848c4b764f080b1b31a51eab73d?d=identicon&s=25 Phlip (Guest)
on 2009-03-30 02:05
(Received via mailing list)
ara.t.howard wrote:
> On Mar 29, 2009, at 5:01 PM, Sean O'Halpin wrote:
>
>> Phlip is referring to his own assert2, not the Test::Unit one. You
>> should check it out - it's really a great idea.
>
> ah - reports via the binding i assume - that *is* a nice idea.

If only the binding were enough...

You gotta use Ripper to convert the block to opcodes, then eval them one
by one.
1bac2e65d64faf472cf2ebc94f0f5ee0?d=identicon&s=25 Ara Howard (ahoward)
on 2009-03-30 02:10
(Received via mailing list)
On Mar 29, 2009, at 5:14 PM, Phlip wrote:

> Don't integrate a broken build!
>
> Are you implying that an incremental build server, such as
> CruiseControl, should have enough errors it needs to count and
> report them?



i'm saying that whatever you are doing with automated tests, be it ci
or simply editor support, it's vastly easier for people to build tools
if the output is parsed in one line of YAML.load - that's all.

cheers.

a @ http://codeforpeople.com/
1bac2e65d64faf472cf2ebc94f0f5ee0?d=identicon&s=25 Ara Howard (ahoward)
on 2009-03-30 02:11
(Received via mailing list)
On Mar 29, 2009, at 5:47 PM, Jeremy Hinegardner wrote:

> This, of course, just means you are not writing big enough rails
> apps :-).


doh! i knew i was doing it wrong!


a @ http://codeforpeople.com/
1bac2e65d64faf472cf2ebc94f0f5ee0?d=identicon&s=25 Ara Howard (ahoward)
on 2009-03-30 02:12
(Received via mailing list)
On Mar 29, 2009, at 6:04 PM, Phlip wrote:

> If only the binding were enough...
>
> You gotta use Ripper to convert the block to opcodes, then eval them
> one by one.



dude - that is *hardcore*  ;-)

a @ http://codeforpeople.com/
Aafa8848c4b764f080b1b31a51eab73d?d=identicon&s=25 Phlip (Guest)
on 2009-03-30 02:13
(Received via mailing list)
Jeremy Hinegardner wrote:
> On Mon, Mar 30, 2009 at 07:22:35AM +0900, ara.t.howard wrote:
>> if we accept the research and assume that bugs scale linerarly with the #
>> of lines of code this is not good for robustness.  this is one of my main
>> gripes with current ruby testing - my current rails app has about 1000
>> lines of code and 25,000 lines of testing framework!

I incredibly don't understand that. Our apps are big by Rails standards,
yet our
test:code ratio is only 2.5:1. That's mostly because we can't "refactor"
the
tests too much, or they are hard to read.
4feed660d3728526797edeb4f0467384?d=identicon&s=25 Bill Kelly (Guest)
on 2009-03-30 02:31
(Received via mailing list)
From: "ara.t.howard" <ara.t.howard@gmail.com>
>
> On Mar 29, 2009, at 6:04 PM, Phlip wrote:
>
>> If only the binding were enough...
>>
>> You gotta use Ripper to convert the block to opcodes, then eval them
>> one by one.
>
> dude - that is *hardcore*  ;-)

I think some examples might help.  These are from Phlip's
January 26, 2008 post Re: assert_{ 2.0 } on the XP list:


assert_{ 'a topic' == topics['first'] }    output:

  "a topic" == ( topics["first"] )        --> false
       topics                --> {"first"=>"wrong topic"}
        ( topics["first"] )  --> "wrong topic".



assert_{ [false] == shrew.accessories.map(&:active).uniq }  output:

  [true] == shrew.accessories.map(&:active).uniq    --> false
     shrew                                --> User 4
     shrew.accessories                    --> [Accessory 501]
     shrew.accessories.map(&:active)      --> [false]
     shrew.accessories.map(&:active).uniq --> [false].


It's badass.

Phlip's work on assert_{ 2.0 } immediately left me with two
impressons:

  1. Having seen this, why would we ever accept anything less
     awesome from any testing rig?  :)

  2. The facilities needed to implement this kind of reflection
     should be part of ruby core.

....right?  :)


Regards,

Bill
1bac2e65d64faf472cf2ebc94f0f5ee0?d=identicon&s=25 Ara Howard (ahoward)
on 2009-03-30 02:38
(Received via mailing list)
On Mar 29, 2009, at 6:30 PM, Bill Kelly wrote:

>
> ....right?  :)

yes.  it addresses one of my biggest issues with the current
standards: too many assertion methods and crappy reporting.

a @ http://codeforpeople.com/
1bac2e65d64faf472cf2ebc94f0f5ee0?d=identicon&s=25 Ara Howard (ahoward)
on 2009-03-30 02:44
(Received via mailing list)
On Mar 29, 2009, at 6:09 PM, Phlip wrote:

> I incredibly don't understand that. Our apps are big by Rails
> standards, yet our test:code ratio is only 2.5:1. That's mostly
> because we can't "refactor" the tests too much, or they are hard to
> read.


i'm not talking about code to test ratios, but code to test
*framework* ratios.  of course these numbers are a little bit high
(whitespace, comments, etc) but




cfp:/opt/local/lib/ruby/gems/1.8/gems > for gem in thoughtbot-* rspec*
faker* mocha*;do echo $gem;find $gem/lib -type f|grep .rb|xargs -n1
cat|wc -l;done
thoughtbot-factory_girl-1.2.0
      937
thoughtbot-shoulda-2.9.1
     3854
rspec-1.1.12
     8773
rspec-1.1.3
     7785
rspec-1.1.4
     8083
faker-0.3.1
      299
mocha-0.9.5
     3294


most people would consider an 8000 line rails app 'large'.

a @ http://codeforpeople.com/
4feed660d3728526797edeb4f0467384?d=identicon&s=25 Bill Kelly (Guest)
on 2009-03-30 02:44
(Received via mailing list)
From: "Bill Kelly" <billk@cts.com>
>
> assert_{ [false] == shrew.accessories.map(&:active).uniq }  output:
>
>  [true] == shrew.accessories.map(&:active).uniq    --> false
>     shrew                                --> User 4
>     shrew.accessories                    --> [Accessory 501]
>     shrew.accessories.map(&:active)      --> [false]
>     shrew.accessories.map(&:active).uniq --> [false].

Er, sorry, I think I failed at reconstructing that from
the original email.

Presumably should be:

assert_{ [true] == shrew.accessories.map(&:active).uniq }  output:

  [true] == shrew.accessories.map(&:active).uniq    --> false
     shrew                                --> User 4
     shrew.accessories                    --> [Accessory 501]
     shrew.accessories.map(&:active)      --> [false]
     shrew.accessories.map(&:active).uniq --> [false].


Regards,

Bill
Aafa8848c4b764f080b1b31a51eab73d?d=identicon&s=25 Phlip (Guest)
on 2009-03-30 03:25
(Received via mailing list)
ara.t.howard wrote:

> thoughtbot-factory_girl-1.2.0
>       937
> thoughtbot-shoulda-2.9.1
>      3854
> rspec-1.1.12
>      8773

Well, the good news is Ruby is essentially a "post-Agile" environment.
Your boss
can no longer fire you for writing "too many" unit tests. Many
environments,
even in this day and age, have not matured so far!

The other good news is our enthusiasm has made us forget many of the
original
Agile ground rules. TDD requires rapid feedback over tiny code changes,
close to
the tested code, and setting up a Cucumber suite with a handful of mock
objects
does not qualify as rapid feedback, running close to the code. Mocks are
indeed
a way to help your code's design couple together. So BigAgileUpFront,
Ruby-style, might indeed be a big risk here.

If it isn't, we have to learn from it, because Ruby's BDD community
represents
an amazing sight. The original BDD, FIT and Fitnesse, required hordes of
overpaid consultants to install, typically at big-iron sites. Ruby BDD,
by
contrast, is just as light, free, and community-supported as Rails
itself, and I
suspect it's the leading edge of "customer testing" research in general.
Aafa8848c4b764f080b1b31a51eab73d?d=identicon&s=25 Phlip (Guest)
on 2009-03-30 03:31
(Received via mailing list)
>>> You gotta use Ripper to convert the block to opcodes, then eval them
>>> one by one.
>> dude - that is *hardcore*  ;-)

Who was just fussing about "too many lines of code in the framework"?

>   2. The facilities needed to implement this kind of reflection
>      should be part of ruby core.

Right:

   report = journal do
              my_code()
            end

Inside journal{}, everything the Ruby VM does gets packed into a report.
Then
you rip the report to get back to your details. That's what assert{ 2.0
}
_can't_ do, so it goes in two passes.

The first pass is a normal block.call, to detect success or failure.
This is so
any bugs in the second pass don't throw the test run.

The second pass puts the block into Ripper to decompile it, and then the
fun starts.
1bac2e65d64faf472cf2ebc94f0f5ee0?d=identicon&s=25 Ara Howard (ahoward)
on 2009-03-30 03:37
(Received via mailing list)
On Mar 29, 2009, at 7:29 PM, Phlip wrote:

> Who was just fussing about "too many lines of code in the framework"?

i meant hardcore in a good way - really!  ;-)

a @ http://codeforpeople.com/
753dcb78b3a3651127665da4bed3c782?d=identicon&s=25 Brian Candler (candlerb)
on 2009-03-30 10:12
Ara Howard wrote:
>>  assert_equal 42, ultimate.answer, "name"
>
> you can basically do that too, but i continually forget which is
> expected and which is actual and, as you know, that's a slippery error
> to track down at times.

Perhaps - but it's one rule that only needs to be learned once.

I notice that testy supports check <name>, <expected>, <actual> too.

Testy does (intentially) force you to name your tests, whereas
Test::Unit will happily let you write

  check <expected>, <actual>

I really don't like having to name each assertion, maybe because I'm
lazy or maybe because it feels like DRY violation. I've already said
what I want to compare, why say it again?

> because littering the example code with esoteric testing framework
> voodoo turns it into code in the testing language that does not
> resemble how people might actually use the code

I agree with this. This is why I absolutely prefer Test::Unit (and
Shoulda on top of that) over Rspec.

> i always end up writing
> both samples and tests - one of the goals of testy is that, by having
> a really simple interface and really simple human friendly output we
> can just write examples that double as tests.

Hmm, this is probably an argument *for* having a DSL for assertions - to
make the assertions read as much like example code ("after running this
example, you should see that A == B and C < D")

Neither

  result.check "bar attribute", :expected => 123, :actual => res.bar

nor

  assert_equal 123, res.bar, "bar attribute"

reads particularly well here, I think. Ideally it should be as simple as
possible to write these statements of expectation. How about some eval
magic?

  expect[
    "res.foo == 456",
    "res.bar == 123",
    "res.baz =~ /wibble/"
  ]

Maybe need to pass a binding here, but you get the idea. (Before someone
else points it out, this is clearly a case which LISP would be very well
suited to handling - the same code to execute can also be displayed in
the results)

The problem here is reporting on expected versus actual, but perhaps you
could split on space and report the value of the first item.

  expected:
    - res.foo == 456
    - res.bar == 123
  unexpected:
    -
      test: res.baz =~ /wibble/
      term: res.baz
      value: "unexpected result"

Going too far this way down this path ends up with rspec, I think.

In fact, I don't really have a problem with writing

  res.foo.should == 456

The trouble is the hundreds of arcane variations on this.

You solve this problem by only having a single test (Result#check), and
indeed if rspec only had a single method (should_equal) that would be
fairly clean too. However this is going to lead to awkwardness when you
want to test for something other than equality: e.g.

   res = (foo =~ /error/) ? true : false
   result.check "foo should contain 'error'", :expected=>true,
:actual=>res

Apart from being hard to write and read, that also doesn't show you the
actual value of 'foo' when the test fails.

Is it worth passing the comparison method?

   result.check "foo should contain 'error'", foo, :=~, /error/

But again this is getting away from real ruby for the assertions, in
which case it isn't much better than

   assert_match /error/, foo, "foo should contain 'error'"

   assert_match /error/, foo  # lazy/DRY version

> get a listing of which tests/examples i can run

Yes, parseable results and test management are extremely beneficial.
Those could be retro-fitted to Test::Unit though (or whatever its
replacement in ruby 1.9 is called)

Getting rid of the at_exit magic is also worth doing.

> you can also do something like this (experimental) to just make a
> simple example
>
> cfp:~/src/git/testy > cat a.rb
> require 'testy'
>
> Testy.testing 'my lib' do
>
>    test 'just an example of summing an array using inject' do
>      a = 1,2
>      a.push 3
>      sum = a.inject(0){|n,i| n += i}
>    end
>
> end

Nice, could perhaps show the (expected) result inline too?

  test 'an example of summing and array using inject' do
    a = 1,2
    a.push 3
    sum = a.inject(0){|n,i| n += i}
  end.<< 6

A bit magical though. Also, we can only test the result of the entire
block, whereas a more complex example will want to create multiple
values and test them all.

> so the goal is making it even easier to have a user play with your
> tests/examples to see how they work, and even to allow simple examples
> to be integrated with your test suite so you make sure you samples
> still run without error too.  of course you can do this with test/unit
> or rspec but the output isnt' friendly in the least - not from the
> perspective of a user trying to learn a library, nor is it useful to
> computers because it cannot be parsed - basically it's just vomiting
> stats and backtraces to the console that are hard for people to read
> and hard for computers to read.  surely i am not the only one that
> sometimes resorts to  factoring out a failing test in a separate
> program because test/unit and rspec output is too messy to play nice
> with instrumenting code?

I agree. Part of the problem is that when one thing is wrong making 20
tests fail, all with their respective backtraces, it can be very hard to
see the wood for the trees. What would be nice would be a folding-type
display with perhaps one line for each failed assertion, and a [+] you
can click on to get the detail for that particular one.

> yeah that's on deck for sure.  i *do* really like contexts with
> shoulda.  but still
>
>
> cfp:/opt/local/lib/ruby/gems/1.8/gems/thoughtbot-shoulda-2.9.1 > find
> lib/ -type f|xargs -n1 cat|wc -l
>      3910
>
>
> if we accept the research and assume that bugs scale linerarly with
> the # of lines of code this is not good for robustness.

I disagree there - not with the research, but the implied conclusion
that you should never use a large codebase. Shoulda works well, and I've
not once found a bizarre behaviour in the testing framework itself that
I've had to debug, so I trust it.

(This is not true of other frameworks though. e.g. I spent a while
tracking this one down:
https://rspec.lighthouseapp.com/projects/5645-rspe...)

> this is one
> of my main gripes with current ruby testing - my current rails app has
> about 1000 lines of code and 25,000 lines of testing framework!

Yeah, but how many lines of Rails framework? :-)

Cheers,

Brian.
753dcb78b3a3651127665da4bed3c782?d=identicon&s=25 Brian Candler (candlerb)
on 2009-03-30 10:27
Ara Howard wrote:
> i always end up writing both samples and tests

Just a thought: I've found myself doing that too, but in particular I've
ran an example externally to be sure it works, then ended up pasting it
into the rdoc. So it would be really cool if rdoc could integrate the
examples directly in the appropriate place(s). I want to be sure that my
examples actually run as advertised, and at the moment I risk them
becoming out of date, so I'm definitely with you on that point.

However, one problem with having runnable examples/specs is that you
often have a lot of setup (and/or mock) code to make something run. This
risks adding too much noise to the examples, so structuring the code so
as to be able to filter that out would be a bonus - i.e. show me just
the example, not the setup or post-conditions. Not easy to do without
ripping the Ruby though.

I find the other kill-yourself part of testing is behavioural testing
with mock objects, but that's out of scope here I think :-)

Regards,

Brian.
1101fee96cbaa23a1d4ec3322b994fff?d=identicon&s=25 Bil Kleb (Guest)
on 2009-03-30 12:27
(Received via mailing list)
On Mar 29, 8:42 pm, "ara.t.howard" <ara.t.how...@gmail.com> wrote:
>
> cfp:/opt/local/lib/ruby/gems/1.8/gems > for gem in thoughtbot-* rspec*  
> faker* mocha*;do echo $gem;find $gem/lib -type f|grep .rb|xargs -n1  
> cat|wc -l;done

Can you please rerun that with flog, flay, or reek in place of find
+xargs+cat+wc?

Thanks,
1101fee96cbaa23a1d4ec3322b994fff?d=identicon&s=25 Bil Kleb (Guest)
on 2009-03-30 12:30
(Received via mailing list)
On Mar 28, 7:01 pm, "ara.t.howard" <ara.t.how...@gmail.com> wrote:
>
>    . hi-jacking at_exit sucks ass

IIRC, PragDave suggested the at_exit trick to Talbott during
the elder days.

Later,
1101fee96cbaa23a1d4ec3322b994fff?d=identicon&s=25 Bil Kleb (Guest)
on 2009-03-30 12:31
(Received via mailing list)
On Mar 30, 4:27 am, Brian Candler <b.cand...@pobox.com> wrote:
>
> Just a thought: I've found myself doing that too, but in particular I've
> ran an example externally to be sure it works, then ended up pasting it
> into the rdoc. So it would be really cool if rdoc could integrate the
> examples directly in the appropriate place(s). I want to be sure that my
> examples actually run as advertised, and at the moment I risk them
> becoming out of date, so I'm definitely with you on that point.

Sounds like you're pining for Python's doctest?

  http://docs.python.org/library/doctest.html

Later,
753dcb78b3a3651127665da4bed3c782?d=identicon&s=25 Brian Candler (candlerb)
on 2009-03-30 12:55
Bil Kleb wrote:
> Sounds like you're pining for Python's doctest?
>
>   http://docs.python.org/library/doctest.html

Ah, now that's a really interesting way of thinking about
examples/testing: in the form of an irb session. You can write your
tests just by mucking about in irb, and when it makes sense, just paste
the output somewhere.

  irb> foo = generate_foo
  => #<Foo:0xb7cd041c @attr1="hello", @attr2="world">
  irb> foo.attr1
  => "hello"
  irb> foo.attr2
  => "world"

Presumably you could avoid the fragile comparison on Object#inspect
output by deleting it from the transcript.

  irb> foo = generate_foo
  ...
  irb> foo.attr1
  => "hello"
  irb> foo.attr2
  => "world"

Writing something which parses that an (re)runs it to verify the output
should be pretty straightforward.

It can also handle the 'assert_raises' case nicely.

  irb> f.attr3
  NoMethodError: undefined method `attr3' for ...

Ara, how about it? :-)
Da57cffaec80533bee0d0446b3ea6fcd?d=identicon&s=25 Michel Demazure (badal)
on 2009-03-30 13:05
Brian Candler wrote:
> Bil Kleb wrote:
>> Sounds like you're pining for Python's doctest?
>>
>>   http://docs.python.org/library/doctest.html
>
> Ah, now that's a really interesting way of thinking about
> examples/testing: in the form of an irb session. You can write your
> tests just by mucking about in irb, and when it makes sense, just paste
> the output somewhere.
>
>   irb> foo = generate_foo
>   => #<Foo:0xb7cd041c @attr1="hello", @attr2="world">
>   irb> foo.attr1
>   => "hello"
>   irb> foo.attr2
>   => "world"
>
>

This has been done : I found it quite some time ago on the Web and
polished it a bit :

#!/usr/bin/env ruby

# ---------------------------------------------------

# Copyright 2007 Clinton Forbes
# Modified Michel Demazure 11/07

# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 3 of the License, or
# (at your option) any later version.

# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.

# You should have received a copy of the GNU General Public License
# along with this program.  If not, see http://www.gnu.org/licenses/.

# ---------------------------------------------------

# Put this in the script directory of your Rails app

# this script is inspired by the doctest feature in Python:
# (http://docs.python.org/lib/module-doctest.html)
# :although I never did get around to reading the original Python code.
#
# Feel free to improve upon this script, on the condition that you post
# a comment on
http://clintonforbes.blogspot.com/2007/08/doctest-...
# and let me know about it so I can use it as well.
#
# to use, just put some doctests in your code and run this tool using
#
# script/doctest

# doctests look like this
=begin
#doctest Check that 1 + 1 = 2
>> x = 1
=> 1
>> x + x
=> 3
>> 2 + 3
=> 6
=end

# or like this
#=begin
##doctest Test creating a Member
#>> m = Member.new()
#=> #nil, "name"=>nil, "hashed_password"=>nil, "admin"=>false,
"reset_key"=>nil, "newsletter_level"=>nil, "created_at"=>nil,
"email"=>nil}>
#>> m.name = "Clinton"
#=> "Clinton"
#>> m.admin = true
#=end

# or like this
=begin
#doctest Check that 1 + 2 = 3
irb(main):001:0> 1 + 2
=> 3
irb(main):002:0> 2 + 3
=> 5
=end

#get all of our Rails stuff
#require File.dirname(__FILE__) + '/../config/boot'
#require File.dirname(__FILE__) + '/../config/environment'

#build array of .rb files
#only looks in ./app directory by default.  you can change this if you
keep
#extra code in other places (eg. ./vendor/plugins)

  # CODE_REGEX caters for standard IRB prompt and Rails script/console
  # prompt.
  CODE_REGEX = Regexp.new(/(>>|irb.*?>) (.*)/)
  RESULT_REGEX = Regexp.new(/=> (.*)/)

def get_ruby_files(dir_name)
  ruby_file_names = []

  Dir.foreach(dir_name) do |file_name|
    unless file_name == '.' || file_name == '..'
      full_name = File.join(dir_name, file_name)
      if /.*\.rb$/ =~ full_name
        ruby_file_names << full_name
      elsif File.directory? full_name
        sub_files = get_ruby_files(full_name)
        ruby_file_names.concat(sub_files) unless sub_files.empty?
      end
    end
  end

  ruby_file_names
end

# When running tests, addresses of objects are never likely
# to be the same, so we wipe them out so tests don't fail
#
# for example: #
def normalize_result(input)
  input.gsub(/:0x([a-f0-9]){8}/, ':0xXXXXXXXX')
end

def failure_report(statement, expected_result, result)
  report = "\n    FAILED" #add line number logic here
  report << "     Code: " << statement << "\n"
  report << "       Expected: " << expected_result << "\n"
  report << "       But got: " << result
end

def run_doc_tests(doc_test)
  execution_context = binding()
  statement, report = '', ''
  wrong, passed = 0, 0
  doc_test.split("\n").each do |line|
    case line
    when CODE_REGEX
      statement << CODE_REGEX.match(line)[2]
    when RESULT_REGEX
      expected_result = normalize_result(RESULT_REGEX.match(line)[1])
      result = normalize_result(eval(statement,
execution_context).inspect)
      unless result == expected_result
        report << failure_report(statement, expected_result, result)
        wrong += 1
      else
        passed += 1
      end
      statement = ''
    end
  end
  return passed, wrong, report
end

def process_ruby_file(file_name)
  tests, succeeded, failed = 0, 0, 0
  file_report = ''
  code = File.read(file_name)
  code.scan(/=begin\s#doctest ([^\n]*)\n(.*?)=end/m) do |doc_test|
    file_report << "\n  Testing '#{doc_test[0]}'..."
    passed, wrong, report = run_doc_tests(doc_test[1])
    file_report += (wrong == 0 ? "OK" : report)
    tests += 1
    succeeded += passed
    failed += wrong
  end
  file_report = "Processing '#{file_name}'" + file_report unless
file_report.empty?
  return tests, succeeded, failed, file_report
end

ruby_file_names = get_ruby_files(File.dirname(__FILE__))

total_report = "Looking for doctests in #{ruby_file_names.length}
files\n"
total_files, total_tests, total_succeeded, total_failed = 0, 0, 0, 0
ruby_file_names.each do |ruby_file_name|
  tests, succeeded, failed, report = process_ruby_file(ruby_file_name)
  total_files += 1 if tests > 0
  total_tests += tests
  total_succeeded += succeeded
  total_failed += failed
  total_report << report << "\n" unless report.empty?
end
total_report << "Total files: #{total_files}, total tests:
#{total_tests}, assertions succeeded: #{total_succeeded}, assertions
failed: #{total_failed}"
puts total_report
1bac2e65d64faf472cf2ebc94f0f5ee0?d=identicon&s=25 Ara Howard (ahoward)
on 2009-03-30 17:30
(Received via mailing list)
On Mar 30, 2009, at 2:12 AM, Brian Candler wrote:


( a LOT of good stuff to which i'll reply to selectively here)



> lazy or maybe because it feels like DRY violation. I've already said
> what I want to compare, why say it again?

hmmm.  yeah i see that, but disagree that the effort isn't worth it
for the *next* programer.


>  result.check "bar attribute", :expected => 123, :actual => res.bar
>
> nor
>
>  assert_equal 123, res.bar, "bar attribute"
>
> reads particularly well here, I think.

yeah i agree.  i'm open to suggestion, just has to be very very simple.


> Going too far this way down this path ends up with rspec, I think.
>
> In fact, I don't really have a problem with writing
>
>  res.foo.should == 456
>
> The trouble is the hundreds of arcane variations on this.


bingo!  i really think the key is having *one* assertion method.

> You solve this problem by only having a single test (Result#check),
> and
> indeed if rspec only had a single method (should_equal) that would be
> fairly clean too. However this is going to lead to awkwardness when
> you
> want to test for something other than equality: e.g.

i dunno - ruby is pretty good at this

   value = begin; object.call; rescue => e; e.class; end

   result.check :error, SomeError, value

that seems perfectly fine to me.  gives me an idea though - maybe
check should take a block

   result.check(:error, SomeError){ something that raises an error }

and use the block to get the actual value as in

   value = begin; block.call; rescue Object => e; e; end


>
>   result.check "foo should contain 'error'", foo, :=~, /error/
>
> But again this is getting away from real ruby for the assertions, in
> which case it isn't much better than
>
>   assert_match /error/, foo, "foo should contain 'error'"
>
>   assert_match /error/, foo  # lazy/DRY version
>

check actually uses === for the comparison so you can do

   result.check :instance_of, SomeClass, object

   result.check :matches, /pattern/, string


i need to nail that down though.



>> get a listing of which tests/examples i can run
>
> Yes, parseable results and test management are extremely beneficial.
> Those could be retro-fitted to Test::Unit though (or whatever its
> replacement in ruby 1.9 is called)
>
> Getting rid of the at_exit magic is also worth doing.
>

i actually thought of simply patching test/unit... but then there are
good test names, contexts, etc.


>>     a.push 3
>>     sum = a.inject(0){|n,i| n += i}
>>   end
>>
>> end
>
> Nice, could perhaps show the (expected) result inline too?


well - here actual always === expected with my current impl.  it's
essentially example code wrapped in assert_nothing_raised ;-)

>>
> I agree. Part of the problem is that when one thing is wrong making 20
> tests fail, all with their respective backtraces, it can be very
> hard to
> see the wood for the trees. What would be nice would be a folding-type
> display with perhaps one line for each failed assertion, and a [+] you
> can click on to get the detail for that particular one.
>

funny you mention that as i also hate that.  my first version of testy
actually just failed fast - if one test failed the code reported and
aborted.  maybe i should consider going back to that?  i am finding
that good output makes a ton of failures much easer - even using less
and searching is easier with testy  that anything else.

>> if we accept the research and assume that bugs scale linerarly with
>> the # of lines of code this is not good for robustness.
>
> I disagree there - not with the research, but the implied conclusion
> that you should never use a large codebase. Shoulda works well, and
> I've
> not once found a bizarre behaviour in the testing framework itself
> that
> I've had to debug, so I trust it.

shoulda does work well - i stole it's context concept just yesterday ;-)


>> about 1000 lines of code and 25,000 lines of testing framework!
>
> Yeah, but how many lines of Rails framework? :-)
>


i used to be on the ramaze list alot too you know ;-)


a @ http://codeforpeople.com/
B1b1d33e0655e841d4fd8467359c58d0?d=identicon&s=25 Yossef Mendelssohn (Guest)
on 2009-03-30 20:19
(Received via mailing list)
Quite a bit of discussion since I looked at this last. It's almost as
if people care about testing.

On Mar 28, 11:27 pm, "ara.t.howard" <ara.t.how...@gmail.com> wrote:
> i'm open to suggestion on format though.  requirements are
>
>         . readable by humans
>          . easily parsed by computers
>
> basically that means some yaml format.  honestly open to suggestion  
> here...

Agreed on YAML. That's just a really simple way to go if you're going
to stick to those two (very sensible) requirements. Maybe JSON would
work as well. I think the trouble I had with the specific example is
that it's listed as a failure, yet shows 'a' matching. Maybe instead
of

    returning unexpected results:
      failure:
        expect:
          a: 42
          b: forty-two
        actual:
          a: 42
          b: 42.0

you could have something like

    returning unexpected results:
      status: failure
      vars:
        matched:
        - a
        unmatched:
          expect:
            b: forty-two
          actual:
            b: 42.0

It doesn't have to be exactly like that, of course, but I'd call
attention to specifically the status being something you can easily
check by using the 'status' key instead of looking for 'failure'. Also
that the variables are split into good and bad.

The status thing, admittedly, is more of a point if you're going to
have YAML reports of tests that passed as well. One of the things I
like about RSpec and bacon is the specdoc format, so if you have
sensibly-named contexts and examples, you can get a document
explaining the behavior of the code under test. In fact, I added this
output format to shoulda in the pre-git hullabaloo. I don't think it
ever made it in.

> i have major issues with points two and three wrst to most ruby  
> testing frameworks.  one of the main points of testy is to combine  
> examples with testing.  rspec and all the others do not serve as  
> examples unless you are a ruby master.  that is to say they introduce  
> too many additions to the code that's supposed to be an example to  
> really preserve it's 'exampleness'.  and of course the output is  
> utterly useless to normal humans.  if a framework provides 1000  
> asset_xxxxxxxx methods ad nausea then the point of the code - it's  
> level of example-good-ness - is lost to mere mortals

The examples I tend to give are just very high-level overviews, mostly
showing the API and drawing the reader into other documentation, the
specs, or finally the code if they really want to see everything my
lib/module/gem/whathaveyou can do. As such, they're usually not very
helpful as far as specs go.

I agree that having a glut of assertions/checks isn't useful, and is
fact hiding a very important fact when it comes to BDD (or really
testing in general): If something is painful to test, it should be
changed. What I love about BDD is having that something using my code
and defining its API as I go, so I don't get stuck in nasty test-land
with shitty implementation-specific tests, or at least not as easily.
So that pain, like any pain, is a signal that something is wrong, and
it should be fixed. It shouldn't be hidden behind a special assertion
or matcher or any other rug-sweeping activity unless there's good
reason, like the code that's hurting you to test isn't available to
you to fix.

>
> cfp:~/redfission > find app -type f|xargs -n1 cat|wc -l
>      1828
>
> rspec and co might be fine but seriously, the above is insane right?

It's bordering on nuts. It'd be better to show LoC than just wc -l, of
course, and are all the thoughtbot gems for your tests? Also, someone
else mentioned flay/flog/reek output, which could be illuminating. And
someone also brought up the app framework. After all, what you're
doing here is comparing your app code to your test framework. Seems
like you should be comparing app to test, or app framework to test
framework, or app to test with their respective frameworks. (And don't
forget to include Test::Unit for the testing frameworks that are
simply a layer on top of that.)

And I'd like to once again bring up bacon. I say I live with a few
methods on Object and Kernel to be able to test without going crazy,
and I mean it. I don't care how much how much you say assert_equal
makes sense or result.check 'some name', :expect => x, :actual => y is
great, I'm never going to get confused about some_value.should == 5.
It's readable, and it's close to English. Where I think bacon wins out
huge over RSpec is that bacon keeps all those methods on the Should
object instead of making them global. It's a small change to syntax
(some_obj.should.respond_to(:meth) vs. some_obj.should respond_to
(:meth)), but it makes a world of difference as far as sensibility
goes.

Oh, and bacon is < 300 LoC, with facon at 365 (a line of code for
every day in the year!).
852a62a28f1de229dc861ce903b07a60?d=identicon&s=25 Gavin Kistner (phrogz)
on 2009-03-30 20:20
(Received via mailing list)
On Mar 30, 2:12 am, Brian Candler <b.cand...@pobox.com> wrote:
> Ara Howard wrote:
> >>  assert_equal 42, ultimate.answer, "name"
>
> > you can basically do that too, but i continually forget which is
> > expected and which is actual and, as you know, that's a slippery error
> > to track down at times.
>
> Perhaps - but it's one rule that only needs to be learned once.

I disagree. I 'learn' it each time I look at it, and then I forget it.
I think possibly because I think it's backwards.

The same goes for alias_method for me. I cannot tell you how many
times I've had to look up the order of old_name/new_name. And with
this, it's certainly because I think the values are backwards. (I just
had to look up the order to be sure.)
1bac2e65d64faf472cf2ebc94f0f5ee0?d=identicon&s=25 Ara Howard (ahoward)
on 2009-03-30 21:49
(Received via mailing list)
On Mar 30, 2009, at 12:20 PM, Phrogz wrote:

> I disagree. I 'learn' it each time I look at it, and then I forget it.
> I think possibly because I think it's backwards.
>
> The same goes for alias_method for me. I cannot tell you how many
> times I've had to look up the order of old_name/new_name. And with
> this, it's certainly because I think the values are backwards. (I just
> had to look up the order to be sure.)

100% agree on both counts.  i personally would always use the options
approach (:expected, :actual =>)  because of that.

a @ http://codeforpeople.com/
Beb77c4602c3cac7a12149431366ed11?d=identicon&s=25 The Higgs bozo (higgsbozo)
on 2009-03-30 23:21
Ara Howard wrote:
> On Mar 30, 2009, at 12:20 PM, Phrogz wrote:
>
>> I disagree. I 'learn' it each time I look at it, and then I forget it.
>> I think possibly because I think it's backwards.
>>
>> The same goes for alias_method for me. I cannot tell you how many
>> times I've had to look up the order of old_name/new_name. And with
>> this, it's certainly because I think the values are backwards. (I just
>> had to look up the order to be sure.)
>
> 100% agree on both counts.  i personally would always use the options
> approach (:expected, :actual =>)  because of that.

Ha, and I thought it was only me with alias_method and assert_equal
dyslexia.  With alias_method, I find myself expecting it to act like
'cp' or 'ln', and I have to keep remembering it's the opposite of that,
even after a hundred times of re-remembering.  And assert_equal order is
the opposite to the usual order in English, which I guess is a selling
point of rspec.

Along the same lines, I always ask irb about methods(true) vs
methods(false).  I'm like, OK I just want what's recently defined --
true?  Doh.  OK then, false, I only want the recent definitions.  False
-- do not include super.  See? I am even forgetting, right now, as I
type this, which is which.

  "I curse you in ways multifarious!"
  "OK, will that be Curses.methods(true) or Curses.methods(false)?"
  "ARGGH!"
753dcb78b3a3651127665da4bed3c782?d=identicon&s=25 Brian Candler (candlerb)
on 2009-03-31 12:10
Ara Howard wrote:
> shoulda does work well - i stole it's context concept just yesterday ;-)

Looks like you're not the only one:
http://github.com/citrusbyte/contest

A nice thing about tiny test frameworks like this (and yours) is that
they are easily vendorized, so the app is no longer dependent on a
specific test framework and/or version being available.
47b1910084592eb77a032bc7d8d1a84e?d=identicon&s=25 Joel VanderWerf (Guest)
on 2009-04-02 22:03
(Received via mailing list)
Phlip wrote:
>
>
> The second pass puts the block into Ripper to decompile it, and then the
> fun starts.

So you have to be careful in the block to avoid side effects, right?
Otherwise pass1 and pass2 might start with different state.
Aafa8848c4b764f080b1b31a51eab73d?d=identicon&s=25 Phlip (Guest)
on 2009-04-02 23:36
(Received via mailing list)
Joel VanderWerf wrote:

> So you have to be careful in the block to avoid side effects, right?
> Otherwise pass1 and pass2 might start with different state.

No, I just document: Don't put anything in the block with a side-effect.
The
money-line (the line that calls your production code to test it), for
example,
should never be inside the assert{}.

The first pass is just a raw block.call, so the assertion must always
get its
actual status right...
This topic is locked and can not be replied to.