# Print - and strip text between tags using Nokogiri

**URL:** <https://www.ruby-forum.com/t/print-and-strip-text-between-tags-using-nokogiri/226555>\
**Category:** Ruby\
**Created:** [December 16, 2012, 12:10am UTC](https://www.ruby-forum.com/t/print-and-strip-text-between-tags-using-nokogiri/226555 "2012-12-16T00:10:35Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![paul-au](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/f475e1/32.png) [@paul-au](https://www.ruby-forum.com/u/paul-au)\
**Post date:** [December 16, 2012, 12:10am UTC](https://www.ruby-forum.com/t/print-and-strip-text-between-tags-using-nokogiri/226555/1 "2012-12-16T00:10:35Z")

</div>

I’m a Ruby N. trying to write a program to process thousands of HTML  
files, extracting pertinent text and inserting it into a MySQL database.  
Ruby seems ideally suited to the task in general, and I’ve already used  
Nokogiri to extract comment text. What I need to do next is to print -  
and then ultimately delete or strip - the text between “pre” tags.

Picture some html like this:

My Title
# My Heading
**From:** Me  
**Date:** Wed Dec 05 2012 - 18:17:49 EST

text line 1  
text line 2  
text line 3

```
very important text
more important text
would you believe even more important text?
```

I basically need to do 2 things: 1) to print only the text between the 2  
“pre” tags, and then 2) to print all of the non-tagged text between the  
“body” comments - minus the text between the “pre” tags. I’ve been  
messing with this for a couple of hours - unsuccessfully - but I’m still  
convinced that this is the right tool for the job.

---

<div class="post-metadata">

**Author:** ![paul-au](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/f475e1/32.png) [@paul-au](https://www.ruby-forum.com/u/paul-au)\
**Post date:** [December 16, 2012, 12:29am UTC](https://www.ruby-forum.com/t/print-and-strip-text-between-tags-using-nokogiri/226555/2 "2012-12-16T00:29:17Z")

</div>

On Sun, Dec 16, 2012 at 12:10 AM, Paul M. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
wrote:

> My Title  
> 
> I basically need to do 2 things: 1) to print only the text between the 2  
> “pre” tags, and then 2) to print all of the non-tagged text between the  
> “body” comments - minus the text between the “pre” tags. I’ve been  
> messing with this for a couple of hours - unsuccessfully - but I’m still  
> convinced that this is the right tool for the job.

If you need to do more HTML and XML manipulation, learning XPath is a  
good investment! You can look here for a start:  
[http://www.w3schools.com/Xpath/default.asp](http://www.w3schools.com/Xpath/default.asp)

_One_ way to achieve what you want:

require ‘nokogiri’

text = \<\<HTML

My Title
# My Heading
**From:** Me  
**Date:** Wed Dec 05 2012 - 18:17:49 EST

text line 1  
text line 2  
text line 3

```
very important text
more important text
would you believe even more important text?
```

HTML

dom = Nokogiri.HTML(text)

puts dom.xpath(‘/html/body//pre/text()’).map(&:to\_s)

puts ‘—’

puts dom.xpath(‘/html/body//text()[not(ancestor::pre)]’).map(&:to\_s)

You can also process nodes individually if you replace “.map…” with  
“.each” and a block which receives the node and does something with  
it.

Kind regards

robert

---

<div class="post-metadata">

**Author:** ![paul-au](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/f475e1/32.png) [@paul-au](https://www.ruby-forum.com/u/paul-au)\
**Post date:** [December 16, 2012, 3:23am UTC](https://www.ruby-forum.com/t/print-and-strip-text-between-tags-using-nokogiri/226555/3 "2012-12-16T03:23:01Z")

</div>

Thank you for the swift reply! I tried running the above against my  
“test.html” snippet and ended up getting the following:

## pablo@cochituate=\> ./extract\_text.rb ./test.html

./test.html

---

<div class="post-metadata">

**Author:** ![paul-au](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/f475e1/32.png) [@paul-au](https://www.ruby-forum.com/u/paul-au)\
**Post date:** [December 16, 2012, 4:21am UTC](https://www.ruby-forum.com/t/print-and-strip-text-between-tags-using-nokogiri/226555/4 "2012-12-16T04:21:47Z")

</div>

On Sat, Dec 15, 2012 at 8:23 PM, Paul M. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> Thank you for the swift reply! I tried running the above against my  
> “test.html” snippet and ended up getting the following:
> 
> ## pablo@cochituate=\> ./extract\_text.rb ./test.html
> 
> ./test.html
> 
> –  
> Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/).

You passed in the file name string to nokugiri, not the contents.

---

<div class="post-metadata">

**Author:** ![paul-au](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/f475e1/32.png) [@paul-au](https://www.ruby-forum.com/u/paul-au)\
**Post date:** [December 16, 2012, 11:37am UTC](https://www.ruby-forum.com/t/print-and-strip-text-between-tags-using-nokogiri/226555/5 "2012-12-16T11:37:23Z")

</div>

There should be a way to match the text of the first comment, but I  
couldn’t get this to work:

comment()[text()=‘body=“start”’]

---

<div class="post-metadata">

**Author:** ![paul-au](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/f475e1/32.png) [@paul-au](https://www.ruby-forum.com/u/paul-au)\
**Post date:** [December 16, 2012, 11:18am UTC](https://www.ruby-forum.com/t/print-and-strip-text-between-tags-using-nokogiri/226555/6 "2012-12-16T11:18:39Z")

</div>

Robert K. wrote in post #1089225:

> puts dom.xpath(‘/html/body//pre/text()’).map(&:to\_s)

Calling map() is redundant because puts calls to\_s on its arguments.

> 1. to print all of the non-tagged text between the  
> “body” comments

Your html doesn’t even test your requirements because there is no text  
after the body="end’ comment. And there is no non-tagged text:

require ‘nokogiri’

html = \<\<HTML

My Title
# My Heading
**From:** Me  
**Date:** Wed Dec 05 2012 - 18:17:49 EST

text line 1  
text line 2  
text line 3

```
very important text
more important text
would you believe even more important text?
```

text line 4  
text line 5 HTML

doc = Nokogiri.HTML(html)

my\_xpath = “/html/body/comment()[1]/following-sibling::\*”

doc.xpath(my\_xpath).each do |node|  
puts node.name  
puts node.text  
puts ‘\*’ \* 20  
end

–output:–  
p

text line 1

text line 2

text line 3

* * *

p

* * *

pre

very important text  
more important text  
would you believe even more important text?

* * *

p

* * *

p

text line 4

text line 5

* * *

doc = Nokogiri.HTML(html)

my\_xpath =  
“/html/body/comment()[1]/following-sibling::\*[not(self::pre)]”

catch :found\_ending\_text do  
doc.xpath(my\_xpath).each do |node|  
node.children.each do |child|  
text = child.text  
throw :found\_ending\_text if text.include? %q{body=“end”}  
next if text.empty?  
puts text.strip  
end  
end  
end

–output:–  
text line 1  
text line 2  
text line 3

---

<div class="post-metadata">

**Author:** ![paul-au](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/f475e1/32.png) [@paul-au](https://www.ruby-forum.com/u/paul-au)\
**Post date:** [December 16, 2012, 11:52am UTC](https://www.ruby-forum.com/t/print-and-strip-text-between-tags-using-nokogiri/226555/7 "2012-12-16T11:52:14Z")

</div>

This ugly xpath will select the comment based on its text:

comment()[. =’ body=“start” ']

my\_xpath = %Q{/html/body/comment()[. =’ body=“start”  
']/following-sibling::\*[not(self::pre)]}

---

<div class="post-metadata">

**Author:** ![paul-au](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/f475e1/32.png) [@paul-au](https://www.ruby-forum.com/u/paul-au)\
**Post date:** [December 17, 2012, 12:57am UTC](https://www.ruby-forum.com/t/print-and-strip-text-between-tags-using-nokogiri/226555/8 "2012-12-17T00:57:08Z")

</div>

7stud - that’s perfect. Thank you so much!

---

<div class="post-metadata">

**Author:** ![paul-au](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/f475e1/32.png) [@paul-au](https://www.ruby-forum.com/u/paul-au)\
**Post date:** [December 16, 2012, 5:11pm UTC](https://www.ruby-forum.com/t/print-and-strip-text-between-tags-using-nokogiri/226555/9 "2012-12-16T17:11:47Z")

</div>

I want the thank everyone for their quick replies and helpful  
suggestions. I realized that I should probably be using the real - and  
admittedly poorly-formed - HTML for this question and not the test HTML  
I’ve tried to concoct for this example. The real HTML was generated by  
the Hypermail program, basically converting an email from mbox form to  
HTML. Here is one such file:

haiku\_archive: watching the news
# watching the news
**From:** Paul David Mena (_removed\_email\_address@domain.invalid_)  
**Date:** Fri Dec 14 2012 - 18:51:14 EST

* * *

watching the news  
I feel guilty  
for being alive

```
--
Paul David Mena
--------------------
removed_email_address@domain.invalid
```

My ultimate goal is to extract all of the comment text between and but _not_ what is between the  
two “pre” tags. So far I’ve been able to extract all of the comment  
text but not exclude the “pre” text, using the following code:

#!/usr/bin/env ruby

require “rubygems”  
require “nokogiri”

class PlainTextExtractor \< Nokogiri::XML::SAX::Document

attr\_reader :plaintext

# Initialize the state of interest variable with false

def initialize  
@interesting = false  
@plaintext = “”  
end

# This method is called whenever a comment occurs and

# the comments text is passed in as string.

def comment(string)  
case string.strip # strip leading and trailing whitespaces  
when /^body=“start”/ # match starting comment  
@interesting = true  
when /^body=“end”/  
@interesting = false # match closing comment  
end  
end

# This callback method is called with any string between

# a tag.

def characters(string)  
@plaintext \<\< string if @interesting  
end  
end

# write to the screen

pte = PlainTextExtractor.new  
parser = Nokogiri::HTML::SAX::Parser.new(pte)  
parser.parse\_file ARGV[0]

# puts pte.plaintext

# write to a file

begin  
file = File.open(“snippet.txt”, “w”)  
file.write pte.plaintext  
rescue IOError =\> e  
#some error occur, dir not writable etc.  
ensure  
file.close unless file == nil  
end

# get the date written

fname = ARGV[0]  
start\_column = 3  
end\_column = 5

target\_range = (start\_column-1)…(end\_column-1)

IO.foreach(fname) do |line|  
if line.match(/**Date:\</strong\>/)  
pieces = line.split(" “)  
puts pieces[target\_range].join(”-")  
end  
end**

# remove blank lines from file

fh = File.open(‘snippet.txt’)  
while( !fh.eof)  
line = fh.readline.chomp  
# remove leading and trailing blanks  
line.strip!  
# skip empty lines  
next if line == ‘’  
# convert tab chars to blanks  
line.gsub!(/\t/,’ ‘)  
# substitute a single blank for a sequence of blanks  
line.squeeze!(’ ')  
# add code to process line if needed  
puts line  
end  
fh.close  
exit(0)

The output is as follows:

## pablo@cochituate=\> ./extract\_haiku.rb /export/www/html/haikupoet/archive/0925.html watching the news I feel guilty for being alive

## Paul David Mena

removed\_email\_address@domain.invalid

Basically I want to omit the signature (everything below the “–”,  
inclusive), which is wrapped in the “pre” tags.

---

<div class="post-metadata">

**Author:** ![paul-au](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/f475e1/32.png) [@paul-au](https://www.ruby-forum.com/u/paul-au)\
**Post date:** [December 17, 2012, 2:01am UTC](https://www.ruby-forum.com/t/print-and-strip-text-between-tags-using-nokogiri/226555/10 "2012-12-17T02:01:16Z")

</div>

Paul M. wrote in post #1089283:

> # remove blank lines from file
> 
> fh = File.open(‘snippet.txt’)  
> while( !fh.eof)  
> line = fh.readline.chomp  
> # remove leading and trailing blanks  
> line.strip!  
> # skip empty lines  
> next if line == ‘’  
> # convert tab chars to blanks  
> line.gsub!(/\t/,’ ‘)  
> # substitute a single blank for a sequence of blanks  
> line.squeeze!(’ ')  
> # add code to process line if needed  
> puts line  
> end  
> fh.close  
> exit(0)

I forgot to mention. There are several ways to read line by line from a  
file, but your loop is particularly ugly. If you use my favorite:

IO.foreach(fname) do |line|  
line = line.chomp  
…  
..  
end

…an added benefit is that the file is automatically closed when the  
block exits.

---

<div class="post-metadata">

**Author:** ![paul-au](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/f475e1/32.png) [@paul-au](https://www.ruby-forum.com/u/paul-au)\
**Post date:** [December 17, 2012, 7:12am UTC](https://www.ruby-forum.com/t/print-and-strip-text-between-tags-using-nokogiri/226555/11 "2012-12-17T07:12:38Z")

</div>

7stud – wrote in post #1089314:

> I forgot to mention. There are several ways to read line by line from a  
> file,

Here are some others:

f = File.new(fname)

f.each do |line|  
line = line.chomp

end

f.close

====

File.open(fname) do |f|  
while line = f.gets #gets() returns nil at eof  
…  
end  
end #file is automatically closed here

---

<div class="post-metadata">

**Author:** ![paul-au](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/f475e1/32.png) [@paul-au](https://www.ruby-forum.com/u/paul-au)\
**Post date:** [December 16, 2012, 10:17pm UTC](https://www.ruby-forum.com/t/print-and-strip-text-between-tags-using-nokogiri/226555/12 "2012-12-16T22:17:40Z")

</div>

> My ultimate goal is to extract all of the comment text between and

Yet none of your example html is setup to test that requirement–because  
there is no text after the body=end comment.

require “nokogiri”

class PlainTextExtractor \< Nokogiri::XML::SAX::Document  
attr\_reader :plaintext

# Initialize the state of interest variable with false

def initialize  
@interesting = false  
@pre = false  
@plaintext = “”  
end

def start\_element(name, attrs = )  
if name == “pre”  
@pre = true  
end  
end

def end\_element(name, attrs = )  
if name == “pre”  
@pre = false  
end  
end

# This method is called whenever a comment occurs and

# the comments text is passed in as string.

def comment(string)  
case string.strip # strip leading and trailing whitespaces  
when /^body=“start”/ # match starting comment  
@interesting = true  
when /^body=“end”/  
@interesting = false # match closing comment  
end  
end

# This callback method is called with any string between

# a tag.

def characters(string)  
if @interesting and not @pre  
@plaintext \<\< string  
end  
end  
end

pte = PlainTextExtractor.new  
parser = Nokogiri::HTML::SAX::Parser.new(pte)  
parser.parse\_file ARGV[0]

p pte.plaintext

–output:–  
“\n\nwatching the news\n\nI feel guilty\n\nfor being alive\n\n”

---

<div class="post-metadata">

**Author:** ![paul-au](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/f475e1/32.png) [@paul-au](https://www.ruby-forum.com/u/paul-au)\
**Post date:** [December 17, 2012, 9:12am UTC](https://www.ruby-forum.com/t/print-and-strip-text-between-tags-using-nokogiri/226555/13 "2012-12-17T09:12:09Z")

</div>

On Sun, Dec 16, 2012 at 11:18 AM, 7stud – [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> Robert K. wrote in post #1089225:
> 
> > puts dom.xpath(‘/html/body//pre/text()’).map(&:to\_s)
> 
> Calling map() is redundant because puts calls to\_s on its arguments.

Right, thanks for the reminder! That was an artifact of IRB testing.  
🙂

> > 1. to print all of the non-tagged text between the  
> > “body” comments
> 
> Your html doesn’t even test your requirements because there is no text  
> after the body="end’ comment. And there is no non-tagged text:

I overlooked the comment thing.

#!/usr/bin/ruby

require ‘nokogiri’

# require ‘irb’

text = \<\<HTML

My Title
# My Heading
**From:** Me  
**Date:** Wed Dec 05 2012 - 18:17:49 EST

text line 1  
text line 2  
text line 3

```
very important text
more important text
would you believe even more important text?
```

not to print HTML

dom = Nokogiri.HTML(text)

puts dom.xpath(‘/html/body//pre/text()’)

puts ‘—’

puts dom.xpath(‘//text()[contains(preceding::comment(),“start”) and  
contains(following::comment(),“end”) and not(ancestor::pre)]’)

Kind regards

robert
