# Simple question: combine a quoted string into a single token

**URL:** <https://www.ruby-forum.com/t/simple-question-combine-a-quoted-string-into-a-single-token/101747>\
**Category:** Ruby\
**Created:** [August 7, 2006, 7:43pm UTC](https://www.ruby-forum.com/t/simple-question-combine-a-quoted-string-into-a-single-token/101747 "2006-08-07T19:43:01Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Squeamizh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/f05b48/32.png) [@Squeamizh](https://www.ruby-forum.com/u/Squeamizh)\
**Post date:** [August 7, 2006, 7:43pm UTC](https://www.ruby-forum.com/t/simple-question-combine-a-quoted-string-into-a-single-token/101747/1 "2006-08-07T19:43:01Z")

</div>

Hi,

I have a program which separates each line of a text file into tokens,  
using whitespace as a delimiter (I do this with String.split). This  
suits my needs for the most part, but now I need the ability to treat  
quoted strings as single tokens. Note that the quoted string could be  
multiple words, or even a 0-length string.

Could anyone recommend a basic strategy for doing this? Should I deal  
with this when I first tokenize each line, or should I combine tokens  
appropriately during parsing when I see a double-quote?

Help would be greatly appreciated.

---

<div class="post-metadata">

**Author:** ![Squeamizh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/f05b48/32.png) [@Squeamizh](https://www.ruby-forum.com/u/Squeamizh)\
**Post date:** [August 7, 2006, 7:49pm UTC](https://www.ruby-forum.com/t/simple-question-combine-a-quoted-string-into-a-single-token/101747/2 "2006-08-07T19:49:02Z")

</div>

On Tuesday, August 08, 2006, at 2:40 AM, Squeamizh wrote:

> appropriately during parsing when I see a double-quote?
> 
> Help would be greatly appreciated.

You could pull out all the quoted strings into an array and then delete  
them from the original before processing it normally.

\_Kevin  
[www.sciwerks.com](http://www.sciwerks.com)

---

<div class="post-metadata">

**Author:** ![Squeamizh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/f05b48/32.png) [@Squeamizh](https://www.ruby-forum.com/u/Squeamizh)\
**Post date:** [August 7, 2006, 8:37pm UTC](https://www.ruby-forum.com/t/simple-question-combine-a-quoted-string-into-a-single-token/101747/3 "2006-08-07T20:37:06Z")

</div>

Squeamizh wrote:

> appropriately during parsing when I see a double-quote?
> 
> Help would be greatly appreciated.

Something along the lines of

line.scan %r{  
“[^”]\*" |  
\S+  
}x

HTH

```
robert
```