HTTP::POST extra character

Ruby
#1

Hello,

I write a script by ruby where it connects to a server and communicates
with it by HTTP protocol.
When I logged my request I can see that the line end is not correct.
So the end of the lines need to have \r\n but I see \r\r\n.
It is not clear for me why? I use this script on windows and I suspect
it is OS specific problem. Here is the main part of the code regarding
this issue:
…
request = Net::HTTP::Post.new(uri.request_uri)
request.body = $post_body.join
headers = {
‘Pragma’ => “no-cache”,
.
.
.
‘Connection’ => ‘Keep-Alive’,
‘Cache-Control’ => ‘no-cache’
}
request.initialize_http_header(headers)
response = http.request(request)

– end of code —

I can see this:

POST / HTTP/1.1
“[80, 79, 83, 84, 32, 47, 32, 72, 84, 84, 80, 47, 49, 46, 49, 13, 13,
10]”

So the lineend is \r\r\n (13,13,10).
And this is same at other header and body lines.

Any idea?

Thanks in advance

Rainman

#2

Any idea?

If you write the string “\n” to a file in text mode, e.g. using
mode “w”, then ruby does automatic newline conversions
appropriate for your OS, so on windows ruby converts “\n”
to “\r\n” before writing to the file. That regime allows you
to write “\n” in a ruby string to represent a newline, and
no matter what OS the program runs on, the proper newline
will get written to the file.

That means if you write the string “\r\n” to a file in text mode on
windows, the “\n” part of that string will get converted to “\r\n”,
so ruby will write “\r\r\n” to the file.

When I logged my request

If the first line of the request produced by the code:

 request = Net::HTTP::Post.new(uri.request_uri)

was:

"POST / HTTP/1.1\r\n"

…and you wrote that string to a file in text mode(i.e. "w’) on
windows, then ruby will naturally convert any “\n”
characters in the string to “\r\n” before writing to the file, so
ruby will end up writing the following to the file:

"POST / HTTP/1.1\r\r\n"

If you want to see the exact characters in the request, then on
windows you need to write the request to a file using binary
mode, so that ruby does not implicitly (i.e. behind your back)
change some of the characters. In binary mode, e.g. “bw”
(= binary + write mode), ruby will not do any newline conversions.

#3

Thanks for your feedback.

Any idea?

If you write the string “\n” to a file in text mode, e.g. using
mode “w”, then ruby does automatic newline conversions
appropriate for your OS, so on windows ruby converts “\n”
to “\r\n” before writing to the file. That regime allows you
to write “\n” in a ruby string to represent a newline, and
no matter what OS the program runs on, the proper newline
will get written to the file.

Yes, I read it but I did not add \r or \n here in the code.
See here:

headers = {
‘Pragma’ => “no-cache”,
.
.
.
‘Connection’ => ‘Keep-Alive’,
‘Cache-Control’ => ‘no-cache’
}

But I can see \r\r\n characters at end of the header lines.

I use the body this way:

$post_body << “Content-Disposition: form-data; name=“Comments”\r\n”
$post_body << “\r\n”

Here I can remove \r and this case \n will stay which will converted by
ruby to \r\n.

But how can I correct the header.

When I logged my request

If the first line of the request produced by the code:

 request = Net::HTTP::Post.new(uri.request_uri)

was:

"POST / HTTP/1.1\r\n"

…and you wrote that string to a file in text mode(i.e. "w’) on
windows, then ruby will naturally convert any “\n”
characters in the string to “\r\n” before writing to the file, so
ruby will end up writing the following to the file:

"POST / HTTP/1.1\r\r\n"

I have a script written in tcl for logging. So it do not perform any
charcter conversion as I know.

#4

Yes, I read it but I did not add \r or \n here in the code.

Yes, I read your code, so I know that you did not add \r or \n when
creating your request, but you said this:

When I logged my request

That means you wrote to a file somewhere.

I have a script written in tcl for logging. So it do not perform
any charcter conversion as I know.

tcl docs for puts (http://www.tcl.tk/man/tcl8.4/TclCmd/puts.htm):

Newline characters in the output are translated by puts to
platform-specific end-of-line sequences according to the current value
of the -translation option for the channel

settings for -translation option (from “Practical Programming in Tcl and
Tk”):

auto The default behavior. On input, all end of line conventions are
converted to a newline. Output is in native format.

#5

Thanks for your feedback :slight_smile:

My script is working now :slight_smile:

I send a multipart form through HTTP protcol. The main problem was that
the multipart text was not correct.

The server can understand my request now despite of that the characters
are \r\r\n on the end of the lines.

Sorry, perhaps I did not explain my problem perfectly, but thanks for
your help.

Regarding tcl, I use it this way:

puts -nonewline $f $data

“Puts normally outputs a newline character after string, but this
feature may be suppressed by specifying the -nonewline switch.”

Sorry, I did not mentioned it.

You have right only the puts adds newline character to the string.

Thanks

#6

Zsolt B. wrote in post #1146192:

Regarding tcl, I use it this way:

puts -nonewline $f $data

“Puts normally outputs a newline character after string, but this
feature may be suppressed by specifying the -nonewline switch.”

Sorry, I did not mentioned it.

The first line of the request produced by ruby is this:

$data = “POST / HTTP/1.1\r\n”

If you do this in tcl:

puts $f $data

…then tcl will add a newline ("\n") to the string, like this:

"POST / HTTP/1.1\r\n\n"

Then, before writing to the file, tcl will convert any “\n” characters
in the string to the OS’s “line ending conventions”, which on
windows are “\r\n”, so tcl ends up writing the following to the file:

"POST / HTTP/1.1\r\r\n\r\n"

If instead you write the string to the file with this command:

 puts -nonewline $f $data

…that doesn’t tell tcl to chop off newlines at the end of $data.
The -nonewline option tells tcl not to add a newline to the
end of $data. As a result, that command tells tcl to write
the following string to the file:

 "POST / HTTP/1.1\r\n"

And before writing that string to the file, tcl will convert any “\n”
characters to “\r\n”, so tcl ends up writing the following to the file:

 "POST / HTTP/1.1\r\r\n"

Automatic newline conversions are also done in ruby, python, perl,
and many other languages.

The server can understand my request now despite of that the
characters are \r\r\n on the end of the lines

That is not what the server is seeing. As in quantuum mechanics, by
trying to observe the thing you want to look at, you changed it.