Using HTTParty to post a very large set of name/value pairs

Ruby
1

I’ve got a pretty large text file (a few thousand lines on average)
consisting of all key/value pairs. If I use wget to run the post from a
command line, it seems to work fine.

However, I’m trying to do it in Ruby, and I’ve already got the rest of
the API working with HTTParty. So if I’ve got the data in a big
concatenated string:

data=“foo=bar\nbaz=quux\nhamlet=ophelia\nclaudius=gertrude …”

It seems the only way I can wrap it for sending is like this:

post('/processor.php', :query => {:data=>data})

But after a certain size for data, I start getting “414 Request-URI Too
Large” errors.

Since it works as a direct post from wget, the best I can figure is that
the difference lies in how exactly HTTParty is formatting it for
sending. Is there a good way for me to see what the actual outgoing
request looks like coming out of that post? I have a simple test jig on
the other side (processor.php) that dumps out the $_POST variables it
gets, but when I get the 414 error it never reaches my tester.

At some point I would scrap it and switch over to something more
upload/attachment based, but the fact that it works with wget bothers
me, and the existing system is already architected to work that way and
I hate to start tearing stuff apart.

2

Aha, may have solved my own problem. Looks like using the :body option
instead of :query might be just what I need!

Sorry for the interruption.

D

Duane M. wrote:

I’ve got a pretty large text file (a few thousand lines on average)
consisting of all key/value pairs. If I use wget to run the post from a
command line, it seems to work fine.

However, I’m trying to do it in Ruby, and I’ve already got the rest of
the API working with HTTParty. So if I’ve got the data in a big
concatenated string:

data=“foo=bar\nbaz=quux\nhamlet=ophelia\nclaudius=gertrude …”

It seems the only way I can wrap it for sending is like this:

post('/processor.php', :query => {:data=>data})

But after a certain size for data, I start getting “414 Request-URI Too
Large” errors.

Since it works as a direct post from wget, the best I can figure is that
the difference lies in how exactly HTTParty is formatting it for
sending. Is there a good way for me to see what the actual outgoing
request looks like coming out of that post? I have a simple test jig on
the other side (processor.php) that dumps out the $_POST variables it
gets, but when I get the 414 error it never reaches my tester.

At some point I would scrap it and switch over to something more
upload/attachment based, but the fact that it works with wget bothers
me, and the existing system is already architected to work that way and
I hate to start tearing stuff apart.