HTTP PUT for file uploads

Server NGINX
1

It appears that nginx has built-in support for the PUT method.

My company needs a file upload solution that will support large file
uploads (2GB limit is optional - if we have to tell them less than 2GB
that’s fine) that will keep re-trying the upload until it is done. We
have slow geo users and then just flat out large files to deal with
even from fast connections.

There’s a variety of Java-based PUT uploaders.

So far, we haven’t found any Flash ones (we’d love to NOT use Java) -
but there is a way to do it apparently, we just can’t find anyone
who’s done it yet.

Anyway, from the nginx side, I assume I would need to have some
timeouts set pretty long. Would anyone here have any ideas about that?

I’m assuming that we should keep the connection open as long as there
is some activity and maybe timeout after a minute or two… the
client-side applet should have the logic to continue retrying and
since it is PUT, the PHP script will accept the data and use fseek()
on the file to resume at the offset supplied (the client will have to
give us that info)

See the examples here:
http://www.radinks.com/upload/examples/ - look at the “Handlers that
support resume” section.

Anyone have any thoughts? I think I also need to tweak PHP settings
too possibly as well, for max execution time and such.

2

mike wrote:

My company needs a file upload solution that will support large file
uploads (2GB limit is optional - if we have to tell them less than 2GB
that’s fine) that will keep re-trying the upload until it is done. We
have slow geo users and then just flat out large files to deal with
even from fast connections.

Don’t want to sound off, but isn’t that what FTP was designed for?

There’s a variety of Java-based PUT uploaders.

So far, we haven’t found any Flash ones (we’d love to NOT use Java) -
but there is a way to do it apparently, we just can’t find anyone
who’s done it yet.

You may have to test this, but I’d assume the maint-stream browsers
support PUT via the XMLHTTPRequest javascript object.

–

Phillip B Oldham
The Activity People
[email protected] mailto:[email protected]

Policies

This e-mail and its attachments are intended for the above named
recipient(s) only and may be confidential. If they have come to you in
error, please reply to this e-mail and highlight the error. No action
should be taken regarding content, nor must you copy or show them to
anyone.

This e-mail has been created in the knowledge that Internet e-mail is
not a 100% secure communications medium, and we have taken steps to
ensure that this e-mail and attachments are free from any virus. We must
advise that in keeping with good computing practice the recipient should
ensure they are completely virus free, and that you understand and
observe the lack of security when e-mailing us.

3

Michael wrote:

But 2GB? That’s huge… a lot for an end-user to just sit and watch without
clicking reload or something.

Exactly - 2GB is going to be hell to upload. At least with FTP it can
use multiple ports for data transfer. And with a web interface I’d
expect users to get annoyed with clicking “back” accidentally or having
their email-client open a link using the same window. With FTP you can
“background” the process.

As for firewalls; just use an active connection rather than passive. Job
done. You’ll loose the benefit of the different ports, though.

You may have to test this, but I’d assume the maint-stream browsers support
PUT via the XMLHTTPRequest javascript object.

I was just doing some of this myself yesterday - doesn’t seem like there’s a
way to do file uploads with AJAX, but you can easily make a form with a target
iframe… but this is really offtopic for the nginx list.

–

Phillip B Oldham
The Activity People
[email protected] mailto:[email protected]

Policies

This e-mail and its attachments are intended for the above named
recipient(s) only and may be confidential. If they have come to you in
error, please reply to this e-mail and highlight the error. No action
should be taken regarding content, nor must you copy or show them to
anyone.

This e-mail has been created in the knowledge that Internet e-mail is
not a 100% secure communications medium, and we have taken steps to
ensure that this e-mail and attachments are free from any virus. We must
advise that in keeping with good computing practice the recipient should
ensure they are completely virus free, and that you understand and
observe the lack of security when e-mailing us.

4

On Fri, Aug 01, 2008 at 13:10:46, Phillip B Oldham said…

Don’t want to sound off, but isn’t that what FTP was designed for?

FTP is such an awful protocol to deal with regarding firewalls, I’d much
rather
go the way the original poster is and use HTTP PUT for insecure stuff,
and SFTP
for secure stuff.

But 2GB? That’s huge… a lot for an end-user to just sit and watch
without
clicking reload or something.

You may have to test this, but I’d assume the maint-stream browsers support
PUT via the XMLHTTPRequest javascript object.

I was just doing some of this myself yesterday - doesn’t seem like
there’s a
way to do file uploads with AJAX, but you can easily make a form with a
target
iframe… but this is really offtopic for the nginx list.

5

On Fri, Aug 01, 2008 at 15:17:20, Phillip B Oldham said…

Exactly - 2GB is going to be hell to upload. At least with FTP it can use
multiple ports for data transfer. And with a web interface I’d expect users
to get annoyed with clicking “back” accidentally or having their
email-client open a link using the same window. With FTP you can
“background” the process.

Yeah, users would do something like that… I wonder how youtube deals
with it?
Those uploads take a loooong time… do users get frustrated and abort
the
transfer?

Handling the upload in a web server or backend application wouldn’t be
hard at
all though, it’s mostly just handling the user that’s the problem.

As for firewalls; just use an active connection rather than passive. Job
done. You’ll loose the benefit of the different ports, though.

There’s more to it than that… but again, not nginx related.

6

On Sat, Aug 02, 2008 at 13:15:27, Mike M. said…

File:

I’ve had that work with POST, though I know that’s not what you’re
aiming for.

Shouldn’t that work? It keeps issuing a GET:

Just tried it myself, it does send GET rather than PUT. Weird.

dav_methods PUT;
create_full_put_path on;
dav_access all:r;

Do you mean to have ‘rw’ there?

7

FYI: this thread did not seem to get sent to the nginx mailing list for
some reason, oddly enough.

Michael wrote:

On Fri, Aug 01, 2008 at 13:10:46, Phillip B Oldham said…

Don’t want to sound off, but isn’t that what FTP was designed for?

I always tell people - FTP was designed for this. But this is a) not a
solution for our stakeholders and end-users and b) a common thing coming
up nowadays on many sites. you can’t just give out FTP accounts and hope
to have some smart system that understands when files are done, how to
integrate them into the system, etc.

FTP is such an awful protocol to deal with regarding firewalls, I’d much
rather
go the way the original poster is and use HTTP PUT for insecure stuff,
and SFTP
for secure stuff.

Exactly. and I believe since PUT uses HTTP, it can be sent over a secure
channel as well just the same.

But 2GB? That’s huge… a lot for an end-user to just sit and watch
without
clicking reload or something.

That’s why an intelligent uploader would handle the re-trying/etc. until
it was complete. Machines don’t get bored :slight_smile:

You may have to test this, but I’d assume the maint-stream browsers support
PUT via the XMLHTTPRequest javascript object.

I was just doing some of this myself yesterday - doesn’t seem like
there’s a
way to do file uploads with AJAX, but you can easily make a form with a
target
iframe… but this is really offtopic for the nginx list.

I don’t believe it can either. I am fine with it not supporting that as
well.

Also if anyone has any pointers on how to get nginx to accept PUTs… I
was trying to proof of concept this quick with an HTML form + simple PHP
script:

File:

Shouldn’t that work? It keeps issuing a GET:

192.168.1.2 - - [02/Aug/2008:04:08:07 -0700] GET /put/work.php? HTTP/1.1
“200” 23 “http://192.168.1.3/put/” “Mozilla/4.0 (compatible; MSIE 6.0;
Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR
3.0.04506.30; .NET CLR 3.0.04506.648)” “-”

Seems a bit odd. I’ve done this in nginx config too:

dav_methods PUT;
create_full_put_path on;
dav_access all:r;

(nginx is compiled with the dav module)

(I figured I’d at least see a PUT request with a failure)

8

On 8/2/08, Michael [email protected] wrote:

I’ve had that work with POST, though I know that’s not what you’re aiming for.

Yeah obviously not standard post multipart/enc-type stuff

Just tried it myself, it does send GET rather than PUT. Weird.

This is my issue :slight_smile: Hoping someone can help say “duh stupid - you missed
this”

dav_methods PUT;
create_full_put_path on;
dav_access all:r;

Do you mean to have ‘rw’ there?

No; at first I was thinking it would try to write and give me access
denied, or perhaps pass the data to the PHP script and not just
clobber/overwrite the script. But at least I’d see which activity it
chose first.

9

On 8/2/08, Michael [email protected] wrote:

Do you mean to have ‘rw’ there?

Nope :slight_smile:

Got it to work:

nginx access log:

192.168.1.3 - - [02/Aug/2008:11:39:53 -0700] PUT /put/work.php
HTTP/1.1 “200” 5 “-” “-” “-”

PHP curl script to test (ran from shell):

$file = ‘china.txt’;
$length = filesize($file);
$ch = curl_init();
$url = “http://192.168.1.3/put/work.php”;
$fh = fopen($file, ‘r’);
curl_setopt($ch, CURLOPT_INFILE, $fh);
curl_setopt($ch, CURLOPT_INFILESIZE, $length);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_PUT, 1);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_exec($ch);
curl_close($ch);

i believe you can use a @ or something for the input to a curl script
(instead of opening a filehandle) and it will consider that a file on
the filesystem, but i was just looking for a quick test script. it
will actually be an in-browser applet uploading this, i just wanted to
pick apart the receiving end and test it can be done.

PHP work.php script to receive (VERY crappy code, just trying
different things out):

$fname = ‘/tmp/uploadtest.txt’;
$data = ‘’;
$putdata = fopen(‘php://input’, ‘r’);
while(!feof($putdata)) {
$data .= fread($putdata,1024);
}
fclose($putdata);
file_put_contents($fname, $data);

exit();

10

On 8/2/08, Michael [email protected] wrote:

Just read another email I had posted to the PHP list.

"> I can appreciate why one might imagine otherwise, but XHTML 1.x forms
only

support GET and POST. GET and POST are the only allowed values for the
“method” attribute."

Sounds like it’s a browser/HTML thing. So I need to use curl/another
tool to test this. Oh well :slight_smile:

11

On 8/2/08, Michael [email protected] wrote:

On Sat, Aug 02, 2008 at 11:13:15, mike said…

Sounds like it’s a browser/HTML thing. So I need to use curl/another
tool to test this. Oh well :slight_smile:

Ah darn, that’s prety limiting. Nice that we know nginx can do this though :slight_smile:

well a browser doesn’t have the functionality we’re looking for anyway:

  • multiple file upload
  • progress bar
  • retransmission on timeout, failure, etc.

i just wanted to see if i could get the PHP server side piece of it as
clean and working as possible since we’re still not sure what our
applet-based client will be yet.

12

On Sat, Aug 02, 2008 at 11:13:15, mike said…

Sounds like it’s a browser/HTML thing. So I need to use curl/another
tool to test this. Oh well :slight_smile:

Ah darn, that’s prety limiting. Nice that we know nginx can do this
though :slight_smile:

13

On 8/4/08, Phillip B Oldham [email protected] wrote:

PUT support for forms is down to the browser. IE doesn’t support it, but I’m
pretty sure FF/Safari/Opera do.

Actually, I got my answer from the PHP list. Someone looked it up and
it’s not in W3C spec for XHTML. Only GET/POST are. People want to put
PUT/DELETE in, but it’s not supported.

This was just to make a simple test script for myself anyway. I wound
up just using curl via PHP to PUT the file to the work.php script to
mess around with it.

14

Michael wrote:

Just tried it myself, it does send GET rather than PUT. Weird.

PUT support for forms is down to the browser. IE doesn’t support it, but
I’m pretty sure FF/Safari/Opera do.

Phillip B Oldham
The Activity People
[email protected] mailto:[email protected]

Policies

This e-mail and its attachments are intended for the above named
recipient(s) only and may be confidential. If they have come to you in
error, please reply to this e-mail and highlight the error. No action
should be taken regarding content, nor must you copy or show them to
anyone.

This e-mail has been created in the knowledge that Internet e-mail is
not a 100% secure communications medium, and we have taken steps to
ensure that this e-mail and attachments are free from any virus. We must
advise that in keeping with good computing practice the recipient should
ensure they are completely virus free, and that you understand and
observe the lack of security when e-mailing us.

15

On Mon, Aug 04, 2008 at 09:30:33, Phillip B Oldham said…

Just tried it myself, it does send GET rather than PUT. Weird.

PUT support for forms is down to the browser. IE doesn’t support it, but
I’m pretty sure FF/Safari/Opera do.

I tested with Firefox 3.

I’m sure you could use AJAX to do it (something like prototype.js), but
in a
plain form, it doesn’t seem to work.

Also, it’s not real useful if the one of the most popular browsers
doesn’t
support it, yeah?

16

On 8/4/08, Michael [email protected] wrote:

I tested with Firefox 3.

They went all Microsoft then and started implementing things prior to
them being W3C :stuck_out_tongue:

I’m sure you could use AJAX to do it (something like prototype.js), but in a
plain form, it doesn’t seem to work.

AJAX would be neat for this but again, can’t do it in just a plain old
browser.

Also, it’s not real useful if the one of the most popular browsers doesn’t
support it, yeah?

Correct. and it doesn’t have the bells and whistles previously
mentioned:

  • multiple file upload
  • progress bar
  • retransmission on timeout, failure, etc.

It would be great if even single files could be re-transmitted, etc. I
am sure I could get buyoff from people if we had a fully non-applet
required method for this. But we don’t right now. It still requires
Java, Flash, or some other thick client crap.

17

Thanks…

I’ve put together what I think is a decent solution below:
http://michaelshadle.com/2008/08/26/large-file-uploads-over-http-the-final-solution-i-think/

that jupload might be worthwhile as it sounds like it already has the
splitting idea but i haven’t looked into it. this should allow for
anyone to support any file sizes (within filesystem limits) and if
something gets off the ground i am quite sure my company would sponsor
someone to write an nginx module to offload PHP-based handling of the
file uploading…

18

mike <mike503@…> writes:

There’s a variety of Java-based PUT uploaders.

So far, we haven’t found any Flash ones (we’d love to NOT use Java) -
but there is a way to do it apparently, we just can’t find anyone
who’s done it yet.

Check out SWFUPLOAD – swfupload.org - flash based (with some
javascript);
relatively robust; out of the box it’s not persistent, but you can get
around
that with re-submitting failed uploads using some Javascript-fu.

You might want to check jupload.sf.net, which is java based, however
it
handles HTTP or FTP upload methods, can do chunk-of-a-file-at-a-time,
and
probably could be made as robust as you’d like it to be.

19

mike wrote:

I’ve put together what I think is a decent solution below:
The Life and Times of Michael Shadle » Large file uploads over HTTP – the final solution (I think)

  1. multipart/form-data does not bloat the size of the file, it doesn’t
    encode anything. rfc 1867 doesn’t explicitly say that there any encoding
    should be applied;
  2. the ideal solution is to have byte ranges instead of Segment ID,
    since concatenation of parts is not a scalable operation. With byte
    ranges the server will be able to put the part into the proper place in
    the file, while leaving other parts empty. I.e. if I have two parts
    with byte ranges 0-19999/80000 and 40000-59999/80000, I can lseek to the
    second part and start writing two parts simultaneously:

|<-- part1 0-19999/80000 -->|<-- zeroes 000000 -->|<-- part2
40000-59999/80000 -->|

20

On Fri, Sep 5, 2008 at 11:48 PM, mike [email protected] wrote:

Thanks…

I’ve put together what I think is a decent solution below:
The Life and Times of Michael Shadle » Large file uploads over HTTP – the final solution (I think)

Interesting. This is exactly what we have been doing to get files
transmitted over satellite for many years ;-). Ofcourse the problem
there is one of no ACKs or NACKs during the transmission – they must
be batched up and sent via backchannel after the transmission has
ended, for retransmit. If this is something that interests you I would
suggest looking at protocols like NORM.

Cheers
Kon