Upload module parameters issue

Server NGINX
1

Thanks very much, that really helped!

On Mon Oct 3 19:04:22 UTC 2011, Andrew H. wrote:

Hi there,

  • upload_set_form_field “${upload_field_name}_name” $upload_file_name;> >
    upload_set_form_field “${upload_field_name}_content_type” $upload_content_type;*>
  • upload_set_form_field “${upload_field_name}_path” $upload_tmp_path;> >
    upload_aggregate_form_field “${upload_field_name}_sha1” $upload_file_sha1;> >
    upload_aggregate_form_field “${upload_field_name}_size” $upload_file_size;*>
    Note that “resumable” can apply to a single file, without any
    $upload_field_name set, so you’ll get things like “_name” and
    “_content_type” in your final post content.

What would be the right way to get the $upload_field_name populated?
Or should I even worry about that? I don’t imagine that my usage will
care about it (I’m going to be including the module and uploading a
single file at a time), however I’d like the library code to not suck
too much for the next guy.

  •    top_bound = plus_chunk if plus_chunk < self.total_file_size else

self.total_file_size - 1

     return 'bytes %d-%d/%d' % (self.last_byte_uploaded, top_bound,

self.total_file_size)

@property
  • Please note that despite the error message on line 680, the uploaded file
    is*> >* byte-for-byte identical to the source file. Error message follows:> > *>
  • 2011/10/03 10:58:46 [error] 8111#0: 35 file offset at the end of a part> >*
    210000 does not match the end specified range 204796-210000/210000, client:> >
    10.178.51.115, server: account.nutricateonline.com, request: “POST*> >*
    /rspool/?a=b&c=d HTTP/1.0”, host: “209.114.46.109”*>
    The range is expected to be 204796-209999/210000, as patched above.

Patch applied, thanks! I also updated my UTs to reflect this change
(and the 50kb default chunk size).

  • I think that having request.FILES empty is correct. The two parameters in*>
  • the GET dictionary are from the command line (also correct). However I think*>
  • the upload parameters should be represented somewhere. Can anyone suggest*> >*
    next steps?*>
    They’ll be in the POST dictionary when the client is right.

Yup, they’re there now. Perfect!

Note that you are also double-sending the last byte of each chunk –
make chunk_size be 1 or 2 and send a small file, and you’ll see the
problem. The receiving side can handle the double-send, so it’s an
inefficiency rather than a protocol error.

I’ll root this bug out later, for now it appears to be working
end-to-end.

2

On Mon Oct 3 19:04:22 UTC 2011, Andrew H. wrote:

Hi there,

upload_set_form_field “${upload_field_name}_name” $upload_file_name;
upload_set_form_field “${upload_field_name}_content_type” $upload_content_type;
upload_set_form_field “${upload_field_name}_path” $upload_tmp_path;
upload_aggregate_form_field “${upload_field_name}_sha1” $upload_file_sha1;
upload_aggregate_form_field “${upload_field_name}_size” $upload_file_size;

Note that “resumable” can apply to a single file, without any
$upload_field_name set, so you’ll get things like “_name” and
“_content_type” in your final post content.

I upload to the system using the following (see
https://github.com/SmartReceipt/py_lightweight_uploader):

One important off-by-one error there:

— py_lightweight_uploader.py 2011-10-03 23:37:18.000000000 +0100
+++ py_lightweight_uploader.py 2011-10-03 23:40:24.000000000 +0100
@@ -188,7 +188,7 @@
@property
def next_content_range(self):
plus_chunk = self.last_byte_uploaded + self.chunk_size - 1

  •    top_bound = plus_chunk if plus_chunk < self.total_file_size

else self.total_file_size

  •    top_bound = plus_chunk if plus_chunk < self.total_file_size

else self.total_file_size - 1
return ‘bytes %d-%d/%d’ % (self.last_byte_uploaded, top_bound,
self.total_file_size)

 @property

Please note that despite the error message on line 680, the uploaded file is
byte-for-byte identical to the source file. Error message follows:

2011/10/03 10:58:46 [error] 8111#0: *35 file offset at the end of a part
210000 does not match the end specified range 204796-210000/210000, client:
10.178.51.115, server: account.nutricateonline.com, request: “POST
/rspool/?a=b&c=d HTTP/1.0”, host: “209.114.46.109”

The range is expected to be 204796-209999/210000, as patched above.

I think that having request.FILES empty is correct. The two parameters in
the GET dictionary are from the command line (also correct). However I think
the upload parameters should be represented somewhere. Can anyone suggest
next steps?

They’ll be in the POST dictionary when the client is right.

Note that you are also double-sending the last byte of each chunk –
make chunk_size be 1 or 2 and send a small file, and you’ll see the
problem. The receiving side can handle the double-send, so it’s an
inefficiency rather than a protocol error.

But with 50kB chunks, that last range should really be
204800-209999/210000.

Good luck with it,

f

Francis D. [email protected]

3

On Tue, Oct 4, 2011 at 10:06 AM, Francis D. [email protected]
wrote:

What would be the right way to get the $upload_field_name populated?
Or should I even worry about that? I don’t imagine that my usage will
care about it (I’m going to be including the module and uploading a
single file at a time), however I’d like the library code to not suck
too much for the next guy.

I’d say “don’t worry about it”.

Done. I refactored the code to clean it up a little and correct that
overlap
bug. It’s on github and can be forked if anyone wants to go further.

My last (I think) issue is that I’m seeing most, but not all of the
upload
form fields get set. The sha1, md5 and crc32 aggregates below are
blanks.

request.POST <QueryDict: {
u’upload_name’: [u’py_lightweight_uploader.py’],
u’upload_size’: [u’11345’],
u’upload_sha1’: [u’‘],
u’upload_content_type’: [u’text/x-python’],
u’upload_path’: [u’/var/lib/nginx/resumable_download/7/64565707’],
u’upload_md5’: [u’‘],
u’upload_crc32’: [u’']}>

I see that libssl is installed on the server. I don’t see any entries in
the
error log that would make me think there is anything failing here. I
have
the following in the nginx config:

upload_set_form_field upload_name $upload_file_name;
upload_set_form_field upload_content_type $upload_content_type;
upload_set_form_field upload_path $upload_tmp_path;
upload_aggregate_form_field upload_sha1 $upload_file_sha1;
upload_aggregate_form_field upload_size $upload_file_size;
upload_aggregate_form_field upload_crc32 $upload_file_crc32;
upload_aggregate_form_field upload_md5 $upload_file_md5;

I based the above on the documentation and example config. What am I
missing
here?

Andrew

4

On Mon, Oct 03, 2011 at 05:35:05PM -0700, Andrew H. wrote:

Hi there,

Thanks very much, that really helped!

Note that “resumable” can apply to a single file, without any
$upload_field_name set, so you’ll get things like “_name” and
“_content_type” in your final post content.

What would be the right way to get the $upload_field_name populated?
Or should I even worry about that? I don’t imagine that my usage will
care about it (I’m going to be including the module and uploading a
single file at a time), however I’d like the library code to not suck
too much for the next guy.

I’d say “don’t worry about it”.

To upload multiple files, you use a form, which will set
$upload_field_name for each file.

Resumable upload with a single file just needs a single set of
variables,
so an empty prefix is fine.

All the best,

f

Francis D. [email protected]