You might see that the Dropbox Community team have been busy working on some major updates to the Community itself! So, here is some info on what’s changed, what’s staying the same and what you can expect from the Dropbox Community overall.

Forum Discussion

Stefan D.6's avatar
Stefan D.6
New member | Level 2
9 years ago

Concurrent uploads completely broken since V2 api.

It appears that uploading multiple different files to the same dropbox at the same time, is completely broken in the V2 api. Even if 2 different computers are uploading a file. Please see the following testcase:

 

 

#!/bin/env python2
import tempfile
 
import dropbox
 
FILE = ""
ACCESSTOKEN = ""
 
client = dropbox.Dropbox(ACCESSTOKEN)
 
for i in range(10000):
    with tempfile.NamedTemporaryFile() as fp:
        metadata, restdata = client.files_download(FILE)
        fp.write(restdata.content)
        fp.write("{}\n".format(i))
        fp.seek(0)
        client.files_upload(fp, FILE, dropbox.files.WriteMode('overwrite', None))
 
 
 
 
Steps to reproduce:
fill in your accesstoken.
fill in in the variables FILE, to a file that exists ('/foo/test.txt'). Start the program.
Change the variable FILE to another file that exists ('/foo/test2.txt'). Start another instance of the program
 
 
What should happen is that both test.txt and test2.txt are continuously appended with an new integer followed by a newline. But in practice, the files occasionally end up completely empty after an update cycle. The download of the file does not fail. client.files_upload simply creates an completely empty file.
 
This behavior is not reproduceable when running only one instance of this testcase. Running the testcase on one computer or on different computers/ip's with different/same accesstokens/accounts does not make a difference in my testing as long as the same dropbox (or dropbox folder?) is accessed.
 
Time to reproduce: about 10 updates. Might depend on your internet connection and proximity to dropbox' servers. (not tested)
 

Tested with dropbox 6.8.0 on windows 10 and dropbox 6.5.0 on debian. Only tested with python 2.7

  • Greg-DB's avatar
    Greg-DB
    Icon for Dropbox Staff rankDropbox Staff

    Thanks for the report Stefan! I just tried running two of these though and I'm not seeing the same behavior, using the same version of Python and the Dropbox library.

    How are you checking the file size to see the unexpected empty file? Can you add some logging and share the output showing the issue? In particular, it might be good to check the size in three places: in the metadata returned by files_download, of fp itself, and in the metadata returned by files_upload. That way we can see exactly when/where the empty file originates.

    Thanks in advance! 

  • Stefan D.6's avatar
    Stefan D.6
    New member | Level 2

    files_download always returns an error code 200 and the correct file size. Calling tell() on the file pointer after writing also always returns the correct file size. The metadata returned by files_upload (occasionally) claims to have resulted in a zero bytes upload.

    Based on the behavior i see in my app, i am quite confident that is is the upload that fails.

    I also ran a few experiments with the files in different folders / in shared folders, but that doesn't seem to make a difference.

    Is there any internel dropbox lib logging i can enable?

  • Greg-DB's avatar
    Greg-DB
    Icon for Dropbox Staff rankDropbox Staff

    You can add the following to enable some debugging output that would show the HTTP requests:

    import httplib
    httplib.HTTPConnection.debuglevel = 1

  • Stefan D.6's avatar
    Stefan D.6
    New member | Level 2

    Good idea. I modified the inner loop to this:

     

    for i in range(10000):
        with tempfile.NamedTemporaryFile() as fp:
            metadata, restdata = client.files_download(FILE)
            fp.write(restdata.content)
            fp.write("{}\r".format(i))

            fp.seek(0)

            print '=================================='
            httplib.HTTPConnection.debuglevel = 1
            md2 = client.files_upload(fp, FILE, dropbox.files.WriteMode('overwrite', None))
            httplib.HTTPConnection.debuglevel = 0
            assert md2.size

     

     

     

    Output:

     

     

    ==================================
    send: 'POST /2/files/upload HTTP/1.1\r\nHost: content.dropboxapi.com\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: OfficialDropboxPythonSDKv2/6.8.0\r\nContent-Type: application/octet-stream\r\nDropbox-API-Arg: {"path": "/foo/test.txt", "mode": {".tag": "overwrite"}, "autorename": false, "mute": false}\r\nAuthorization: Bearer CENSORED\r\nContent-Length: 38\r\n\r\n'
    send: <open file '<fdopen>', mode 'w+b' at 0x0000000003A57390>
    sendIng a read()able
    reply: 'HTTP/1.1 200 OK\r\n'
    header: Server: nginx
    header: Date: Tue, 20 Sep 2016 17:28:13 GMT
    header: Content-Type: application/json
    header: Transfer-Encoding: chunked
    header: Connection: keep-alive
    header: pragma: no-cache
    header: cache-control: no-cache
    header: X-Server-Response-Time: 1122
    header: X-Dropbox-Request-Id: 754c6605f013cd1dcad17ad97551c672
    header: X-Robots-Tag: noindex, nofollow, noimageindex
    header: Content-Encoding: gzip
    ==================================
    send: 'POST /2/files/upload HTTP/1.1\r\nHost: content.dropboxapi.com\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: OfficialDropboxPythonSDKv2/6.8.0\r\nContent-Type: application/octet-stream\r\nDropbox-API-Arg: {"path": "/foo/test.txt", "mode": {".tag": "overwrite"}, "autorename": false, "mute": false}\r\nAuthorization: Bearer CENSORED\r\nContent-Length: 41\r\n\r\n'
    send: <open file '<fdopen>', mode 'w+b' at 0x0000000003771E40>
    sendIng a read()able
    reply: 'HTTP/1.1 200 OK\r\n'
    header: Server: nginx
    header: Date: Tue, 20 Sep 2016 17:28:14 GMT
    header: Content-Type: application/json
    header: Transfer-Encoding: chunked
    header: Connection: keep-alive
    header: pragma: no-cache
    header: cache-control: no-cache
    header: X-Server-Response-Time: 837
    header: X-Dropbox-Request-Id: d716609d9fdfd3cb17001967b6f82009
    header: X-Robots-Tag: noindex, nofollow, noimageindex
    header: Content-Encoding: gzip
    ==================================
    send: 'POST /2/files/upload HTTP/1.1\r\nHost: content.dropboxapi.com\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: OfficialDropboxPythonSDKv2/6.8.0\r\nContent-Type: application/octet-stream\r\nDropbox-API-Arg: {"path": "/foo/test.txt", "mode": {".tag": "overwrite"}, "autorename": false, "mute": false}\r\nAuthorization: Bearer CENSORED\r\nContent-Length: 44\r\n\r\n'
    send: <open file '<fdopen>', mode 'w+b' at 0x0000000003A57390>
    sendIng a read()able
    reply: 'HTTP/1.1 429 too_many_write_operations/..\r\n'
    header: Server: nginx
    header: Date: Tue, 20 Sep 2016 17:28:16 GMT
    header: Content-Type: application/json
    header: Transfer-Encoding: chunked
    header: Connection: keep-alive
    header: Retry-After: 1
    header: X-Dropbox-Request-Id: 99633340ed675344e6a7d07bad3ddb4b
    header: X-Robots-Tag: noindex, nofollow, noimageindex
    send: 'POST /2/files/upload HTTP/1.1\r\nHost: content.dropboxapi.com\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: OfficialDropboxPythonSDKv2/6.8.0\r\nContent-Type: application/octet-stream\r\nDropbox-API-Arg: {"path": "/foo/test.txt", "mode": {".tag": "overwrite"}, "autorename": false, "mute": false}\r\nAuthorization: Bearer CENSORED\r\nTransfer-Encoding: chunked\r\n\r\n'
    send: '0\r\n\r\n'
    reply: 'HTTP/1.1 200 OK\r\n'
    header: Server: nginx
    header: Date: Tue, 20 Sep 2016 17:28:18 GMT
    header: Content-Type: application/json
    header: Transfer-Encoding: chunked
    header: Connection: keep-alive
    header: pragma: no-cache
    header: cache-control: no-cache
    header: X-Server-Response-Time: 598
    header: X-Dropbox-Request-Id: 71ff38bee72b24a9afd7172830e85671
    header: X-Robots-Tag: noindex, nofollow, noimageindex
    header: Content-Encoding: gzip
    Traceback (most recent call last):
      File "C:/Users/Stefan/git/dropbox_test.py", line 28, in <module>
        assert md2.size
    AssertionError

     

    Seems that the upload fails for some reason (reply: 'HTTP/1.1 429 too_many_write_operations/..\r\n') and when it retries, it sends an "0\r\n\r\n" instead of "<open file '<fdopen>', mode 'w+b' at 0x0000000003A57390>" ?

  • Greg-DB's avatar
    Greg-DB
    Icon for Dropbox Staff rankDropbox Staff

    Thanks! That's very helpful. We're looking into it.