You might see that the Dropbox Community team have been busy working on some major updates to the Community itself! So, here is some info on what’s changed, what’s staying the same and what you can expect from the Dropbox Community overall.
Forum Discussion
Stefan D.6
9 years agoNew member | Level 2
Concurrent uploads completely broken since V2 api.
It appears that uploading multiple different files to the same dropbox at the same time, is completely broken in the V2 api. Even if 2 different computers are uploading a file. Please see the following testcase:
#!/bin/env python2
import tempfile
import dropbox
FILE = ""
ACCESSTOKEN = ""
client = dropbox.Dropbox(ACCESSTOKEN)
for i in range(10000):
with tempfile.NamedTemporaryFile() as fp:
metadata, restdata = client.files_download(FILE)
fp.write(restdata.content)
fp.write("{}\n".format(i))
fp.seek(0)
client.files_upload(fp, FILE, dropbox.files.WriteMode('overwrite', None))
Steps to reproduce:
fill in your accesstoken.
fill in in the variables FILE, to a file that exists ('/foo/test.txt'). Start the program.
Change the variable FILE to another file that exists ('/foo/test2.txt'). Start another instance of the program
What should happen is that both test.txt and test2.txt are continuously appended with an new integer followed by a newline. But in practice, the files occasionally end up completely empty after an update cycle. The download of the file does not fail. client.files_upload simply creates an completely empty file.
This behavior is not reproduceable when running only one instance of this testcase. Running the testcase on one computer or on different computers/ip's with different/same accesstokens/accounts does not make a difference in my testing as long as the same dropbox (or dropbox folder?) is accessed.
Time to reproduce: about 10 updates. Might depend on your internet connection and proximity to dropbox' servers. (not tested)
Tested with dropbox 6.8.0 on windows 10 and dropbox 6.5.0 on debian. Only tested with python 2.7
- Greg-DBDropbox Staff
Thanks for the report Stefan! I just tried running two of these though and I'm not seeing the same behavior, using the same version of Python and the Dropbox library.
How are you checking the file size to see the unexpected empty file? Can you add some logging and share the output showing the issue? In particular, it might be good to check the size in three places: in the metadata returned by files_download, of fp itself, and in the metadata returned by files_upload. That way we can see exactly when/where the empty file originates.
Thanks in advance!
- Stefan D.6New member | Level 2
files_download always returns an error code 200 and the correct file size. Calling tell() on the file pointer after writing also always returns the correct file size. The metadata returned by files_upload (occasionally) claims to have resulted in a zero bytes upload.
Based on the behavior i see in my app, i am quite confident that is is the upload that fails.
I also ran a few experiments with the files in different folders / in shared folders, but that doesn't seem to make a difference.
Is there any internel dropbox lib logging i can enable?
- Greg-DBDropbox Staff
You can add the following to enable some debugging output that would show the HTTP requests:
import httplib
httplib.HTTPConnection.debuglevel = 1 - Stefan D.6New member | Level 2
Good idea. I modified the inner loop to this:
for i in range(10000):
with tempfile.NamedTemporaryFile() as fp:
metadata, restdata = client.files_download(FILE)
fp.write(restdata.content)
fp.write("{}\r".format(i))
fp.seek(0)
print '=================================='
httplib.HTTPConnection.debuglevel = 1
md2 = client.files_upload(fp, FILE, dropbox.files.WriteMode('overwrite', None))
httplib.HTTPConnection.debuglevel = 0
assert md2.sizeOutput:
==================================
send: 'POST /2/files/upload HTTP/1.1\r\nHost: content.dropboxapi.com\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: OfficialDropboxPythonSDKv2/6.8.0\r\nContent-Type: application/octet-stream\r\nDropbox-API-Arg: {"path": "/foo/test.txt", "mode": {".tag": "overwrite"}, "autorename": false, "mute": false}\r\nAuthorization: Bearer CENSORED\r\nContent-Length: 38\r\n\r\n'
send: <open file '<fdopen>', mode 'w+b' at 0x0000000003A57390>
sendIng a read()able
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: nginx
header: Date: Tue, 20 Sep 2016 17:28:13 GMT
header: Content-Type: application/json
header: Transfer-Encoding: chunked
header: Connection: keep-alive
header: pragma: no-cache
header: cache-control: no-cache
header: X-Server-Response-Time: 1122
header: X-Dropbox-Request-Id: 754c6605f013cd1dcad17ad97551c672
header: X-Robots-Tag: noindex, nofollow, noimageindex
header: Content-Encoding: gzip
==================================
send: 'POST /2/files/upload HTTP/1.1\r\nHost: content.dropboxapi.com\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: OfficialDropboxPythonSDKv2/6.8.0\r\nContent-Type: application/octet-stream\r\nDropbox-API-Arg: {"path": "/foo/test.txt", "mode": {".tag": "overwrite"}, "autorename": false, "mute": false}\r\nAuthorization: Bearer CENSORED\r\nContent-Length: 41\r\n\r\n'
send: <open file '<fdopen>', mode 'w+b' at 0x0000000003771E40>
sendIng a read()able
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: nginx
header: Date: Tue, 20 Sep 2016 17:28:14 GMT
header: Content-Type: application/json
header: Transfer-Encoding: chunked
header: Connection: keep-alive
header: pragma: no-cache
header: cache-control: no-cache
header: X-Server-Response-Time: 837
header: X-Dropbox-Request-Id: d716609d9fdfd3cb17001967b6f82009
header: X-Robots-Tag: noindex, nofollow, noimageindex
header: Content-Encoding: gzip
==================================
send: 'POST /2/files/upload HTTP/1.1\r\nHost: content.dropboxapi.com\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: OfficialDropboxPythonSDKv2/6.8.0\r\nContent-Type: application/octet-stream\r\nDropbox-API-Arg: {"path": "/foo/test.txt", "mode": {".tag": "overwrite"}, "autorename": false, "mute": false}\r\nAuthorization: Bearer CENSORED\r\nContent-Length: 44\r\n\r\n'
send: <open file '<fdopen>', mode 'w+b' at 0x0000000003A57390>
sendIng a read()able
reply: 'HTTP/1.1 429 too_many_write_operations/..\r\n'
header: Server: nginx
header: Date: Tue, 20 Sep 2016 17:28:16 GMT
header: Content-Type: application/json
header: Transfer-Encoding: chunked
header: Connection: keep-alive
header: Retry-After: 1
header: X-Dropbox-Request-Id: 99633340ed675344e6a7d07bad3ddb4b
header: X-Robots-Tag: noindex, nofollow, noimageindex
send: 'POST /2/files/upload HTTP/1.1\r\nHost: content.dropboxapi.com\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: OfficialDropboxPythonSDKv2/6.8.0\r\nContent-Type: application/octet-stream\r\nDropbox-API-Arg: {"path": "/foo/test.txt", "mode": {".tag": "overwrite"}, "autorename": false, "mute": false}\r\nAuthorization: Bearer CENSORED\r\nTransfer-Encoding: chunked\r\n\r\n'
send: '0\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: nginx
header: Date: Tue, 20 Sep 2016 17:28:18 GMT
header: Content-Type: application/json
header: Transfer-Encoding: chunked
header: Connection: keep-alive
header: pragma: no-cache
header: cache-control: no-cache
header: X-Server-Response-Time: 598
header: X-Dropbox-Request-Id: 71ff38bee72b24a9afd7172830e85671
header: X-Robots-Tag: noindex, nofollow, noimageindex
header: Content-Encoding: gzip
Traceback (most recent call last):
File "C:/Users/Stefan/git/dropbox_test.py", line 28, in <module>
assert md2.size
AssertionErrorSeems that the upload fails for some reason (reply: 'HTTP/1.1 429 too_many_write_operations/..\r\n') and when it retries, it sends an "0\r\n\r\n" instead of "<open file '<fdopen>', mode 'w+b' at 0x0000000003A57390>" ?
- Greg-DBDropbox Staff
Thanks! That's very helpful. We're looking into it.
- Greg-DBDropbox Staff
About Dropbox API Support & Feedback
Find help with the Dropbox API from other developers.
5,884 PostsLatest Activity: 5 hours agoIf you need more help you can view your support options (expected response time for an email or ticket is 24 hours), or contact us on X or Facebook.
For more info on available support options for your Dropbox plan, see this article.
If you found the answer to your question in this Community thread, please 'like' the post to say thanks and to let us know it was useful!