You might see that the Dropbox Community team have been busy working on some major updates to the Community itself! So, here is some info on what’s changed, what’s staying the same and what you can expect from the Dropbox Community overall.
Forum Discussion
How long should I wait for /upload_session/finish_batch/check to complete?
In order to complete a batch it is necessary to call upload_session/finish_batch/check . The docs for this don't give any guidance on
- How long to try calling this API for
- How often to call it
In the batch upload I'm developing for rclone I've arbitrarily chosen 2 minutes and 1 second so rclone checks for completion for a maximum of 120 times at 1 second intervals.
This seems to work very well, but occasionally I seem to need more than 120 seconds to complete a batch. I think this might be a bug at as the next batch complete returns a 500 error.
Here is the last transaction (which looks normal) before rclone gives up
2021/04/19 20:10:19 DEBUG : HTTP REQUEST (req 0xc001103900)
2021/04/19 20:10:19 DEBUG : POST /2/files/upload_session/finish_batch/check HTTP/1.1
Host: api.dropboxapi.com
User-Agent: rclone/v1.56.0-beta.5394.d16702566.fix-dropbox-batch-sync
Content-Length: 113
Authorization: XXXX
Content-Type: application/json
Accept-Encoding: gzip
2021/04/19 20:10:19 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/19 20:10:19 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/19 20:10:19 DEBUG : HTTP RESPONSE (req 0xc001103900)
2021/04/19 20:10:19 DEBUG : HTTP/2.0 200 OK
Content-Length: 23
Accept-Encoding: identity,gzip
Cache-Control: no-cache
Content-Type: application/json
Date: Tue, 20 Apr 2021 00:10:19 GMT
Server: envoy
X-Content-Type-Options: nosniff
X-Dropbox-Request-Id: f746ed86c87a4fdcaf660a12d19f1a24
X-Dropbox-Response-Origin: far_remote
X-Frame-Options: SAMEORIGIN
X-Server-Response-Time: 109
{".tag": "in_progress"}
2021/04/19 20:10:20 ERROR : Dropbox root 'jbod2': sync batch commit: failed to commit batch length 1: wait for batch failed after 120 tries: batch didn't complete
So rclone gives up here and tries to finish the next batch
2021/04/19 20:10:21 DEBUG : HTTP REQUEST (req 0xc001373b00) 2021/04/19 20:10:21 DEBUG : POST /2/files/upload_session/finish_batch HTTP/1.1 Host: api.dropboxapi.com User-Agent: rclone/v1.56.0-beta.5394.d16702566.fix-dropbox-batch-sync Content-Length: 907 Authorization: XXXX Content-Type: application/json Accept-Encoding: gzip 2021/04/19 20:10:21 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2021/04/19 20:10:21 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 2021/04/19 20:10:21 DEBUG : HTTP RESPONSE (req 0xc001373b00) 2021/04/19 20:10:21 DEBUG : HTTP/2.0 500 Internal Server Error Content-Length: 0 Accept-Encoding: identity,gzip Content-Security-Policy: sandbox allow-forms allow-scripts Content-Type: text/plain; charset=utf-8 Date: Tue, 20 Apr 2021 00:10:21 GMT Server: envoy X-Dropbox-Request-Id: c787b53ec12e42eba8e0d650a28086ff X-Dropbox-Response-Origin: far_remote
And immediately gets a 500 error. Rclone retries this 10 times then gives up.
Some guidance here would be appreciated
- What is the longest time to commit a batch?
- What is the best polling frequency for batch completion?
- Is the above normal or a bug?
This problem was reported in the rclone issue tracker #5008
Thanks
Nick
- Greg-DB
Dropbox Staff
There isn't a guaranteed maximum time that one of these batch jobs may take. The processing time will depend on a few factors, such as how many files there are, etc., so it can vary.
Likewise, there isn't a best frequency for how often to poll these jobs. Once a second is fine. For larger batches, which may take longer given the larger amount of work to do, you could increase that a bit if you wish.
The 500 Internal Server Error indicates an issue on the Dropbox servers, so we'll have to look into that. For reference though, can you let me know if that's transient or if you can reliably reproduce that error, e.g., if you try again now? Thanks in advance!
> There isn't a guaranteed maximum time that one of these batch jobs may take. The processing time will depend on a few factors, such as how many files there are, etc., so it can vary.
There was only one file in the batch above so for it to take 2 minutes seems a long time. But then I have no idea what is going on behind the scenes!
> Likewise, there isn't a best frequency for how often to poll these jobs. Once a second is fine. For larger batches, which may take longer given the larger amount of work to do, you could increase that a bit if you wish.
OK.
> The 500 Internal Server Error indicates an issue on the Dropbox servers, so we'll have to look into that. For reference though, can you let me know if that's transient or if you can reliably reproduce that error, e.g., if you try again now?
I got that trace second hand from an rclone tester, but I believe it only happens quite rarely when doing a lot of transfers over a long time.
Thanks
- Greg-DB
ncw wrote:
I got that trace second hand from an rclone tester, but I believe it only happens quite rarely when doing a lot of transfers over a long time.
ncw Thanks! That seems to match what I can see on our side. It appears this was a transient availability issue.
ncw wrote:
Rclone retries this 10 times then gives up.
How quickly does Rclone perform these retries on a 500 like this? If it's in relatively quick succession, you may want to implement an exponential backoff to give the service more time to recover when this does occur.
ncw wrote:
...
So rclone gives up here and tries to finish the next batch
...Hi ncw,
Key moment here may be that only one active batch could work at a moment for the same user. According your description the last running batch might not successfully finished yet (or unsuccessfully), while you are trying a new one! This might confuse Dropbox server and provokes some rare bug as a response. 🤷 As a workaround (temporary or not) you can stop any follow up ongoing automatic processing, signal ongoing error, and wait for new user command.
Let's hope this will be traced successfully and fixed soon.
About Dropbox API Support & Feedback
Find help with the Dropbox API from other developers.
