You might see that the Dropbox Community team have been busy working on some major updates to the Community itself! So, here is some info on what’s changed, what’s staying the same and what you can expect from the Dropbox Community overall.
Forum Discussion
Custom Tool - Slow File Upload Speed
Hello Greg-DB
We have developed a tool to help our customers move data from Windows File System to Dropbox.
The upload is done in batches of 8 files using the Dropbox APIs - UploadSessio...
Hello Greg-DB
Thank you for the reply !!!
We used the new Dropbox API - UploadSessionStartBatchAsync and this helped in reducing the "too_many_requests" errors.
We tested 2.2 TB data sample containing 427,002 files for migration using our custom app.
Last time we faced 26,000 failures which has been reduced to 1,753 in the latest test.What more can be done to further reduce failures due to "too_many_requests" error response from Dropbox.
We do wait and retry as shown in the last code snippet.Thanks,
Gagan
Greg-DB
Dropbox Staff
Dropbox StaffYou can use UploadSessionStartBatchAsync to start up to 1000 upload sessions. If you have at least 1000 upload sessions to start, you can do so in one call; you do not need to lower that. Calling UploadSessionStartBatchAsync once only counts as one call, regardless of what you set numSessions to.
And you can call UploadSessionStartBatchAsync multiple times before calling UploadSessionFinishBatchAsync if needed. You don't need to wait for one batch to finish before starting another. Just keep in mind that you should only finish one batch per namespace at a time to avoid lock contention.
Hello Greg-DB
Thank you for the reply !!!
We used the new Dropbox API - UploadSessionStartBatchAsync and this helped in reducing the "too_many_requests" errors.
We tested 2.2 TB data sample containing 427,002 files for migration using our custom app.
Last time we faced 26,000 failures which has been reduced to 1,753 in the latest test.
What more can be done to further reduce failures due to "too_many_requests" error response from Dropbox.
We do wait and retry as shown in the last code snippet.
Thanks,
Gagan
- Greg-DB
It sounds like you're using all of the available batch endpoints for this use cases, so the only remaining change that comes to mind would be to increase the chunk size, so that you use fewer requests for sending the file data overall.
You can use a chunk size of up to 150 MB, but be aware that increasing the chunk size increases how long each request takes and consequently increases the chances of the network connection failing before it completes. And further, using a larger chunk size means more data needs to be sent on a retry.
About Dropbox API Support & Feedback
Find help with the Dropbox API from other developers.
