We're making changes to the Community, so you may have received some notifications - thanks for your patience and welcome back. Learn more here.
Forum Discussion
Custom Tool - Slow File Upload Speed
Hello Greg-DB
We have developed a tool to help our customers move data from Windows File System to Dropbox.
The upload is done in batches of 8 files using the Dropbox APIs - UploadSessionStartAsync, UploadSessionFinishBatchAsync and UploadSessionFinishBatchCheckAsync. Uploads were tested keeping the chunk size between 10-40 MB.
The upload speed we experienced using our tool varied from 1 Mbps to 10 Mbps (Megabits per second) which was very slow.
We want our tool to upload data in tune of several TBs which would take unreasonable time to complete.
Our experience with a third party tool - CFP (Cloud Fast Path) was very good as we experienced speeds of up to 210 Mbps (Megabits per second) on the same machine.
We tested both tools on a machine in our network as well as on an Azure VM.
How can we optimize and improve our tool to achieve faster upload speeds ?
Thanks,
Gagan
Hello Greg-DB
Thank you for the reply !!!
We used the new Dropbox API - UploadSessionStartBatchAsync and this helped in reducing the "too_many_requests" errors.
We tested 2.2 TB data sample containing 427,002 files for migration using our custom app.
Last time we faced 26,000 failures which has been reduced to 1,753 in the latest test.What more can be done to further reduce failures due to "too_many_requests" error response from Dropbox.
We do wait and retry as shown in the last code snippet.Thanks,
Gagan
- Greg-DB
Dropbox Staff
Check out the Performance Guide for some optimizations you can make to upload in a performant manner.
Additionally, for uploading large files, check out the new "concurrent" UploadSessionType option, which lets you upload multiple pieces of a single file/upload session in parallel. There's also a new UploadSessionStartBatchAsync for starting multiple upload sessions at once. These haven't been added to the Performance Guide yet, but they can also be helpful.
Hello Greg-DB
Thank-you for your reply.
I went through your suggestions and the Performance guide.
I used the new "concurrent" UploadSessionType option and that increased the upload speeds.After the above change, for 1.5 GB and 6481 Files on Azure VM we got Sync Time = 10 mins which earlier was above 50 minutes.
For 1 TB and 446 Files, Sync Time = 2 hours 47 mins which was very fast.Upload speeds were between 300-700 Mbps and sometimes even 1.5 Gbps which is way above what we got earlier i.e. 1-7 Mbps.
However, this did not last long as few hours later when I tested again with same data, upload speeds were back to 1-7 Mbps.What could be the reason for the fallback to lower upload speeds of 1-7 Mbps ?
Does Dropbox throttles multiple file uploads coming from the same IP or App ?
However, I did not encounter any upload failures or Rate Limit exceptions.I know Dropbox has a rate limit policy which can be found out using https://www.dropbox.com/developers/documentation/http/teams#team-features-get_values
For the team we tested with the Upload Rate Limit was reported as 1,000,000,000 (1 billion).We tested uploading files > 6000 using Batch upload APIs (with UploadSessionType:concurrent) for one of the Team account member. The upload speeds were again between 1-7 Mbps when we tested with another member of the same Team.
In real-life scenarios, the number of files can be as large as 200,000 files.We are using a Dropbox Teams App for one-time Admin authentication and token generation.
I came across this forum post: Forum PostOur Dropbox App is Development stage as of now and we are not Dropbox partners. Can that be one of the reasons ?
Can you please help us figure out the real reason for the reduction in upload speeds after we got great speeds for some hours.
Thanks,
Gagan- Greg-DB
Dropbox does not throttle connection speeds like that. While Dropbox has a rate limiting system in place, that operates on the basis of calls per time period, and any rate limited calls would result in a specific error, not limited connection speed.
Also, an app's development/production status does not affect connection speed, nor does whether or not it is a partner app.
As for your actual connection speed, I'm afraid I can't offer much insight based on this. Your connection speed to Dropbox depends on the routing you get between your ISP and our servers, and may be slower than your ISP's rated speeds. Sometimes resetting or retrying your connection gets you a different route and better speeds, but that is outside of our control. Some ISPs also throttle sustained connections so if you see an initial high connection speed followed by lower speeds, that could be the reason.
Beyond that, you may want to add some logging to see exactly what operations are being performed (or not) when your overall speed drops. It'spossible this is due to how you're ordering/parallelizing the calls. For instance, you might be running multiple sessions in parallel initially, but then "run out" of things to run in parallel, for example. If something on the Dropbox API doesn't appear to be working properly though, please share whatever output you can showing the issue so we can take a look.
About Dropbox API Support & Feedback
Find help with the Dropbox API from other developers.
