You might see that the Dropbox Community team have been busy working on some major updates to the Community itself! So, here is some info on what’s changed, what’s staying the same and what you can expect from the Dropbox Community overall.
Forum Discussion
Gags
3 years agoHelpful | Level 6
Custom Tool - Slow File Upload Speed
Hello Greg-DB
We have developed a tool to help our customers move data from Windows File System to Dropbox.
The upload is done in batches of 8 files using the Dropbox APIs - UploadSessio...
- 3 years ago
Hello Greg-DB
Thank you for the reply !!!
We used the new Dropbox API - UploadSessionStartBatchAsync and this helped in reducing the "too_many_requests" errors.
We tested 2.2 TB data sample containing 427,002 files for migration using our custom app.
Last time we faced 26,000 failures which has been reduced to 1,753 in the latest test.What more can be done to further reduce failures due to "too_many_requests" error response from Dropbox.
We do wait and retry as shown in the last code snippet.Thanks,
Gagan
gagsbh80
Explorer | Level 4
Hello Greg-DB
Thank-you for your reply.
I went through your suggestions and the Performance guide.
I used the new "concurrent" UploadSessionType option and that increased the upload speeds.
After the above change, for 1.5 GB and 6481 Files on Azure VM we got Sync Time = 10 mins which earlier was above 50 minutes.
For 1 TB and 446 Files, Sync Time = 2 hours 47 mins which was very fast.
Upload speeds were between 300-700 Mbps and sometimes even 1.5 Gbps which is way above what we got earlier i.e. 1-7 Mbps.
However, this did not last long as few hours later when I tested again with same data, upload speeds were back to 1-7 Mbps.
What could be the reason for the fallback to lower upload speeds of 1-7 Mbps ?
Does Dropbox throttles multiple file uploads coming from the same IP or App ?
However, I did not encounter any upload failures or Rate Limit exceptions.
I know Dropbox has a rate limit policy which can be found out using https://www.dropbox.com/developers/documentation/http/teams#team-features-get_values
For the team we tested with the Upload Rate Limit was reported as 1,000,000,000 (1 billion).
We tested uploading files > 6000 using Batch upload APIs (with UploadSessionType:concurrent) for one of the Team account member. The upload speeds were again between 1-7 Mbps when we tested with another member of the same Team.
In real-life scenarios, the number of files can be as large as 200,000 files.
We are using a Dropbox Teams App for one-time Admin authentication and token generation.
I came across this forum post: Forum Post
Our Dropbox App is Development stage as of now and we are not Dropbox partners. Can that be one of the reasons ?
Can you please help us figure out the real reason for the reduction in upload speeds after we got great speeds for some hours.
Thanks,
Gagan
Greg-DB
3 years agoDropbox Staff
Dropbox does not throttle connection speeds like that. While Dropbox has a rate limiting system in place, that operates on the basis of calls per time period, and any rate limited calls would result in a specific error, not limited connection speed.
Also, an app's development/production status does not affect connection speed, nor does whether or not it is a partner app.
As for your actual connection speed, I'm afraid I can't offer much insight based on this. Your connection speed to Dropbox depends on the routing you get between your ISP and our servers, and may be slower than your ISP's rated speeds. Sometimes resetting or retrying your connection gets you a different route and better speeds, but that is outside of our control. Some ISPs also throttle sustained connections so if you see an initial high connection speed followed by lower speeds, that could be the reason.
Beyond that, you may want to add some logging to see exactly what operations are being performed (or not) when your overall speed drops. It'spossible this is due to how you're ordering/parallelizing the calls. For instance, you might be running multiple sessions in parallel initially, but then "run out" of things to run in parallel, for example. If something on the Dropbox API doesn't appear to be working properly though, please share whatever output you can showing the issue so we can take a look.
- gagsbh803 years agoExplorer | Level 4
Hello Greg-DB
We tested with Upload Session Finish list count of Max = 1000 in Upload SessionFinishBatchAsync and we started getting better speeds. We are continuing our testing and will let you know if we need further help.
Thanks,
Gagan
- Gags3 years agoHelpful | Level 6
Hello Greg-DB
Upload speeds are not major concern as of now but failures are.
We did a 2 TB migration using our custom tool and encountered "too_many_requests" failure as the end stages of the migration.
Thousands of files reported such failures which won't be acceptable to users.
We upload many files in parallel but using UploadSessionStartAsync, UploadSessionAppendV2Async & UploadSessionFinishBatchAsync APIs.
We do a wait and retry for Rate Failures as recommended by Dropbox and so we thought we should be good.
However, many files still fail due to "too_many_requests".
I am writing the relevant C# code snipped we have implemented for your reference.
Can you please give us some direction to minimize the Rate Failure exceptions and hence reduce upload failures.
public async Task SendUploadFragment(DropboxUploadSession uploadSession, byte[] fragmentBuffer, long offset) { string errorlog = ConfigurationManager.AppSettings["ErrorLog"]; int retry = 3; DropboxApi.DropboxClient clientadmin = null; DropboxApi.DropboxTeamClient teamclient = null; try { for (int i = 1; i <= retry; i++) { try { Item data = new Item(); if (!String.IsNullOrEmpty(this.CurrentToken.RefreshToken)) { string appkey = SyncProDropboxAppKey; string appsecret = SyncProDropboxAppSecret; teamclient = new DropboxApi.DropboxTeamClient(this.CurrentToken.RefreshToken, appkey, appsecret); } else { teamclient = new DropboxApi.DropboxTeamClient(this.CurrentToken.AccessToken); } clientadmin = teamclient.AsMember(this.TeamMemberId); using (var memStream = new MemoryStream(fragmentBuffer, 0, fragmentBuffer.Length)) { //1. When No Upload session has been created so far if (String.IsNullOrEmpty(uploadSession.SessionId)) { try { //If multiple fragment session if (!uploadSession.islastfragment) { UploadSessionStartArg args = new UploadSessionStartArg(); UploadSessionType sessiontype = new UploadSessionType(); var result = await clientadmin.Files.UploadSessionStartAsync(false, sessiontype.AsConcurrent, memStream); uploadSession.SessionId = result.SessionId; } else { //If single fragment session UploadSessionType sessiontype = new UploadSessionType(); var result = await clientadmin.Files.UploadSessionStartAsync(true, sessiontype.AsConcurrent, memStream); uploadSession.SessionId = result.SessionId; var cursor2 = new UploadSessionCursor(uploadSession.SessionId, (ulong)(offset + fragmentBuffer.Length)); UploadSessionFinishArg arg = new UploadSessionFinishArg(cursor2, new CommitInfo(uploadSession.Item.PathDisplay)); args.Add(arg); break; } } catch (Exception ex) { //Log Exception if (ex is DropboxApi.RateLimitException) { try { Thread.Sleep(((DropboxApi.RateLimitException)ex).RetryAfter * 1000); } catch (Exception ex1) { } } Thread.Sleep(1000); throw ex; } } else { var cursor = new UploadSessionCursor(uploadSession.SessionId, (ulong)offset); //If last fragment of session if (uploadSession.islastfragment) { try { await clientadmin.Files.UploadSessionAppendV2Async(cursor, true, memStream); var cursor2 = new UploadSessionCursor(uploadSession.SessionId, (ulong)(offset + fragmentBuffer.Length)); UploadSessionFinishArg arg = new UploadSessionFinishArg(cursor2, new CommitInfo(uploadSession.Item.PathDisplay)); args.Add(arg); break; } catch (Exception ex) { //Log Exception if (ex is DropboxApi.RateLimitException) { try { Thread.Sleep(((DropboxApi.RateLimitException)ex).RetryAfter * 1000); } catch (Exception ex1) { } } Thread.Sleep(1000); } } else //If not the last fragment of Session { try { await clientadmin.Files.UploadSessionAppendV2Async(cursor, false, memStream); break; } catch (Exception ex) { //Log Exception if (ex is DropboxApi.RateLimitException) { try { Thread.Sleep(((DropboxApi.RateLimitException)ex).RetryAfter * 1000); } catch (Exception ex1) { } } Thread.Sleep(1000); } } } } break; } catch (Exception ex) { string isgeneratelog = Global.IsErrorLog == false ? "0" : "1"; if (ex is DropboxApi.RateLimitException) { try { Thread.Sleep(((DropboxApi.RateLimitException)ex).RetryAfter * 1000); } catch (Exception ex1) { } } Thread.Sleep(1000); } finally { try { if (teamclient != null) { teamclient.Dispose(); teamclient = null; } if (clientadmin != null) { clientadmin.Dispose(); clientadmin = null; } } catch (Exception ex) { } } } } catch (Exception ex) { string isgeneratelog = Global.IsErrorLog == false ? "0" : "1"; if (isgeneratelog == "1") { try { System.IO.File.AppendAllText(errorlog, "\r\nDateTime: " + DateTime.Now.ToString() + "\r\nPathDisplay: " + uploadSession.Item.PathDisplay + "\r\n" + ex.Message + "\r\n" + ex.StackTrace + "\r\n"); } catch (Exception ex1) { } } throw ex; } finally { clientadmin = null; fragmentBuffer = null; teamclient = null; } } public async Task<string> UploadBatchFinish() { DropboxApi.DropboxTeamClient teamclient = null; DropboxApi.DropboxClient clientadmin = null; try { if (!String.IsNullOrEmpty(this.CurrentToken.RefreshToken)) { string appkey = SyncProDropboxAppKey; string appsecret = SyncProDropboxAppSecret; teamclient = new DropboxApi.DropboxTeamClient(this.CurrentToken.RefreshToken, appkey, appsecret); } else { teamclient = new DropboxApi.DropboxTeamClient(this.CurrentToken.AccessToken); } clientadmin = teamclient.AsMember(this.TeamMemberId); UploadSessionFinishBatchArg batchArg = new UploadSessionFinishBatchArg(args); var result = await clientadmin.Files.UploadSessionFinishBatchAsync(batchArg); if (result != null && result.AsAsyncJobId != null && result.AsAsyncJobId.Value != null) { return result.AsAsyncJobId.Value; } } catch (Exception ex) { throw ex; } finally { teamclient = null; clientadmin = null; } return ""; }
Thanks,
Gagan
- Greg-DB3 years agoDropbox Staff
The 'too_many_requests' error indicates that your app is hitting the actual Dropbox rate limiting system. We don't have specific rate numbers documented, nor can we increase the rate limits for any particular, app, user, or team. Apps should be written to handle these rate limit responses automatically, respecting the Retry-After value, and you should try to minimize the number of calls however you can. I recommend referring to the error documentation and Error Handling Guide for more information.
For example, I see you're using UploadSessionStartAsync and you mentioned you're uploading many files in parallel, so it would be better to use UploadSessionStartBatchAsync instead, as that would let you start multiple upload sessions in one call, reducing how many calls you need to make.
About Dropbox API Support & Feedback
Find help with the Dropbox API from other developers.
5,883 PostsLatest Activity: 23 hours agoIf you need more help you can view your support options (expected response time for an email or ticket is 24 hours), or contact us on X or Facebook.
For more info on available support options for your Dropbox plan, see this article.
If you found the answer to your question in this Community thread, please 'like' the post to say thanks and to let us know it was useful!