You might see that the Dropbox Community team have been busy working on some major updates to the Community itself! So, here is some info on what’s changed, what’s staying the same and what you can expect from the Dropbox Community overall.
Forum Discussion
Custom Tool - Slow File Upload Speed
Hello Greg-DB
We have developed a tool to help our customers move data from Windows File System to Dropbox.
The upload is done in batches of 8 files using the Dropbox APIs - UploadSessio...
Hello Greg-DB
Thank you for the reply !!!
We used the new Dropbox API - UploadSessionStartBatchAsync and this helped in reducing the "too_many_requests" errors.
We tested 2.2 TB data sample containing 427,002 files for migration using our custom app.
Last time we faced 26,000 failures which has been reduced to 1,753 in the latest test.What more can be done to further reduce failures due to "too_many_requests" error response from Dropbox.
We do wait and retry as shown in the last code snippet.Thanks,
Gagan
Hello Greg-DB
We tested with Upload Session Finish list count of Max = 1000 in Upload SessionFinishBatchAsync and we started getting better speeds. We are continuing our testing and will let you know if we need further help.
Thanks,
Gagan
Hello Greg-DB
Upload speeds are not major concern as of now but failures are.
We did a 2 TB migration using our custom tool and encountered "too_many_requests" failure as the end stages of the migration.
Thousands of files reported such failures which won't be acceptable to users.
We upload many files in parallel but using UploadSessionStartAsync, UploadSessionAppendV2Async & UploadSessionFinishBatchAsync APIs.
We do a wait and retry for Rate Failures as recommended by Dropbox and so we thought we should be good.
However, many files still fail due to "too_many_requests".
I am writing the relevant C# code snipped we have implemented for your reference.
Can you please give us some direction to minimize the Rate Failure exceptions and hence reduce upload failures.
public async Task SendUploadFragment(DropboxUploadSession uploadSession, byte[] fragmentBuffer, long offset)
{
string errorlog = ConfigurationManager.AppSettings["ErrorLog"];
int retry = 3;
DropboxApi.DropboxClient clientadmin = null;
DropboxApi.DropboxTeamClient teamclient = null;
try
{
for (int i = 1; i <= retry; i++)
{
try
{
Item data = new Item();
if (!String.IsNullOrEmpty(this.CurrentToken.RefreshToken))
{
string appkey = SyncProDropboxAppKey;
string appsecret = SyncProDropboxAppSecret;
teamclient = new DropboxApi.DropboxTeamClient(this.CurrentToken.RefreshToken, appkey, appsecret);
}
else
{
teamclient = new DropboxApi.DropboxTeamClient(this.CurrentToken.AccessToken);
}
clientadmin = teamclient.AsMember(this.TeamMemberId);
using (var memStream = new MemoryStream(fragmentBuffer, 0, fragmentBuffer.Length))
{
//1. When No Upload session has been created so far
if (String.IsNullOrEmpty(uploadSession.SessionId))
{
try
{
//If multiple fragment session
if (!uploadSession.islastfragment)
{
UploadSessionStartArg args = new UploadSessionStartArg();
UploadSessionType sessiontype = new UploadSessionType();
var result = await clientadmin.Files.UploadSessionStartAsync(false, sessiontype.AsConcurrent, memStream);
uploadSession.SessionId = result.SessionId;
}
else
{
//If single fragment session
UploadSessionType sessiontype = new UploadSessionType();
var result = await clientadmin.Files.UploadSessionStartAsync(true, sessiontype.AsConcurrent, memStream);
uploadSession.SessionId = result.SessionId;
var cursor2 = new UploadSessionCursor(uploadSession.SessionId, (ulong)(offset + fragmentBuffer.Length));
UploadSessionFinishArg arg = new UploadSessionFinishArg(cursor2, new CommitInfo(uploadSession.Item.PathDisplay));
args.Add(arg);
break;
}
}
catch (Exception ex)
{
//Log Exception
if (ex is DropboxApi.RateLimitException)
{
try
{
Thread.Sleep(((DropboxApi.RateLimitException)ex).RetryAfter * 1000);
}
catch (Exception ex1)
{
}
}
Thread.Sleep(1000);
throw ex;
}
}
else
{
var cursor = new UploadSessionCursor(uploadSession.SessionId, (ulong)offset);
//If last fragment of session
if (uploadSession.islastfragment)
{
try
{
await clientadmin.Files.UploadSessionAppendV2Async(cursor, true, memStream);
var cursor2 = new UploadSessionCursor(uploadSession.SessionId, (ulong)(offset + fragmentBuffer.Length));
UploadSessionFinishArg arg = new UploadSessionFinishArg(cursor2, new CommitInfo(uploadSession.Item.PathDisplay));
args.Add(arg);
break;
}
catch (Exception ex)
{
//Log Exception
if (ex is DropboxApi.RateLimitException)
{
try
{
Thread.Sleep(((DropboxApi.RateLimitException)ex).RetryAfter * 1000);
}
catch (Exception ex1)
{
}
}
Thread.Sleep(1000);
}
}
else //If not the last fragment of Session
{
try
{
await clientadmin.Files.UploadSessionAppendV2Async(cursor, false, memStream);
break;
}
catch (Exception ex)
{
//Log Exception
if (ex is DropboxApi.RateLimitException)
{
try
{
Thread.Sleep(((DropboxApi.RateLimitException)ex).RetryAfter * 1000);
}
catch (Exception ex1)
{
}
}
Thread.Sleep(1000);
}
}
}
}
break;
}
catch (Exception ex)
{
string isgeneratelog = Global.IsErrorLog == false ? "0" : "1";
if (ex is DropboxApi.RateLimitException)
{
try
{
Thread.Sleep(((DropboxApi.RateLimitException)ex).RetryAfter * 1000);
}
catch (Exception ex1)
{
}
}
Thread.Sleep(1000);
}
finally
{
try
{
if (teamclient != null)
{
teamclient.Dispose();
teamclient = null;
}
if (clientadmin != null)
{
clientadmin.Dispose();
clientadmin = null;
}
}
catch (Exception ex)
{
}
}
}
}
catch (Exception ex)
{
string isgeneratelog = Global.IsErrorLog == false ? "0" : "1";
if (isgeneratelog == "1")
{
try
{
System.IO.File.AppendAllText(errorlog, "\r\nDateTime: " + DateTime.Now.ToString() + "\r\nPathDisplay: " + uploadSession.Item.PathDisplay +
"\r\n" + ex.Message + "\r\n" + ex.StackTrace + "\r\n");
}
catch (Exception ex1)
{
}
}
throw ex;
}
finally
{
clientadmin = null;
fragmentBuffer = null;
teamclient = null;
}
}
public async Task<string> UploadBatchFinish()
{
DropboxApi.DropboxTeamClient teamclient = null;
DropboxApi.DropboxClient clientadmin = null;
try
{
if (!String.IsNullOrEmpty(this.CurrentToken.RefreshToken))
{
string appkey = SyncProDropboxAppKey;
string appsecret = SyncProDropboxAppSecret;
teamclient = new DropboxApi.DropboxTeamClient(this.CurrentToken.RefreshToken, appkey, appsecret);
}
else
{
teamclient = new DropboxApi.DropboxTeamClient(this.CurrentToken.AccessToken);
}
clientadmin = teamclient.AsMember(this.TeamMemberId);
UploadSessionFinishBatchArg batchArg = new UploadSessionFinishBatchArg(args);
var result = await clientadmin.Files.UploadSessionFinishBatchAsync(batchArg);
if (result != null && result.AsAsyncJobId != null && result.AsAsyncJobId.Value != null)
{
return result.AsAsyncJobId.Value;
}
}
catch (Exception ex)
{
throw ex;
}
finally
{
teamclient = null;
clientadmin = null;
}
return "";
}Thanks,
Gagan
- Greg-DB
Dropbox Staff
The 'too_many_requests' error indicates that your app is hitting the actual Dropbox rate limiting system. We don't have specific rate numbers documented, nor can we increase the rate limits for any particular, app, user, or team. Apps should be written to handle these rate limit responses automatically, respecting the Retry-After value, and you should try to minimize the number of calls however you can. I recommend referring to the error documentation and Error Handling Guide for more information.
For example, I see you're using UploadSessionStartAsync and you mentioned you're uploading many files in parallel, so it would be better to use UploadSessionStartBatchAsync instead, as that would let you start multiple upload sessions in one call, reducing how many calls you need to make.
Thank-you Greg-DB
I will try to incorporate your suggestions in my code logic.
I would use UploadSessionStartBatchAsync to start multiple upload sessions in one call. Can I use it to start max-allowed 1000 upload sessions or is that an ill-advised decision. Do I need to start with lesser upload sessions - say 500 session using one call to UploadSessionStartBatchAsync to avoid Rate Limit issue ?
Also, once I start 'x' number of upload sessions using UploadSessionStartBatchAsync , can I again start another 'y' no. of upload sessions using UploadSessionStartBatchAsync before calling UploadSessionFinishBatchAsync for the first batch of 'x' upload sessions ?
In other words, should I wait for UploadSessionFinishBatchAsync for the first batch of 'x' upload to be completed before I make another call to UploadSessionStartBatchAsync
Thanks,
Gagan
Thanks,
Gagan
- Greg-DB
You can use UploadSessionStartBatchAsync to start up to 1000 upload sessions. If you have at least 1000 upload sessions to start, you can do so in one call; you do not need to lower that. Calling UploadSessionStartBatchAsync once only counts as one call, regardless of what you set numSessions to.
And you can call UploadSessionStartBatchAsync multiple times before calling UploadSessionFinishBatchAsync if needed. You don't need to wait for one batch to finish before starting another. Just keep in mind that you should only finish one batch per namespace at a time to avoid lock contention.
About Dropbox API Support & Feedback
Find help with the Dropbox API from other developers.
