You might see that the Dropbox Community team have been busy working on some major updates to the Community itself! So, here is some info on what’s changed, what’s staying the same and what you can expect from the Dropbox Community overall.
Forum Discussion
Too Many Write Operations when Uploading Files
Hello Greg-DB
I am getting too_many_write_operations/ at times while uploading a hierarchy of folders each containing a single file.
I am using the UploadSessionStartAsync, UploadSessionAppendV2Async, UploadSessionFinishAsync and UploadAsync APIs to upload files using fragment buffer.
Sometimes all the files get uploaded correctly but in some test runs some files do not get uploaded due to too_many_write_operations error. How do I prevent "too_many_write_operations/" error ?
Error Message:
PathDisplay: /Sandy_Source/Starting folder/SUB1/SUB2/SUB3/SUB4/TriggeredAlarm_Service_Log.txt
Retry Count: 1
too_many_write_operations/...
at Dropbox.Api.DropboxRequestHandler.<RequestJsonString>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Dropbox.Api.DropboxRequestHandler.<RequestJsonStringWithRetry>d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Dropbox.Api.DropboxRequestHandler.<Dropbox-Api-Stone-ITransport-SendUploadRequestAsync>d__13`3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at SyncPro.Adapters.Dropbox.DropboxClient.<SendUploadFragment>d__49.MoveNext()
Sample Source Code:
public async Task SendUploadFragment(DropboxUploadSession uploadSession, byte[] fragmentBuffer, long offset)
{
for (int i = 1; i <= retry; i++)
{
try
{
teamclient = new DropboxApi.DropboxTeamClient(this.CurrentToken.RefreshToken, appkey, appsecret);
DropboxApi.DropboxClient clientadmin = teamclient.AsMember(this.UserProfile.TeamMemberId);
using (var memStream = new MemoryStream(fragmentBuffer, 0, fragmentBuffer.Length))
{
if (String.IsNullOrEmpty(uploadSession.SessionId))
{
if (!uploadSession.islastfragment)
{
UploadSessionStartArg args = new UploadSessionStartArg();
var result = await clientadmin.Files.UploadSessionStartAsync(args, memStream);
uploadSession.SessionId = result.SessionId;
}
else
{
FileMetadata fileMetadata = null;
try
{
fileMetadata = await clientadmin.Files.UploadAsync(new CommitInfo(uploadSession.Item.PathDisplay), memStream);
break;
}
catch (Exception ex)
{
if (ex is DropboxApi.RateLimitException)
{
Thread.Sleep(((DropboxApi.RateLimitException)ex).RetryAfter * 1000);
}
}
}
}
else
{
var cursor = new UploadSessionCursor(uploadSession.SessionId, (ulong)offset);
if (uploadSession.islastfragment)
{
FileMetadata fileMetadata = null;
try
{
fileMetadata = await clientadmin.Files.UploadSessionFinishAsync(cursor, new CommitInfo(uploadSession.Item.PathDisplay), memStream);
break;
}
catch (Exception ex)
{
if (ex is DropboxApi.RateLimitException)
{
Thread.Sleep(((DropboxApi.RateLimitException)ex).RetryAfter * 1000);
}
}
}
}
else
{
try
{
await clientadmin.Files.UploadSessionAppendV2Async(cursor, false, memStream);
break;
}
catch (Exception ex)
{
if (ex is DropboxApi.RateLimitException)
{
Thread.Sleep(((DropboxApi.RateLimitException)ex).RetryAfter * 1000);
}
}
}
}
}
break;
}
catch (Exception ex)
{
if (ex is DropboxApi.RateLimitException)
{
Thread.Sleep(((DropboxApi.RateLimitException)ex).RetryAfter * 1000);
}
if (i == retry)
{
throw ex;
}
}
}
}
Thanks,
Gagan
gagsbh You should set close=true on the last call to on upload session where you need to upload data for that upload session. That might actually be on a UploadSessionStartAsync call (for small files), or a UploadSessionAppendV2Async call, or a UploadSessionFinishAsync call, if not using UploadSessionFinishBatchAsync. If you are using UploadSessionFinishBatchAsync, then you do need to set close=true on the last UploadSessionStartAsync or UploadSessionAppendV2Async for that upload session.
Every upload session is used to upload one file, and has one upload session ID. So, the cursor for any given upload session should have the upload session ID for that upload session, and an offset that indicates how much has been uploaded so far for that upload session.
And yes, UploadSessionFinishBatchAsync takes an IEnumerable<UploadSessionFinishArg>, which should contain one UploadSessionFinishArg per upload session (each of which needs to be closed) that you wish to finish.
- Greg-DB
Dropbox Staff
If you're making multiple changes at the same time in the same account or shared folder, you can run in to this 'too_many_write_operations' error, which is "lock contention". That's not explicit rate limiting, but rather a result of how Dropbox works on the back-end. This is a technical inability to make a modification in the account or shared folder at the time of the API call. This error indicates that there was simultaneous activity in the account or shared/team folder preventing your app from making the state-modifying call (e.g., adding, editing, moving, copying, sharing, or deleting files/folders) it is attempting. The simultaneous activity could be coming from your app itself, or elsewhere, e.g., from the user's desktop client. It can come from the same user, or another member of a shared folder. You can find more information about lock contention here. The app will need to be written to automatically handle this error.
In short, to avoid this error, you should avoid making multiple concurrent state modifications and use batch endpoints where possible, such as UploadSessionFinishBatchAsync. That won't guarantee that you won't run in to this error though, as contention can still come from other sources, so you should also implement error handling and automatic retrying as needed.Hello Greg-DB ,
I reviewed my code and found that multiple concurrent threads were trying to upload files that was causing "too_many_writes" error.
Once I made the uploads serial, the issue got resolved.I would like to make the upload of several files happen is parallel so I like to implement "UploadSessionFinishBatchAsync".
I use the following APIs:
- UploadSessionStartAsync that takes memory stream as parameter:
clientadmin.Files.UploadSessionStartAsync(args, memStream);
- UploadSessionAppendV2Async that also takes memory stream as parameter:
clientadmin.Files.UploadSessionAppendV2Async(cursor, false, memStream);
- UploadSessionFinishAsync that also takes memory stream as parameter:
clientadmin.Files.UploadSessionFinishAsync(cursor, new CommitInfo(Item.PathDisplay), memStream);In case most of the files are small in size and the memory stream is used completely in UploadSessionStartAsync(args, memStream), then what do I pass in memstream parameter in clientadmin.Files.UploadSessionFinishAsync(cursor, new CommitInfo(Item.PathDisplay), memStream);
In other words, in case of single fragment for small files, can I use UploadSessionStartAsync.
If yes, what do I pass for memory stream parameter when I end session with UploadSessionFinishAsync API.If I cannot use UploadSessionStartAsync for single fragment small files then I need to use UploadAsync API to upload small files.
How can I use "UploadSessionFinishBatchAsync" with clientadmin.Files.UploadAsync(new CommitInfo(item.PathDisplay), memStream).I mean how do I prevent "too_many_write" error while uploading multiple files concurrently in case I have to use UploadAsync API.
Thanks,
Gagan
About Dropbox API Support & Feedback
Find help with the Dropbox API from other developers.
