You might see that the Dropbox Community team have been busy working on some major updates to the Community itself! So, here is some info on what’s changed, what’s staying the same and what you can expect from the Dropbox Community overall.
Forum Discussion
Too Many Write Operations when Uploading Files
Hello Greg-DB I am getting too_many_write_operations/ at times while uploading a hierarchy of folders each containing a single file. I am using the UploadSessionStartAsync, UploadSessionAppendV...
gagsbh You should set close=true on the last call to on upload session where you need to upload data for that upload session. That might actually be on a UploadSessionStartAsync call (for small files), or a UploadSessionAppendV2Async call, or a UploadSessionFinishAsync call, if not using UploadSessionFinishBatchAsync. If you are using UploadSessionFinishBatchAsync, then you do need to set close=true on the last UploadSessionStartAsync or UploadSessionAppendV2Async for that upload session.
Every upload session is used to upload one file, and has one upload session ID. So, the cursor for any given upload session should have the upload session ID for that upload session, and an offset that indicates how much has been uploaded so far for that upload session.
And yes, UploadSessionFinishBatchAsync takes an IEnumerable<UploadSessionFinishArg>, which should contain one UploadSessionFinishArg per upload session (each of which needs to be closed) that you wish to finish.
Greg-DB
Dropbox Staff
Dropbox StaffIf you're making multiple changes at the same time in the same account or shared folder, you can run in to this 'too_many_write_operations' error, which is "lock contention". That's not explicit rate limiting, but rather a result of how Dropbox works on the back-end. This is a technical inability to make a modification in the account or shared folder at the time of the API call. This error indicates that there was simultaneous activity in the account or shared/team folder preventing your app from making the state-modifying call (e.g., adding, editing, moving, copying, sharing, or deleting files/folders) it is attempting. The simultaneous activity could be coming from your app itself, or elsewhere, e.g., from the user's desktop client. It can come from the same user, or another member of a shared folder. You can find more information about lock contention here. The app will need to be written to automatically handle this error.
In short, to avoid this error, you should avoid making multiple concurrent state modifications and use batch endpoints where possible, such as UploadSessionFinishBatchAsync. That won't guarantee that you won't run in to this error though, as contention can still come from other sources, so you should also implement error handling and automatic retrying as needed.
Thanks Greg-DB for the information
I will look into my code and let you know if I have any more queries.
Thanks,
Gagan
Hello Greg-DB ,
I reviewed my code and found that multiple concurrent threads were trying to upload files that was causing "too_many_writes" error.
Once I made the uploads serial, the issue got resolved.I would like to make the upload of several files happen is parallel so I like to implement "UploadSessionFinishBatchAsync".
I use the following APIs:
- UploadSessionStartAsync that takes memory stream as parameter:
clientadmin.Files.UploadSessionStartAsync(args, memStream);
- UploadSessionAppendV2Async that also takes memory stream as parameter:
clientadmin.Files.UploadSessionAppendV2Async(cursor, false, memStream);
- UploadSessionFinishAsync that also takes memory stream as parameter:
clientadmin.Files.UploadSessionFinishAsync(cursor, new CommitInfo(Item.PathDisplay), memStream);In case most of the files are small in size and the memory stream is used completely in UploadSessionStartAsync(args, memStream), then what do I pass in memstream parameter in clientadmin.Files.UploadSessionFinishAsync(cursor, new CommitInfo(Item.PathDisplay), memStream);
In other words, in case of single fragment for small files, can I use UploadSessionStartAsync.
If yes, what do I pass for memory stream parameter when I end session with UploadSessionFinishAsync API.If I cannot use UploadSessionStartAsync for single fragment small files then I need to use UploadAsync API to upload small files.
How can I use "UploadSessionFinishBatchAsync" with clientadmin.Files.UploadAsync(new CommitInfo(item.PathDisplay), memStream).I mean how do I prevent "too_many_write" error while uploading multiple files concurrently in case I have to use UploadAsync API.
Thanks,
Gagan
gagsbh wrote:...
I would like to make the upload of several files happen is parallel so I like to implement "UploadSessionFinishBatchAsync".
...
In case most of the files are small in size and the memory stream is used completely in UploadSessionStartAsync(args, memStream), then what do I pass in memstream parameter in clientadmin.Files.UploadSessionFinishAsync(cursor, new CommitInfo(Item.PathDisplay), memStream);
...
Hi gagsbh,
Happy New Year. 🙂
Seems you are mixing different ideas! When you are using UploadSessionFinishAsync, a big file is going to be upload and put in Dropbox namespace at the end. After this call, upload session is closed and can't be used (there is no reason for such use actually); the file upload has finished. For multiple files upload with multiple upload sessions, instead of UploadSessionFinishAsync you can use UploadSessionFinishBatchAsync to finish all sessions at once, not one by one, so contentions appearing becomes less likely.
gagsbh wrote:...
In other words, in case of single fragment for small files, can I use UploadSessionStartAsync.
If yes, what do I pass for memory stream parameter when I end session with UploadSessionFinishAsync API....
Yes, you can use UploadSessionStartAsync. Even more it's mandatory. Do you know other way to start upload session?! 🧐🙂 By the way if you want to use batch mode, as I mentioned above, you haven't to use UploadSessionFinishAsync! It's NOT for use in batch mode!!! UploadSessionFinishBatchAsync UploadSessionFinishAsync are mutually exclusive! 👆
gagsbh wrote:...
If I cannot use UploadSessionStartAsync for single fragment small files then I need to use UploadAsync API to upload small files.
...
No, you can use either UploadSessionStartAsync or UploadAsync for small files (less than 150MB). You can't use UploadAsync for large files, where UploadSessionStartAsync can be used only. Just notice that UploadAsync doesn't start session, so can't be used in batch mode!
gagsbh wrote:...
How can I use "UploadSessionFinishBatchAsync" with clientadmin.Files.UploadAsync(new CommitInfo(item.PathDisplay), memStream)....
You can NOT! As mentioned before, UploadSessionFinishBatchAsync needs set of sessions, but UploadAsync doesn't start any session.
gagsbh wrote:...
I mean how do I prevent "too_many_write" error while uploading multiple files concurrently in case I have to use UploadAsync API.
...
As Greg-DB has mentioned above, you can't prevent entirely contentions ("too_many_write" error). It's matter to decrease appearance probability! You should be still able handle such a exception; take care for such in your code and be ready. Using UploadAsync is definitely not the way.
For you case (multiple small files) start a session with UploadSessionStartAsync for every file and upload it on start together with closing that uploads (use the corresponding flag). After you have all files sessions use them in a single UploadSessionFinishBatchAsync call. 😉 That's it. Instead multiple finishing for every single file, will be just a single one for all files in the set.
Hope this clarifies matter.
About Dropbox API Support & Feedback
Find help with the Dropbox API from other developers.
