You might see that the Dropbox Community team have been busy working on some major updates to the Community itself! So, here is some info on what’s changed, what’s staying the same and what you can expect from the Dropbox Community overall.
Forum Discussion
Engg
11 months agoExplorer | Level 3
System.Threading.Tasks.TaskCanceledException: A task was canceled.
We have 500 machine which upload file at same time. We are getting RateLimitException. We are doing a retry after waiting rateLimitException.ErrorResponse.RetryAfter seconds. to test this. I came ...
- 11 months ago
In general modifying the folder should not affect probability for exception appearing, but cannot prevent it. Some decrement may happen if different files/folders appear in different namespaces (namespaces are locked independently, so less likely compete), but better don't rely on.
Again, try to schedule the uploads in non overlap way and handle the possible exception in proper way. Example of such exception handling (again just example and far from optimal) follows:
private readonly int retryLimit = 20; ... var backoff = 0; var rand = new Random(); var commitInfo = new CommitInfo(filePath.Substring(filePath.LastIndexOf('/')), WriteMode.Overwrite.Instance, false); for (var i = 0; i < retryLimit && !string.IsNullOrEmpty(sessionId); ++i) { if (backoff != 0) { Console.WriteLine("Ограничение при завършване на файл {0}. Нов опит след {1} милисекунди...", commitInfo.Path, backoff); Thread.Sleep(backoff); } lastCursor = new UploadSessionCursor(sessionId, (ulong)fileStream.Length); memStream = new MemoryStream(buffer, 0, 0); try { await dbx.Files.UploadSessionFinishAsync(lastCursor, commitInfo, body: memStream); sessionId = string.Empty; } catch (RateLimitException rateLimit) { backoff = (int)(backoff * 1.1); backoff = Math.Max(backoff, rateLimit.RetryAfter * 1000); backoff = Math.Max(backoff, 1000); backoff += rand.Next(500); } } if (string.IsNullOrEmpty(sessionId)) { Console.WriteLine("Качването на {0} приключи \ud83d\ude09.", commitInfo.Path); } else { Console.WriteLine("Неуспешно завършване на файл {0} след {1} опита \ud83d\ude15!", commitInfo.Path, retryLimit); }
The above just shows how my advices (from previous page) can be implemented. Here sessionId represents a session ready for finishing (i.e. content uploaded already).
Здравко
Legendary | Level 20
Hm..🤔 I thought I was clear, but it seems not enough.
Engg wrote:...
List<Task> tasks = new List<Task>();foreach (var item in dbLocalFilesPath.Select((value, i) => new { i, value }))
{string dbLocalPath = item.value;
int index = item.i;
Logger.Instance.LogInfo($"dbLocalPath: {dbLocalPath}");
tasks.Add(Task.Run(async () =>...
As can be seen here you are starting multiple uploads at the same time (all files at the particular machine). Right? All right up to here. Let see what's next:
Engg wrote:...
FileMetadata fileMetadata = await files.UploadAsync(path, clientModified: clientModified, body: body);...
await client.Files.UploadSessionFinishAsync(cursor: cursor, commit: new CommitInfo(remotePath), body: memStream);
...
In all cases, for any file size, you are using uploads that directly try modify Dropbox account folder tree!!! Dropbox doesn't let this to happen! Every change in the folder tree is strictly serialized and if many waiting query appear, some of them receive rate limit error (read this limit as a limiting change rate, not exactly the upload rate). That's where your issue is coming from. Is it more clear now? What if you upload using sessions regardless file size and instead of using finish at the every file end (something that triggers the change) you use batch version of the finish (just one change - here I assume you have no more than 1000 files per machine that need upload)? 🧐 Where will limiting error happen almost sure and where wouldn't? 😉
PS: In addition you can en-schedule your machines uploads in a slightly different time - let's say 10 seconds time steps one after other.
Engg
11 months agoExplorer | Level 3
One machine will have only one file which may be more than 1 GB. Only issue is all the 500 machine is getting uploaded at same time and each machine will try to modify Dropbox account folder tree. In this case when I should do UploadSessionFinishAsync.
Any way to handle this?
- Здравко11 months agoLegendary | Level 20
Engg wrote:One machine will have only one file which may be more than 1 GB. ...
If per machine is a single file only to upload, you don't need loop. Why at all are you calling multiple parallel uploads in such a case? File size doesn't matter when upload and finishing are separate. Upload itself is not limited as I said already. Just when call UploadSessionFinishAsync try reorder your code so everything is already uploaded - no content in its POST query - faster processing.
Engg wrote:... Only issue is all the 500 machine is getting uploaded at same time and each machine will try to modify Dropbox account folder tree. ...
Hm..🤔 There is no a single way to handle this (neither best). A step in this direction, as I said, is to reschedule upload so the they won't overlap (so less likely to compete to each other). You should continue handle the rate limiting error, but don't need to re-upload the file anew. Once the file is uploaded, you can try finish it as long and many times as needed (i.e. repeat finishing only) in a loop with proper backoff.
Good luck.
- Engg11 months agoExplorer | Level 3
If per machine is a single file only to upload, you don't need loop. Why at all are you calling multiple parallel uploads in such a case?
Just for the testing I created a console app. To replicate the 500 machine test
You should continue handle the rate limiting error, but don't need to re-upload the file anew.
What you mean by this? did you have any code for this?- Здравко11 months agoLegendary | Level 20
To be honest I'm not sure what confuses you. You perform already upload finishing. Right? If you don't need to add some content there, the only needed is session identifier (that represents the uploaded file but not finished yet). If you don't finish the uploaded file successfully in first try, you can try next time (after proper backoff that should grow up continuously and it's a good idea to add some random component) - the session id stays valid for week.
About Dropbox API Support & Feedback
Find help with the Dropbox API from other developers.
5,882 PostsLatest Activity: 3 years agoIf you need more help you can view your support options (expected response time for an email or ticket is 24 hours), or contact us on X or Facebook.
For more info on available support options for your Dropbox plan, see this article.
If you found the answer to your question in this Community thread, please 'like' the post to say thanks and to let us know it was useful!