You might see that the Dropbox Community team have been busy working on some major updates to the Community itself! So, here is some info on what’s changed, what’s staying the same and what you can expect from the Dropbox Community overall.
Forum Discussion
[C#] Large files don't show up in user's cloud after finishing batch upload
Hey there!
I'm trying to upload large files with batch upload (as it is suggested by Dropbox: https://www.dropbox.com/developers/reference/data-ingress-guide).
Everything seems to be working exce...
I see that you are checking the 'IsComplete' status of the UploadSessionFinishBatchAsync job using UploadSessionFinishBatchCheckAsync, but that will only tell you if the operation is complete overall. It does not indicate whether any particular commit succeeded or not. You should additionally check each UploadSessionFinishBatchResultEntry (in UploadSessionFinishBatchResult.Entries) to see if each one was a Success or a Failure. Additionally, if it failed, you can see why from the UploadSessionFinishError available in UploadSessionFinishBatchResultEntry.Failure.Value.
By the way, I redacted it from your post, but for the sake of security, you should disable that access token, since you posted it here. You can do so by revoking access to the app entirely, if the access token is for your account, here:
https://www.dropbox.com/account/connected_apps
Or, you can disable just this access token using the API:
HTTP: https://www.dropbox.com/developers/documentation/http/documentation#auth-token-revoke
API Explorer: https://dropbox.github.io/dropbox-api-v2-explorer/#auth_token/revoke
Greg-DB
Dropbox Staff
Dropbox StaffI see that you are checking the 'IsComplete' status of the UploadSessionFinishBatchAsync job using UploadSessionFinishBatchCheckAsync, but that will only tell you if the operation is complete overall. It does not indicate whether any particular commit succeeded or not. You should additionally check each UploadSessionFinishBatchResultEntry (in UploadSessionFinishBatchResult.Entries) to see if each one was a Success or a Failure. Additionally, if it failed, you can see why from the UploadSessionFinishError available in UploadSessionFinishBatchResultEntry.Failure.Value.
By the way, I redacted it from your post, but for the sake of security, you should disable that access token, since you posted it here. You can do so by revoking access to the app entirely, if the access token is for your account, here:
https://www.dropbox.com/account/connected_apps
Or, you can disable just this access token using the API:
HTTP: https://www.dropbox.com/developers/documentation/http/documentation#auth-token-revoke
API Explorer: https://dropbox.github.io/dropbox-api-v2-explorer/#auth_token/revoke
Thank you Greg!
Turned out that it failed with incorrect offset.
The logic how I calculated the last offset (which was passed to the finishing method) was wrong. But after I repaired it, now everything works just fine.
Also, thanks for erasing the access token, I did forgot about it, but it was just the generated access token that I use during development, so not a big deal (I changed it though).
And lastly, I don't know if you could communicate this towards the people at Dropbox who are responsibe for the .NET SDK documentation, but if you could tell them that they should somehow mention at the UploadSessionFinishBatchResult section how to reach it, I'd be grateful. It took me almost an hour to find out how to reach this class, and I only found out because a discussion (about the Dropbox API Java SDK, not even .NET) here in the forum mentioned that one can reach it through the UploadSessionFinishBatchJobStatus object.
Thanks again, for your help!
Andrew
- Greg-DB
Thanks for following up. I'm glad to hear you got this sorted out.
And thanks for the feedback! I'll send this along to the team.
Hi Guys,
I've re-used this code also to upload many files in a batch - thanks for posting the code, it's saved me alot of time.
But as expected, I did reproduce the problem where large files appear to be uploaded, but then do not show up in Dropbox.
In my test...
1. All the small files that are < 20KB are uploaded successfully
2. Three large files 120MB, 177MB, and 537MB appear to upload over many chunks (viewing debug statements in Visual Studio) but do not exist in the Dropbox folder when completedI'm using the default 4MB chunk size to upload, e.g. so the 120MB file took 30 chunks to upload.
Q1.
As Andrew later discovered, he found the calculation of the last offset to be incorrect. I have reviewed the code a few times and it appears logically correct. Since it isn't correct, can someone please state what is wrong with the last offset caclulation? Thanks.
I'm going to review this code again to see the bug.Q2.
On another issue also mentioned here, I'm currently using the UploadSessionFinishBatchCheckAsync()method and .IsComplete property to check for a successful completion. But as recommend by Greg, I'd like to use the UploadSessionFinishBatchResult() method and .IsSuccess property for each entry, but can't work out how to reach this.
Can you please point me in the correct direction or post a couple of lines of code for this? ThanksQ3.
Is it a good idea to sleep for 500ms (or whatever appropriate value) before each UploadSessionFinishBatchCheckAsync() method .IsComplete property call in a loop?
(await Task.Delay(500);)Thanks.
- Greg-DB
Q1. Perhaps Andrewer016 would be so kind as to share their updated code with that fixed.
Q2. Likewise, perhaps Andrewer016 can share this piece as well, if they updated their code to include this.
Very basically though, it would look like this:
foreach (var entry in status.AsComplete.Value.Entries) { if (entry.IsSuccess) { Console.WriteLine(entry.AsSuccess.Value); } else if (entry.IsFailure) { Console.WriteLine(entry.AsFailure.Value); } }You can likewise drill down further into that 'entry.AsFailure.Value' as needed for more specific error information.
Q3. Dropbox doesn't have official guidance or policy on how often you should poll UploadSessionFinishBatchCheckAsync, so that's up to you, but adding a short delay like that does seem reasonable.
Thanks Greg.
After working on this for most of the day, I think I have the solutions. They appear to be working.
1.
The final offset should be at the position of end of the file (i.e. size of the file)
var uploadSessionCursor = new UploadSessionCursor(uploadSessionID, (ulong)fileStream.Length); await client.Files.UploadSessionAppendV2Async(uploadSessionCursor, true, memoryStream);But too apply this final offset, you will need an extra iteration in the loop, so alter the for loop and if statement (in the final chunk) appropriately:
for (ulong indexOfChunks = 0; indexOfChunks <= numberOfChunks; indexOfChunks++) if (indexOfChunks == numberOfChunks)2. The UploadSessionFinishBatchCheckAsync() method actually returns an object type of UploadSessionFinishBatchJobStatus
var uploadSessionFinishBatchJobStatus = await client.Files.UploadSessionFinishBatchCheckAsync(uploadSessionFinishBatch.AsAsyncJobId.Value);And use your aforementioned code.
If you wanted to simply check if all jobs in the batch committed successfully, I coded it in 1 line of code using Linq:
result = uploadSessionFinishBatchJobStatus.AsComplete.Value.Entries.Select(j => j.IsSuccess).Count() == uploadSessionFinishBatchJobStatus.AsComplete.Value.Entries.Count();
About Dropbox API Support & Feedback
Find help with the Dropbox API from other developers.
