We are aware of the issue with the badge emails resending to everyone, we apologise for the inconvenience - learn more here.

Forum Discussion

0ylanmorisson's avatar
0ylanmorisson
Explorer | Level 3
8 years ago

Issues about Chunkupload

I'm writing a very simple console uploading program using most of the sample code from Dropbox.net Api. But I have two issues now.

 

When the network drops and reconnects, I get an error saying "lookup_failed/incorrect_offset/". Where I go wrong?

 

Another related question is,  say I have a 5120kb file to upload and the chunk is 1024kb. So when the third chunk is uploading,  suppose the network suddently drops after uploading 500kb, then where should I set the offset to continue uploading, from offset=3072 or from offset=3072+500= 3572?

 

Here is my ChunkUpload code:

 

       private async Task ChunkUpload(DropboxClient client, string srcfile, string dstfile)
        {
            if (File.Exists(srcfile))
            {
                Console.WriteLine("\nUploading file {0} to Dropbox path: {1}\n", srcfile, dstfile);
                // Chunk size is 1024KB.
                const int chunkSize = 1024 * 1024;

                FileStream fs = new FileStream(srcfile, FileMode.Open);
                byte[] data = new byte[fs.Length];
                fs.Read(data, 0, data.Length);
                fs.Close();

                using (var stream = new MemoryStream(data))
                {
                    int numChunks = (int)Math.Ceiling((double)stream.Length / chunkSize);

                    byte[] buffer = new byte[chunkSize];
                    string sessionId = null;

                    var idx = 0;
                   do
                    {
                        var byteRead = stream.Read(buffer, 0, chunkSize);
                        var percentage = 0;
                        int row = Console.CursorTop - 1;  //Set cursor position
                        Console.SetCursorPosition(0, row);
                        if ( (idx+1) == numChunks)  //Last chunk, 99% done!
                        {
                            percentage =  99;
                            Console.WriteLine("Uploaded {0}: {1}% of total {2}kb", idx, percentage, (int)Math.Ceiling((double)stream.Length));
                        }
                        else
                        {
                            percentage = 100 * (idx + 1) * chunkSize / (int)Math.Ceiling((double)stream.Length);
                            Console.WriteLine("Uploaded {0}: {1}% of total {2}kb", idx, percentage, (int)Math.Ceiling((double)stream.Length));
                        }
                        try
                        {
                            using (MemoryStream memStream = new MemoryStream(buffer, 0, byteRead))
                            {
                                if (idx == 0)
                                {
                                    var result = await client.Files.UploadSessionStartAsync(body: memStream);
                                    sessionId = result.SessionId;
                                }

                                else
                                {
                                    UploadSessionCursor cursor = new UploadSessionCursor(sessionId, (ulong)(chunkSize * idx));

                                    if (idx == numChunks - 1)
                                    {
                                        await client.Files.UploadSessionFinishAsync(cursor, new CommitInfo(dstfile), memStream);
                                        row = Console.CursorTop - 1;
                                        Console.SetCursorPosition(0, row);
                                        Console.WriteLine("Uploaded {0}: {1}% of total {2}kb", idx, 100, (int)Math.Ceiling((double)stream.Length));
                                    }
                                    else
                                    {
                                        await client.Files.UploadSessionAppendV2Async(cursor, body: memStream);
                                    }
                                }
                            }
                            idx++;
                        }
                        catch (Exception e)
                        {
                            Console.WriteLine("Error uploading:{0}", e.Message);
                            Console.WriteLine("SessionID: {0}", sessionId);
                            Console.WriteLine("\nMake sure everything is OK then press enter. \n");
                            Console.ReadLine();
                        }
                    } while (idx < numChunks);
                }
                Console.WriteLine("Upload OK! Press enter to exit...");
                Console.ReadLine();
            }
            else
            {
                Console.WriteLine("Wrong path:{0}", srcfile);
            }
        }
  • The documentation for UploadSessionLookupError.IncorrectOffset covers an example of how this can occur:

     

    "The specified offset was incorrect. See the value for the correct offset. This error may occur when a previous request was received and processed successfully but the client did not receive the response, e.g. due to a network error."

     

    To get the correct offset, you can catch and inspect the exception like this:

    try {
        await client.Files.UploadSessionAppendV2Async(cursor, body: memStream);
    } catch (ApiException<UploadSessionLookupError> e) {
    
        if (e.ErrorResponse.IsIncorrectOffset) {
            Console.WriteLine("Expected offset: " + e.ErrorResponse.AsIncorrectOffset.Value.CorrectOffset);
        }
        // todo: other error handling
        throw;
    }

    In your 5120kb file scenario, I believe the offset would still be 3072 at that point, since the connection failed. However that may depend on the exact nature of the failure, so you should always check the returned value anyway.

     

     

     

  • Greg-DB's avatar
    Greg-DB
    Icon for Dropbox Staff rankDropbox Staff

    The documentation for UploadSessionLookupError.IncorrectOffset covers an example of how this can occur:

     

    "The specified offset was incorrect. See the value for the correct offset. This error may occur when a previous request was received and processed successfully but the client did not receive the response, e.g. due to a network error."

     

    To get the correct offset, you can catch and inspect the exception like this:

    try {
        await client.Files.UploadSessionAppendV2Async(cursor, body: memStream);
    } catch (ApiException<UploadSessionLookupError> e) {
    
        if (e.ErrorResponse.IsIncorrectOffset) {
            Console.WriteLine("Expected offset: " + e.ErrorResponse.AsIncorrectOffset.Value.CorrectOffset);
        }
        // todo: other error handling
        throw;
    }

    In your 5120kb file scenario, I believe the offset would still be 3072 at that point, since the connection failed. However that may depend on the exact nature of the failure, so you should always check the returned value anyway.

     

     

     

    • 0ylanmorisson's avatar
      0ylanmorisson
      Explorer | Level 3

      I'm getting another related problem here. If the program accendentally exits, then how can I resume upload when it restarts?

       

      I tried a solution which is,

      1. record the sessionID and cursor.offset to a txt after each chunk upload;

      2. every time the program starts, it will check if "record.txt" exists. If txt exists, then it means this is a resume-upload. The program should upload from last offset.

       

      But here is the problem, how can I get the right MemoryStream from last offset position?

      The following code 

       

       

                          if (File.Exists("record.txt"))
                          {
                              foreach (string line in File.ReadLines("record.txt"))
                              {
                                  if (line.Contains("AAAA"))
                                  {
                                      sessionId = line;
                                  }
                                  else
                                  {
                                      currentPosition = long.Parse(line);
                                  }
                              }
                              File.Delete("record.txt");
                          }
      
                          if (isResume)
                          {
                              byteRead = stream.Read(buffer, 0, chunkSize);   //this line doesn't work.
                          }
      
      
      • Greg-DB's avatar
        Greg-DB
        Icon for Dropbox Staff rankDropbox Staff
        It sounds like you already have the right idea, in that you should store upload session ID and resume the upload from where you left off. I don't recommend relying on the session ID having "AAAA" though, as that's not guaranteed. You should store it in some format you can rely on, and use whatever session ID string the API gave you.

        As far as how you interact with the local file, I'm afraid that's a bit outside the scope of Dropbox API support, as that has to do with the local file system and not Dropbox. You'll probably want to refer to the documentation for FileStream and MemoryStream to determine how to start reading from the file at a particular offset in the file.

About Dropbox API Support & Feedback

Node avatar for Dropbox API Support & Feedback

Find help with the Dropbox API from other developers.

5,877 PostsLatest Activity: 12 months ago
326 Following

If you need more help you can view your support options (expected response time for an email or ticket is 24 hours), or contact us on X or Facebook.

For more info on available support options for your Dropbox plan, see this article.

If you found the answer to your question in this Community thread, please 'like' the post to say thanks and to let us know it was useful!