You might see that the Dropbox Community team have been busy working on some major updates to the Community itself! So, here is some info on what’s changed, what’s staying the same and what you can expect from the Dropbox Community overall.

Forum Discussion

michal_k's avatar
michal_k
Explorer | Level 4
9 years ago

[Java SDK V2] How to do batch upload?

Hello,

 

I am trying to make Batch upload work by using JAVA SDK v2. I read this documentation and tried to perform a batch upload.

 

There is a general algorithm, which should use direct HTTP calls.

 

1. First point sais "Group files that are being uploaded to into one or more batches....". I don't know, what exactly is Group to batches, I assume that it is some logical splitting.

 

2. Then there is this instruction "For each file in a batch, in parallel, call /files/upload_session/start"

So I tried this code:

 

UploadSessionStartUploader uploadSessionStartUploader = dbxClientV2.files().uploadSessionStart();

and I got an instance of the UploadSessionStartUploader.

 

But now I don't know, how to use it, because it does not seem to provide any functionality for batch upload.

 

Could you please give me some example, how to do the batch upload?

 

 

 

  • Greg-DB's avatar
    Greg-DB
    9 years ago

    That looks like a bug that was fixed in version 2.1.2. Please make sure you're using the latest version of the SDK, and let me know if you're still seeing that.

  • Greg-DB's avatar
    Greg-DB
    Icon for Dropbox Staff rankDropbox Staff

    1. Yes, regarding the grouping, the data ingress guide elaborates:

     

    "Group files that are being uploaded to into one or more batches. A batch can contain up to 1,000 entries. Entries in a batch don’t all have to target the same namespace, but the fewer namespaces involved in a batch, the more efficient committing the batch will be."

     

    So, you can group files into batches of up to 1,000 files, preferably across as few namespaces as possible.

     

    2. There's a sample of using upload sessions with the Java SDK that should be useful here:

     

    https://github.com/dropbox/dropbox-sdk-java/blob/master/examples/upload-file/src/main/java/com/dropbox/core/examples/upload_file/Main.java#L68

     

    That doesn't use the batch functionality, but it should be a good starting point.

     

    The batch functionality is that instead of calling uploadSessionFinish oncer per file, you can instead call uploadSessionFinishBatch once for up to 1000 files each time. You would then use uploadSessionFinishBatchCheck to check on the status of that operation.

    • michal_k's avatar
      michal_k
      Explorer | Level 4

      I modified the sample into an Unit Test, did the changes you advised, but now I keep getting this error as a result of uploadSessionFinishBatchCheck.

       

       uploadSessionFinishBatchJobStatus={
      ".tag" : "complete",
      "entries" : [ {
      ".tag" : "failure",
      "failure" : {
      ".tag" : "lookup_failed",
      "lookup_failed" : "not_closed"
      }
      } ]
      }

       

      Here is the code, sorry for the messy state:

       

          @Test
          public void chunkedUploadFiles() throws Exception {
              // Adjust the chunk size based on your network speed and reliability. Larger chunk sizes will
              // result in fewer network requests, which will be faster. But if an error occurs, the entire
              // chunk will be lost and have to be re-uploaded. Use a multiple of 4MiB for your chunk size.
      //        final long CHUNKED_UPLOAD_CHUNK_SIZE = 8L << 20; // 8MiB
              final long CHUNKED_UPLOAD_CHUNK_SIZE = 15; //in Bytes
              final int CHUNKED_UPLOAD_MAX_ATTEMPTS = 5;
      
              final String FILE_UPLOADED_BY_PARTS = "FileUploadedByParts.txt";
              String dropboxPath = getDropboxPath(FILE_UPLOADED_BY_PARTS);
              File localFile = createTempFileWithContent(FILE_UPLOADED_BY_PARTS, "The content of the file uploaded by parts 01"); //44B of text
      
      
              long size = localFile.length();
      
              long uploaded = 0L;
              DbxException thrown = null;
      
              // Chunked uploads have 3 phases, each of which can accept uploaded bytes:
              //
              //    (1)  Start: initiate the upload and get an upload session ID
              //    (2) Append: upload chunks of the file to append to our session
              //    (3) Finish: commit the upload and close the session
              //
              // We track how many bytes we uploaded to determine which phase we should be in.
              String sessionId = null;
              for (int i = 0; i < CHUNKED_UPLOAD_MAX_ATTEMPTS; ++i) {
                  if (i > 0) {
                      System.out.printf("Retrying chunked upload (%d / %d attempts)\n", i + 1, CHUNKED_UPLOAD_MAX_ATTEMPTS);
                  }
      
                  try (InputStream in = new FileInputStream(localFile)) {
                      // if this is a retry, make sure seek to the correct offset
                      in.skip(uploaded);
      
                      // (1) Start
                      if (sessionId == null) {
                          sessionId = tested.files().uploadSessionStart()
                                  .uploadAndFinish(in, CHUNKED_UPLOAD_CHUNK_SIZE)
                                  .getSessionId();
                          uploaded += CHUNKED_UPLOAD_CHUNK_SIZE;
                          printProgress(uploaded, size);
                      }
      
                      UploadSessionCursor cursor = new UploadSessionCursor(sessionId, uploaded);
      
                      // (2) Append
                      while ((size - uploaded) > CHUNKED_UPLOAD_CHUNK_SIZE) {
                          tested.files().uploadSessionAppendV2(cursor)
                                  .uploadAndFinish(in, CHUNKED_UPLOAD_CHUNK_SIZE);
                          uploaded += CHUNKED_UPLOAD_CHUNK_SIZE;
                          printProgress(uploaded, size);
                          cursor = new UploadSessionCursor(sessionId, uploaded);
                      }
      
                      final long remaining = size - uploaded;
                      if ((remaining) > 0) {
                          tested.files().uploadSessionAppendV2(cursor)
                                  .uploadAndFinish(in, remaining);
                          uploaded += remaining;
                          printProgress(uploaded, size);
                          cursor = new UploadSessionCursor(sessionId, uploaded);
                      }
      
      
                      // (3) Finish
                      CommitInfo commitInfo = CommitInfo.newBuilder(dropboxPath)
                              .withMode(WriteMode.ADD)
                              .withClientModified(new Date(localFile.lastModified()))
                              .build();
                      UploadSessionFinishArg entry = new UploadSessionFinishArg(cursor, commitInfo);
                      final String asyncJobIdValue = tested.files().uploadSessionFinishBatch(newArrayList(entry)).getAsyncJobIdValue();
      
                      while (tested.files().uploadSessionFinishBatchCheck(asyncJobIdValue).isInProgress()) {
                          Thread.sleep(1000);
                      }
      
                      final UploadSessionFinishBatchJobStatus uploadSessionFinishBatchJobStatus = tested.files().uploadSessionFinishBatchCheck(
                              asyncJobIdValue);
                      System.out.println("uploadSessionFinishBatchJobStatus=" + uploadSessionFinishBatchJobStatus.toStringMultiline());
      
                      return;
                  } catch (RetryException ex) {
                      thrown = ex;
                      // RetryExceptions are never automatically retried by the client for uploads. Must
                      // catch this exception even if DbxRequestConfig.getMaxRetries() > 0.
                      sleepQuietly(ex.getBackoffMillis());
                      continue;
                  } catch (NetworkIOException ex) {
                      thrown = ex;
                      // network issue with Dropbox (maybe a timeout?) try again
                      continue;
                  } catch (UploadSessionLookupErrorException ex) {
                      if (ex.errorValue.isIncorrectOffset()) {
                          thrown = ex;
                          // server offset into the stream doesn't match our offset (uploaded). Seek to
                          // the expected offset according to the server and try again.
                          uploaded = ex.errorValue
                                  .getIncorrectOffsetValue()
                                  .getCorrectOffset();
                          continue;
                      } else {
                          // Some other error occurred, give up.
                          System.err.println("Error uploading to Dropbox: " + ex.getMessage());
                          System.exit(1);
                          return;
                      }
                  } catch (UploadSessionFinishErrorException ex) {
                      if (ex.errorValue.isLookupFailed() && ex.errorValue.getLookupFailedValue().isIncorrectOffset()) {
                          thrown = ex;
                          // server offset into the stream doesn't match our offset (uploaded). Seek to
                          // the expected offset according to the server and try again.
                          uploaded = ex.errorValue
                                  .getLookupFailedValue()
                                  .getIncorrectOffsetValue()
                                  .getCorrectOffset();
                          continue;
                      } else {
                          // some other error occurred, give up.
                          System.err.println("Error uploading to Dropbox: " + ex.getMessage());
                          System.exit(1);
                          return;
                      }
                  } catch (DbxException ex) {
                      System.err.println("Error uploading to Dropbox: " + ex.getMessage());
                      System.exit(1);
                      return;
                  } catch (IOException ex) {
                      System.err.println("Error reading from file \"" + localFile + "\": " + ex.getMessage());
                      System.exit(1);
                      return;
                  }
              }
      
              // if we made it here, then we must have run out of attempts
              System.err.println("Maxed out upload attempts to Dropbox. Most recent error: " + thrown.getMessage());
              System.exit(1);
          }
      
          private static void printProgress(long uploaded, long size) {
              System.out.printf("Uploaded %12d / %12d bytes (%5.2f%%)\n", uploaded, size, 100 * (uploaded / (double) size));
          }
      
          private static void sleepQuietly(long millis) {
              try {
                  Thread.sleep(millis);
              } catch (InterruptedException ex) {
                  // just exit
                  System.err.println("Error uploading to Dropbox: interrupted during backoff.");
                  System.exit(1);
              }
          }
      • michal_k's avatar
        michal_k
        Explorer | Level 4

        I checked the Javadoc and I think, that the problem was in this statement:

         

         

        tested.files().uploadSessionAppendV2(cursor).uploadAndFinish(in, remaining);

        There was the close argument missing, so I think, that correct statement should be this:

         

         

        tested.files().uploadSessionAppendV2(cursor, true).uploadAndFinish(in, remaining);

         

        The file is now uploaded to Dropbox, however I have a problem with following statement:

         

        while (tested.files().uploadSessionFinishBatchCheck(asyncJobIdValue).isInProgress()) {
           Thread.sleep(1000);
        }

         

         

        I get this exception:

         

        java.lang.NoSuchMethodError: com.fasterxml.jackson.core.JsonParseException.<init>(Lcom/fasterxml/jackson/core/JsonParser;Ljava/lang/String;)V
        	at com.dropbox.core.stone.StoneSerializer.expectField(StoneSerializer.java:84)
        	at com.dropbox.core.v2.files.UploadSessionFinishBatchResultEntry$Serializer.deserialize(UploadSessionFinishBatchResultEntry.java:253)
        	at com.dropbox.core.v2.files.UploadSessionFinishBatchResultEntry$Serializer.deserialize(UploadSessionFinishBatchResultEntry.java:205)
        	at com.dropbox.core.stone.StoneSerializers$ListSerializer.deserialize(StoneSerializers.java:265)
        	at com.dropbox.core.stone.StoneSerializers$ListSerializer.deserialize(StoneSerializers.java:244)
        	at com.dropbox.core.v2.files.UploadSessionFinishBatchResult$Serializer.deserialize(UploadSessionFinishBatchResult.java:124)
        	at com.dropbox.core.v2.files.UploadSessionFinishBatchJobStatus$Serializer.deserialize(UploadSessionFinishBatchJobStatus.java:236)
        	at com.dropbox.core.v2.files.UploadSessionFinishBatchJobStatus$Serializer.deserialize(UploadSessionFinishBatchJobStatus.java:190)
        	at com.dropbox.core.stone.StoneSerializer.deserialize(StoneSerializer.java:66)
        	at com.dropbox.core.v2.DbxRawClientV2$1.execute(DbxRawClientV2.java:104)
        	at com.dropbox.core.v2.DbxRawClientV2.executeRetriable(DbxRawClientV2.java:256)
        	at com.dropbox.core.v2.DbxRawClientV2.rpcStyle(DbxRawClientV2.java:97)
        	at com.dropbox.core.v2.files.DbxUserFilesRequests.uploadSessionFinishBatchCheck(DbxUserFilesRequests.java:1565)
        	at com.dropbox.core.v2.files.DbxUserFilesRequests.uploadSessionFinishBatchCheck(DbxUserFilesRequests.java:1592)
        	...nothing interesting here

         

        When I put breakpoint on the line StoneSerializer.java:84, I found out, that there are two problems, probably bugs.

         

        First is, that StoneSerializer is not able to parse the response and second problem is the usage of new JsonParseException, where first argument should be message, not an instance of the JsonParser.

         

        The actual exception message, that is suppressed by this incorrect usage in my case should be:

        expected field 'success', but was: 'name'

         

  • michal_k's avatar
    michal_k
    Explorer | Level 4

    For some reason I can't see my last reply here, so I'll write it again.

     

    I noticed in the Javadoc, that I am probably using the functionality wrong and there is a problem in following statement:

     

    tested.files().uploadSessionAppendV2(cursor).uploadAndFinish(in, remaining);

    It should be used with close argument true like this:

     

     

    tested.files().uploadSessionAppendV2(cursor, true).uploadAndFinish(in, remaining);

     

     

    Now the file is being uploaded to Dropbox well, however following exception is raised from following statement:

     

    while (tested.files().uploadSessionFinishBatchCheck(asyncJobIdValue).isInProgress()) {
        Thread.sleep(1000);
    }

     

     

     

    java.lang.NoSuchMethodError: com.fasterxml.jackson.core.JsonParseException.<init>(Lcom/fasterxml/jackson/core/JsonParser;Ljava/lang/String;)V
    	at com.dropbox.core.stone.StoneSerializer.expectField(StoneSerializer.java:84)
    	at com.dropbox.core.v2.files.UploadSessionFinishBatchResultEntry$Serializer.deserialize(UploadSessionFinishBatchResultEntry.java:253)
    	at com.dropbox.core.v2.files.UploadSessionFinishBatchResultEntry$Serializer.deserialize(UploadSessionFinishBatchResultEntry.java:205)
    	at com.dropbox.core.stone.StoneSerializers$ListSerializer.deserialize(StoneSerializers.java:265)
    	at com.dropbox.core.stone.StoneSerializers$ListSerializer.deserialize(StoneSerializers.java:244)
    	at com.dropbox.core.v2.files.UploadSessionFinishBatchResult$Serializer.deserialize(UploadSessionFinishBatchResult.java:124)
    	at com.dropbox.core.v2.files.UploadSessionFinishBatchJobStatus$Serializer.deserialize(UploadSessionFinishBatchJobStatus.java:236)
    	at com.dropbox.core.v2.files.UploadSessionFinishBatchJobStatus$Serializer.deserialize(UploadSessionFinishBatchJobStatus.java:190)
    	at com.dropbox.core.stone.StoneSerializer.deserialize(StoneSerializer.java:66)
    	at com.dropbox.core.v2.DbxRawClientV2$1.execute(DbxRawClientV2.java:104)
    	at com.dropbox.core.v2.DbxRawClientV2.executeRetriable(DbxRawClientV2.java:256)
    	at com.dropbox.core.v2.DbxRawClientV2.rpcStyle(DbxRawClientV2.java:97)
    	at com.dropbox.core.v2.files.DbxUserFilesRequests.uploadSessionFinishBatchCheck(DbxUserFilesRequests.java:1565)
    	at com.dropbox.core.v2.files.DbxUserFilesRequests.uploadSessionFinishBatchCheck(DbxUserFilesRequests.java:1592)
    ... nothing interesting here

     

     

    I think there are acually two problems. First of according to my debuger the new JsonParseException should be used in a completely different way, first argument should be an error message. I wonder, how the code is able to compile, it's probably some jackson library missmatch.

     

    Nevertheless the main problem is, that the StoneSerializer.java is not able to parse the message and throws exception where the intended error message should be this:

     

    expected field 'success', but was: 'name'

     

     

     

About Dropbox API Support & Feedback

Node avatar for Dropbox API Support & Feedback

Find help with the Dropbox API from other developers.

5,912 PostsLatest Activity: 9 hours ago
333 Following

If you need more help you can view your support options (expected response time for an email or ticket is 24 hours), or contact us on X or Facebook.

For more info on available support options for your Dropbox plan, see this article.

If you found the answer to your question in this Community thread, please 'like' the post to say thanks and to let us know it was useful!