You might see that the Dropbox Community team have been busy working on some major updates to the Community itself! So, here is some info on what’s changed, what’s staying the same and what you can expect from the Dropbox Community overall.

Forum Discussion

Beatso's avatar
Beatso
Explorer | Level 4
5 years ago

File uploading slow

Hi,

 

I'm using the javscript sdk, and trying to upload multiple files (one after another) with dbx.filesUpload. Due to the nature of my projectm I have to upload multiple files when the user sends a request. Each file is taking around a second, which results in long waiting times for the user when they need, say, 50 files.

 

My code:

async function uploadMultipleFiles (storageFilePaths,packFilePaths,packRoot) {
	try {
		for (i in storageFilePaths) {
			console.log(await dbx.filesUpload({ path: packRoot+packFilePaths[i], contents: fs.readFileSync(storageFilePaths[i])}))
		}
	} catch (err) {
		throw err
	}
}

Full code is here. I have to use await; see this thread. Would there be a quicker way to do this without getting too many requests? Would it be quicker to copy the files from another place in my dropbox rather than upload them like this?

  • Greg-DB's avatar
    Greg-DB
    Icon for Dropbox Staff rankDropbox Staff

    There are a number of variables that will affect overall performance of transfers like this, some of which can't be controlled unfortunately. There are some other options that may be helpful though, depending on the specifics of your scenario.

     

    If the files do already exist in the Dropbox account and you just need to make copies of them, using filesCopyBatchV2 may be faster.

     

    Or, if the files already exist in another Dropbox account, and you have authorization to both of them, you may want to try using filesCopyReferenceGet and filesCopyReferenceSave instead.

     

    Otherwise, if you have a significant number of files to upload, you may want to use the "Batch Upload" method described in the Performance Guide. That way, you can upload the file data for several files at a time in parallel, using filesUploadSessionStart and filesUploadSessionAppendV2 as needed, and then commit them all at once when those are done using filesUploadSessionFinishBatch.

    • Beatso's avatar
      Beatso
      Explorer | Level 4

      Thanks for your help. Using the batch upload method, how to I "group files that are being uploaded into one or more batches"? Also, do I use filesUploadSessionStart for each file, or do I use it once, and append for every other file? Is there anywhere where I could find an example of uploading multiple files with a batch upload?

       

      Edit: I found your example here. My code is as follows:

      var entries = []
      for (i in storageFilePathsToUpload) {
      	let fileData = fs.readFileSync(storageFilePathsToUpload[i])
      	dbx.filesUploadSessionStart({
      		contents: fileData,
      		close: true,
      	})
      	.then(function (response) {
      		entries.push({cursor:{session_id:response.session_id,offset:fileData.length},commit:{path:packPath+packFilePathsToUpload[i]}})
      		console.log("entriesinprog")
      	})
      	.catch(function (err) {
      		console.log(err);
      	});
      }
      
      console.log(entries)
      dbx.filesUploadSessionFinishBatch({entries:entries})
      .then(function(response){
      	dbx.filesUploadSessionFinishBatchCheck({async_job_id: response.async_job_id})
      	.then(function(response){
      		console.log(response)
      	})
      	.catch(function(err){
      		console.log(err)
      	})
      
      	}).catch(err => {
      		console.log(err)
      	})

      It's not working because it's not executing in the right order. The filesUploadSessionFinishBatchCheck is logging its response before the filesUploadSessionStart has finished uploading. Why is this, and how could I only finish the batch once the for loop has finished? (I'm not in an async function). It might sort of relate to this thread, but I obviously can't use .then on a for loop.

      Thanks once again for your help.

       

      Edit 2: I got the above working, and now my code looks like this:

       

      async function startSessions () {
      	entries = []
      	for (let [index,val] of storageFilePathsToUpload.entries()) {
      		let fileData = fs.readFileSync(val)
      		await dbx.filesUploadSessionStart({
      			contents: fileData,
      			close: true,
      		})
      		.then(function (response) {
      			entries.push({cursor:{session_id:response.session_id,offset:fileData.length},commit:{path:packPath+packFilePathsToUpload[index]}})
      		})
      		.catch(function (err) {
      			console.error(err)
      		})
      	}
      }
      
      
      startSessions().then(()=>{
      	console.log(entries)
      	dbx.filesUploadSessionFinishBatch({entries:entries})
      	.then(function(response){
      		dbx.filesUploadSessionFinishBatchCheck({async_job_id: response.async_job_id})
      		.then(function(output){
      			console.log(output) // THIS RETURNS { '.tag': 'in_progress' }
      			// get share link and send it
      			getShareLink("/test").then((response)=>{res.send(response)}).catch((error)=>{res.send("error");console.error(error)})
      		})
      		.catch(function(err){
      			console.error(err)
      		})
      	}).catch(err => {
      		console.error(err)
      	})
      	
      }).catch((err)=>{console.error(err)})

      When the output of filesUploadSessionFinishBatchCheck is logged, it gives { '.tag': 'in_progress' }. Is there a way to not run the code until the batch is finished?