API rate limit error

[MapTiler]

Hi all,

I'm using SwiftyDropbox in a Swift 3 app. I managed to get authentication working, I can get the details of the authenticated user, I can upload files, I can download files, I can create folders ... all is fine.

My app allows users to download map tiles. A map tile is a 256x256 image, and there is a folder structure containing zoom level, x and y coordinates, for example 1/1/1.png.

The app downloades these files from the web to local storage. The user can then choose to share these files to an external storage, like iCloud Drive, Dropbox, etc.

I get the odd API rate limit error when uploading these files to Dropbox.

I've tried re-uploading the failed files using the retry_after argument when the error occurs but I find there's no guarantee that the file will actually be uploaded that way.

I've also tried to implement a delay between the upload of every tile, but even a delay of 5 seconds between every upload causes API rate limit errors.

So I'm stuck in a situation where I have (for example) 100 files in various folders in local storage, and about 96 or so make it to Dropbox.

I'm slowly running out of ideas here. I find it very strange that a service like Dropbox produces API rate limit errors. This is 2017 and I'm trying to copy files to Dropbox, not launch a shuttle to Mars.

The only option left I can see is to make a zip bundle in the app's local storage, then upload that zip file to Dropbox. At least the upload is then limited to the one file.

I'd happily share code, but there's nothing but standard API calls in there like they're documented in SwiftyDropbox documentation.

Why is it that the API produces rate limit errors while at the same time I can drag the same folders to my Dropbox folder in Finder and the upload will just work?

[Greg-DB]

Hi, I'll be happy to help with this. Can you share the relevant code snippets and the error output you're getting specifically so I can take a look? Thanks in advance!

[MapTiler]

Hi Greg,

Sure: take the createFolder function from your API and call it a hundred times in a for-loop.

That's all the code you need to simulate.

[Greg-DB]

There are a few different errors you might be getting depending on the specifics of the scenario, so please share the specific code and error output you are getting so I can offer help with the particular issue you're running in to. Thanks in advance!

[MapTiler]

Hi Greg,

You have all the information you need to know to reproduce the problem:

- SwiftyDropbox

- loop a couple of 100 times over your createFolder call

- log the errors from the response callback (response part will be nil, error part will be filled with the error)

- Again, the error = API RATE LIMIT ERROR with a retry_after = 1 in the json reply

That's it.

Storing files is your business, yes? Your API apparently won't let me do that without errors. Asking for source code for something that should be working out of the box in the firstplace is getting downright annoying.

You *know* how to simulate it because I've told you ... twice.

Right now the customer is considering moving away completely from Dropbox. So are we going to get a reply from you or do we move away?

[Greg-DB]

Thanks for following up. There are multiple different rate limit errors you might get, so I asked for the specific error so that I can make sure I offer the right information.

Specifically, with SwiftyDropbox, a RateLimitError will have a RateLimitReason which can be either tooManyRequests or tooManyWriteOperations. 

For 100 calls, I wouldn't expect you to be getting tooManyRequests, but if you are, please let me know as that would likely be a bug. (I did try simulating this by making 100 create folder calls in a loop, but I didn't run in to any errors.)

More likely, I suspect you're getting tooManyWriteOperations. This isn't an imposed rate limit exactly, but rather a technical inability to make multiple modifications at the same time. This is "lock contention", and you can find more information on it in the data ingress guide. The simultaneous activity can be coming from your own app, or from any other source on the user's account, such as other API apps, the official clients, etc.

The way to handle this is the same though. Your app should just retry after the specified Retry-After time. 

One optimization you may be able to make however, is to forgo the create folder calls entirely. If you're uploading files to paths where the parent folders don't already exist, you don't need to make the create folder calls. You can just send the upload calls and the parent folders will be automatically created.

Hope this helps!