409 - Internal Error

[rubyrogue]

I am having an intermittent issue with checking the status of async share jobs. Below is the flow I'm building and below that an real error being returned by the API. This is for an internal Dropbox app that we are building and is not yet in "production" mode.

The Flow:
1) Create new folder inside the team space/folder
2) Create 13 subfolders inside that folder
3) Check the status of sharing each subfolder, polling every 5 seconds until it is "complete"
4) Finally, we add one of our groups to that folder via the "add_folder_member" endpoint.

We ALWAYS create the parent folder successfully. Typically the error occurs around the 3rd or 4th subfolder.

This is currently blocking our integration so any feedback or help would be deeply appreciated.

POST https://api.dropboxapi.com/2/sharing/check_share_job_status
Dropbox-API-Select-Admin: "XX"
Dropbox-API-Path-Root: "{\".tag\":\"namespace_id\",\"namespace_id\":\"6925873200\"}"
Authorization: "Bearer XXX"
User-Agent: "Faraday v1.0.0"
content-type: "application/json"
{"async_job_id":"dbjid:AAAk3C8gUV9HTzIJurjlHh32N2KlIfZ3pgHAgOVrN88ewvaV-fKy-HdKvNPhc7w4JCDB3jxHEsPMJnaOfrVwXhq5"}
Status 409
server: "nginx"
date: "Fri, 17 Jan 2020 22:06:28 GMT"
content-type: "application/json"
transfer-encoding: "chunked"
connection: "close"
content-security-policy: "sandbox; frame-ancestors 'none'"
x-dropbox-request-id: "21f7ab0bb9e6390b090c5c60050ed683"
x-frame-options: "DENY"
x-content-type-options: "nosniff"
content-disposition: "attachment; filename='error'"
{"error_summary": "internal_error/.", "error": {".tag": "internal_error"}}

[Greg-DB]

Thanks for the report! We can help look into this, but it looks like we could use some more information. Could you also:

• share some sample API requests for the earlier API calls you're making (i.e., steps 1 and 2 in your outline), like you did for /check_share_job_status
• let me know if you're sending the requests in step 2 serially or in parallel

Thanks in advance.

[rubyrogue]

Step 1: Create folder inside team folder

I, [2020-01-17T17:00:41.565464 #82529]  INFO -- request: POST https://api.dropboxapi.com/2/files/create_folder
I, [2020-01-17T17:00:41.565544 #82529]  INFO -- request: Dropbox-API-Select-Admin: "dbmid:XXX"
Dropbox-API-Path-Root: "{\".tag\":\"namespace_id\",\"namespace_id\":\"6925873200\"}"
Authorization: "Bearer XXX"
User-Agent: "Faraday v1.0.0"
content-type: "application/json"
I, [2020-01-17T17:00:42.941609 #82529]  INFO -- response: Status 200
I, [2020-01-17T17:00:42.941677 #82529]  INFO -- response: server: "nginx"
date: "Fri, 17 Jan 2020 22:00:42 GMT"
content-type: "application/json"
transfer-encoding: "chunked"
connection: "close"
vary: "Accept-Encoding"
cache-control: "no-cache"
pragma: "no-cache"
x-content-type-options: "nosniff"
x-dropbox-request-id: "e64ce3531e16da2486a525ab2529b08e"
x-frame-options: "SAMEORIGIN"
x-server-response-time: "1045"

Step 2: Create sub folder

I, [2020-01-17T17:00:42.942427 #82529]  INFO -- request: POST https://api.dropboxapi.com/2/sharing/share_folder
I, [2020-01-17T17:00:42.942464 #82529]  INFO -- request: Dropbox-API-Select-Admin: "dbmid:XXXX"
Dropbox-API-Path-Root: "{\".tag\":\"namespace_id\",\"namespace_id\":\"6925873200\"}"
Authorization: "Bearer XXX"
User-Agent: "Faraday v1.0.0"
content-type: "application/json"
I, [2020-01-17T17:00:43.705937 #82529]  INFO -- response: Status 200
I, [2020-01-17T17:00:43.706004 #82529]  INFO -- response: server: "nginx"
date: "Fri, 17 Jan 2020 22:00:43 GMT"
content-type: "application/json"
transfer-encoding: "chunked"
connection: "close"
vary: "Accept-Encoding"
cache-control: "no-cache"
pragma: "no-cache"
x-content-type-options: "nosniff"
x-dropbox-request-id: "46a23463fd10fb41ebec5f5496c35f91"
x-frame-options: "SAMEORIGIN"
x-server-response-time: "583"

Steps 2,3, and 4 are part of a serial process, not concurrent. Basically the flow for creating subfolders is as follows:

1) Share subfolder (this creates a folder if one doesn't exist yet)
2) Check the share_job_status from the previous call
3) If it is not complete, loop every 5 seconds and wait for it to become completed
4) Once complete we add the required group to that subfolder an example of that request is below:

I, [2020-01-17T17:00:49.532937 #82529]  INFO -- request: POST https://api.dropboxapi.com/2/sharing/add_folder_member
I, [2020-01-17T17:00:49.532977 #82529]  INFO -- request: Dropbox-API-Select-Admin: "dbmid:XXX"
Dropbox-API-Path-Root: "{\".tag\":\"namespace_id\",\"namespace_id\":\"6925873200\"}"
Authorization: "Bearer XXX"
User-Agent: "Faraday v1.0.0"
content-type: "application/json"
I, [2020-01-17T17:00:52.328615 #82529]  INFO -- response: Status 200
I, [2020-01-17T17:00:52.328701 #82529]  INFO -- response: server: "nginx"
date: "Fri, 17 Jan 2020 22:00:52 GMT"
content-type: "application/json"
transfer-encoding: "chunked"
connection: "close"
vary: "Accept-Encoding"
cache-control: "no-cache"
pragma: "no-cache"
x-content-type-options: "nosniff"
x-dropbox-request-id: "5769349f140fa5d94a2537455bc7f568"
x-frame-options: "SAMEORIGIN"
x-server-response-time: "2633"

[Greg-DB]

Thanks for the additional information! That's helpful.

Performing these serially is good, but unfortunately due to some complexities with the implementation of nested shares like this on the backend, it is still possible to run in to some asynchronous/delayed contention when doing so. 

That being the case, can you try adding a 10 second delay before starting each subsequent share_folder job? That is, after you've completed sharing a folder and adding member(s) to it, wait 10 seconds before calling share_folder again for the next one.

Please let me know if that does or doesn't seem to avoid the error for you. Thanks!

[rubyrogue]

Thanks for responding Greg. This is interesting. From my logs it isn't the share_folder call that is erroring but `add_folder_member`. So it appears that checking the status of async jobs sometimes fails.

Does your recommendation change with this clarification? Or would you still isolate "shares" to 10 seconds apart?

[Greg-DB]

Apologies, I'm not sure I follow. In your original message, the error output you shared appears to be from the response to a call to /2/sharing/check_share_job_status (for checking the job started by calling /2/sharing/share_folder). In your latest message though, you just said you're getting an error from /2/sharing/add_folder_member.

Was that a typo, or are you also getting errors from /2/sharing/add_folder_member? If it's the latter please share a sample of that. Thanks!

[rubyrogue]

Yes, apologies on that copy/paste fail 😞 The main API call that fails is `check_share_job_status`. It will succeed a few times returned back that it is "in_progress" and then on the 3rd or 4th or 5th call it will return "internal_error". 

What I was curious about was when to do the "waiting". If I am reading your feedback correctly, the following is being suggested:

1) Share folder
2) Check share status (waiting 5 seconds per loop)
3) On success, add member
4) Wait 10 seconds before calling the `share` endpoint for the next folder.

Is this what you meant? Or were you suggesting that I increase the 5 second wait time when polling `check_share_job_status` to 10 seconds?

[Greg-DB]

No problem, thanks for confirming.

Anyway, yes, I meant adding a new 10 second delay before the next /2/sharing/share_folder call.

[rubyrogue]

Thanks Greg. I implemented and tested this change. So far it appears to have reduced the number of times we get the `internal_error`, but it has not eliminated it. Is there anything else you can think of to help make this api flow more reliable?

[Greg-DB]

I see, thanks for trying that and letting me know! Although it hasn't completely avoided the issue, the fact that it reduced it does indicate that this is likely due to the delayed contention issue I mentioned. Unfortunately, there isn't a better workaround for this, and 10 seconds was just an estimate of what would avoid it. I'll send this along to the team to see if we can reduce or eliminate this issue to begin with, but I can't make any promises on if or when that would be possible.

To try to fully work around this though, you may want to try increasing the delay a little further, perhaps to 15-20 seconds. Apologies I don't have a better solution for you!