You might see that the Dropbox Community team have been busy working on some major updates to the Community itself! So, here is some info on what’s changed, what’s staying the same and what you can expect from the Dropbox Community overall.
Forum Discussion
deeuu
7 years agoExplorer | Level 4
list_folder_continue -> 504 errors
Hello, I'm using `list_folder_continue` of the Python SDK to recursively traverse (via pagination) a folder with a very large number of files (hundreds of thousands). After a while, I'm running int...
- 7 years ago
Thanks for the report! This can be related to these operations taking a long time due to a large number of files and/or a large amount of file activity in the affected account(s).
We'll look into it, but there are a few potential workarounds:
1) Use the 'limit' parameter on files_list_folder:
https://dropbox-sdk-python.readthedocs.io/en/latest/moduledoc.html#dropbox.dropbox.Dropbox.files_list_folder
If you specify a smaller value (the default is effectively around 2000), that should help reduce how long each of these calls takes and reduce the likelihood that they will fail. Note that you supply the 'limit' value to files_list_folder itself, and it will apply to all results from files_list_folder_continue using the returned cursor as well.2) If you are using recursive=True, switch to recursive=False when calling files_list_folder. This means you would need to make a call for each sub-folder you need though.
deeuu
Explorer | Level 4
I should also add that when re-running my script, calling list_folder_continue with the most recent cursor prior to catching the InternalServerError, takes a very long time to get going again, with the same API request attempts:
2018-09-25 08:59:14,422 - dropbox - INFO - Request to files/list_folder/continue
2018-09-25 09:00:44,445 - urllib3.connectionpool - DEBUG - https://api.dropboxapi.com:443 "POST /2/files/list_folder/continue HTTP/1.1" 504 35
2018-09-25 09:00:44,446 - dropbox - INFO - HttpError status_code=504: Retrying in 2.0 seconds
2018-09-25 09:00:46,445 - dropbox - INFO - Request to files/list_folder/continue
2018-09-25 09:02:16,463 - urllib3.connectionpool - DEBUG - https://api.dropboxapi.com:443 "POST /2/files/list_folder/continue HTTP/1.1" 504 35
2018-09-25 09:02:16,464 - dropbox - INFO - HttpError status_code=504: Retrying in 3.8 seconds
2018-09-25 09:02:20,277 - dropbox - INFO - Request to files/list_folder/continue
2018-09-25 09:03:50,319 - urllib3.connectionpool - DEBUG - https://api.dropboxapi.com:443 "POST /2/files/list_folder/continue HTTP/1.1" 504 35
2018-09-25 09:03:50,320 - dropbox - INFO - HttpError status_code=504: Retrying in 7.6 seconds
2018-09-25 09:03:57,961 - dropbox - INFO - Request to files/list_folder/continue
2018-09-25 09:05:27,982 - urllib3.connectionpool - DEBUG - https://api.dropboxapi.com:443 "POST /2/files/list_folder/continue HTTP/1.1" 504 35
2018-09-25 09:05:27,984 - dropbox - INFO - HttpError status_code=504: Retrying in 0.3 seconds
2018-09-25 09:05:28,278 - dropbox - INFO - Request to files/list_folder/continue
Greg-DB
7 years agoDropbox Staff
Thanks for the report! This can be related to these operations taking a long time due to a large number of files and/or a large amount of file activity in the affected account(s).
We'll look into it, but there are a few potential workarounds:
1) Use the 'limit' parameter on files_list_folder:
https://dropbox-sdk-python.readthedocs.io/en/latest/moduledoc.html#dropbox.dropbox.Dropbox.files_list_folder
If you specify a smaller value (the default is effectively around 2000), that should help reduce how long each of these calls takes and reduce the likelihood that they will fail. Note that you supply the 'limit' value to files_list_folder itself, and it will apply to all results from files_list_folder_continue using the returned cursor as well.
2) If you are using recursive=True, switch to recursive=False when calling files_list_folder. This means you would need to make a call for each sub-folder you need though.
- deeuu7 years agoExplorer | Level 4
Thanks for getting back to me Greg.
I'd rather not have to go with #2 because this implies I can't reliably check for changes in the folder at a later stage using the cursor obtained from `files_list_folder_get_latest_cursor`. i.e. #1 would require me to traverse the entire folder each time I want to check for changes (or require me to store multiple cursors for sub-directories etc.).
I've already been `running files_list_folder` with `limit` set to 500, and each time the internal server error occurs after about approx 310,000 files/folders. So I can almost predict when the error will occur.
I'll try it again with `limit` set to 50 and report back tomorrow.
Is this issue related to the number of files in a folder, i.e. limit -> 1 as number of files -> infinity? If so, I'll stop using cursors altogether.
- Greg-DB7 years agoDropbox Staff
Yes, this is likely related to the number of files, but possibly also the amount of file activity. (So, yes, as the number of files goes to infinity, the working limit would go down.)
- deeuu7 years agoExplorer | Level 4
Hi Greg-DB,
So, after about 270,000 files/folders, I received an Internal Server Error (with `limit` set to 50), and can't seem to get going again with the cursor I stored prior to the collapse.
I guess I have to conclude that cursors aren't reliable for traversing folders with many files and sub-folders.
I'm not even sure its worth re-designing my app to use cursors for checking the state of each sub-directory, so I'll have to go with your #2 each time.
About Dropbox API Support & Feedback
Find help with the Dropbox API from other developers.
5,911 PostsLatest Activity: 9 hours agoIf you need more help you can view your support options (expected response time for an email or ticket is 24 hours), or contact us on X or Facebook.
For more info on available support options for your Dropbox plan, see this article.
If you found the answer to your question in this Community thread, please 'like' the post to say thanks and to let us know it was useful!