We're making changes to the Community, so you may have received some notifications - thanks for your patience and welcome back. Learn more here.
Forum Discussion
Retrieve thousands of files and folders without throttling
I'm trying to generate a nice big array for each user that contains all the files and folders the user owns as well as the users that each file/folder is shared with (e.g. users: ["John Doe": [{"name":"DemoDoc", "type": "file", "isShared": true, "sharedWithUsers": "bob@company.com", "joe@company.com"},{"name":"DemoFolder", "type": "folder", "isShared": false},...]]). I'll need to do this for a few hundred users. How should I do this with as few calls as possible?
The following is one approach per user:
1) Get Dropbox users (POST request): https://api.dropboxapi.com/2/team/members/list
2) Loop through each member and grab their team_member_id
3) For each member (currently 364 members) get all their folders and files recursively: https://api.dropboxapi.com/2/files/list_folder
4) For each member folder, if it's shared get the members and add folder to user object if user is the owner of shared folder: https://api.dropboxapi.com/2/sharing/list_folder_members
5) If folder isn't shared then add folder to user object
6) If file check to see if user is owner of file: https://api.dropboxapi.com/2/sharing/list_file_members
7) If owner of file add file to user object
This approach could be thousands of http calls just for one user, which leads me to think this could be a very time expensive approach as well as the possibility of throttling.
- Greg-DB
Dropbox Staff
[Cross-linking for reference: https://stackoverflow.com/questions/74813769/retrieve-thousands-of-files-and-folders-without-throttling ]
It sounds like you have the right idea here, though be sure to use batch options/endpoints whenever possible. For instance, use recursive:true on /2/files/list_folder and then page through with /2/files/list_folder/continue instead of recursing for each folder yourself. Also, you can use /2/sharing/list_file_members/batch to get the members of multiple files at once.
Also, you may want to start from /2/team/namespaces/list and /2/team/namespaces/list/continue to iterate over and list unique namespaces (such as team member folders, shared folders, etc.) as that may be more efficient than working from the member list.
And for reference, the Dropbox API does have a rate limiting system, but if/when you do hit it, the back-off period is generally only on the scale of seconds/minutes, so that hopefully wouldn't be too disruptive if you do run in to that. You can find more information in the error documentation and Error Handling Guide.
If I used the /2/team/namespaces/list and /2/team/namespaces/list/continue endpoints how would I know who owns the folder and the files within the folder (and their owners)?
- Greg-DB
For team member folders, the /2/team/namespaces/list and /2/team/namespaces/list/continue results include the relevant team_member_id in the NamespaceMetadata object.
For shared folders, you can still use /2/sharing/list_folder_members and /2/sharing/list_folder_members/continue to list the shared folder's members and owner. (And content within a shared folder that isn't itself specifically shared doesn't have an owner distinct from its parent shared folder.)
