You might see that the Dropbox Community team have been busy working on some major updates to the Community itself! So, here is some info on what’s changed, what’s staying the same and what you can expect from the Dropbox Community overall.
Forum Discussion
Help Understanding how /files/list_folder/continue
I am working on an AWS serverless app that queries a specific DropBox folder tree for daily PDF uploads. My process and config/code are below. I _think_ I understand how the API endpoint is supposed ...
Thanks for the detailed writeup!
First, I should note that the 'dropbox-v2-api' package isn't made by Dropbox itself, so I can't really offer support for that or say what it may actually be doing under the hood, but I'll take a look here and advise with respect to the Dropbox API.
Anyway, looking over your code and description, it looks like you have the right basic idea here for the most part (though it will depend on exactly what you're trying to accomplish of course), but there are a few things to note:
- Regarding the deleted entries, note that the 'include_deleted' parameter only applies to "entries for files and folders that used to exist but were deleted", that is, at the time of the call to /2/files/list_folder/get_latest_cursor. Files or folders that are deleted after that call will still be reported later by /2/files/list_folder/continue as 'deleted'. Does this account for the entries you're seeing? Essentially, it may just be items deleted between 2:00 AM and 3:00 PM. If that doesn't seem to be it, perhaps you could share a sample so we can take a look? Feel free to open an API ticket privately if you'd prefer.
- Also, I don't see you checking the 'has_more' value returned by /2/files/list_folder/continue. You're not guaranteed to get everything back in one call, so you should check that 'has_more' value and call back again to /2/files/list_folder/continue as described in the /2/files/list_folder documentation.
- Also, it may or may not make sense for your use case, but you don't need to call /2/files/list_folder/get_latest_cursor every day. You can store and re-use the last cursor you received to be able to just receive updates about changes that have occurred since you received that cursor. That would let you track all changes over time. As written, it seems you're not monitoring anything that occurs between 3:00 PM and 2:00 AM. The Detecting Changes guide may be helpful, if you haven't already read it.
Greg-DB
Dropbox Staff
Dropbox StaffThanks for the detailed writeup!
First, I should note that the 'dropbox-v2-api' package isn't made by Dropbox itself, so I can't really offer support for that or say what it may actually be doing under the hood, but I'll take a look here and advise with respect to the Dropbox API.
Anyway, looking over your code and description, it looks like you have the right basic idea here for the most part (though it will depend on exactly what you're trying to accomplish of course), but there are a few things to note:
- Regarding the deleted entries, note that the 'include_deleted' parameter only applies to "entries for files and folders that used to exist but were deleted", that is, at the time of the call to /2/files/list_folder/get_latest_cursor. Files or folders that are deleted after that call will still be reported later by /2/files/list_folder/continue as 'deleted'. Does this account for the entries you're seeing? Essentially, it may just be items deleted between 2:00 AM and 3:00 PM. If that doesn't seem to be it, perhaps you could share a sample so we can take a look? Feel free to open an API ticket privately if you'd prefer.
- Also, I don't see you checking the 'has_more' value returned by /2/files/list_folder/continue. You're not guaranteed to get everything back in one call, so you should check that 'has_more' value and call back again to /2/files/list_folder/continue as described in the /2/files/list_folder documentation.
- Also, it may or may not make sense for your use case, but you don't need to call /2/files/list_folder/get_latest_cursor every day. You can store and re-use the last cursor you received to be able to just receive updates about changes that have occurred since you received that cursor. That would let you track all changes over time. As written, it seems you're not monitoring anything that occurs between 3:00 PM and 2:00 AM. The Detecting Changes guide may be helpful, if you haven't already read it.
Ok, that clarifies the deleted file issue perfectly. I thought it meant to exclude all deleted so that makese sense.
I do need to update the cursor every day because I only want the files for that day. If I don't update the cursor, won't that give me all files since the cursor? So that would give me all of the files going back possibly multiple days. That won't fit my use case.
Yes, I caught the has_more issue last night. Technically that is the right thing to do but it won't have any tangible impact since I have the limit set to 2,000 files and we are not coming anywhere close to that. The most I have seen to-date is 900 in one day including the deleted files. I am deploying the update once I have tested it but did not include it here because I have not fully tested the new code.
Thanks for the response. I think this resolves my issue. I mainly needed to confirm I'm understanding the way the API endpoint works and to clarify on the deleted files.
- Greg-DB
Great, I'm glad that helps.
To further clarify a few things though:
"I only want the files for that day. If I don't update the cursor, won't that give me all files since the cursor? So that would give me all of the files going back possibly multiple days."
One option is to always update your stored cursor to be the latest cursor you last received, e.g., from /2/files/list_folder/continue itself. When you then call /2/files/list_folder/continue again, you'll only receive updates that occurred since that call to /2/files/list_folder/continue that gave you that cursor.
"it won't have any tangible impact since I have the limit set to 2,000 files and we are not coming anywhere close to that. "
Be aware that the "limit" is only an approximate upper bound on how many items Dropbox will return per page; it does not affect the lower bound. In some cases, Dropbox may have to return far fewer entries per page, in which case you do need to check and follow 'has_more'.
I know this was posted about 3 years ago, but in case you read this, is there a TTL on the cursor? So if I don't run the script again for, say, 6 months, would the cursor, at least in theory, still be valid? I'm just curious about this. Our solution has been humming along perfectly for the past 2-3 years (thanks to your help) but this would be good to know.
- Greg-DB
iconify These list_folder cursors do not expire by default, so they could last that long. However it is possible for them to become invalid at any time, due to some kinds of operations in the account, at which point /2/files/list_folder/continue would return a 'reset' error. That being the case, make sure your app is able to catch that error at any point; it would then need to restart from /2/files/list_folder to get a new valid cursor.
About Discuss Dropbox Developer & API
Make connections with other developers
