Re: Issues with long term polling for file changes

jacqkl · ‎03-18-2019

My (Java) app. needs to know how a Dropbox filesystem (or part of it) changes and also has changed while she's not running.

The listFolderLongpoll API seems to be well suited for this purpose and my first tests are promissing.

First question: how long will a cursor be valid ?, for a few hours? , days ?, more ?

But there is one problem, with file renaming; it's seen by the app. as the pair (deleted, new) and, unless my test code is buggy, I don't see any way to "reconstruct" the rename infos. , especially in case of several renames (I didn't test yet the "move" which also must work..).

Second question: what are the alternative to long term poll? , knowing that solutions based on "webhooks like" techniques are not an option.

Greg-DB · ‎03-18-2019

My (Java) app. needs to know how a Dropbox filesystem (or part of it) changes and also has changed while she's not running.

The listFolderLongpoll API seems to be well suited for this purpose and my first tests are promissing.

You can use the listFolder[*] methods to see what exists in the account, and what has changed. The listFolderLongpoll method in particular is only meant for when apps are running, not for when they're not running. It allows the app to hold open a long running connection to receive low latency notification of when changes occur, which doesn't work if the app isn't running.

The basic flow should work like this:

The app uses listFolder to begin listing items, and saves the returned cursor.
If 'has_more' was returned as true, the app uses the resulting cursor with listFolderContinue to continue listing more items, and saves the latest returned cursor, and repeats if 'has_more' is still true.
Once caught up (i.e., 'has_more' is false), the app can use the latest cursor with listFolderLongpoll to monitor for changes.
If any changes occur, the apps calls listFolderContinue again to get the changes, and repeats.
The app is quit by the user at any point.
The app is re-launched by the user at any point.
The app uses the latest cursor with listFolderContinue again to get any changes that occurred when it wasn't running, and repeats the process.

First question: how long will a cursor be valid ?, for a few hours? , days ?, more ?

Cursors don't expire automatically, however there are some scenarios where they can become invalid. You should catch ListFolderContinueErrorException when calling listFolderContinue accordingly. A ListFolderContinueError.RESET indicates that the cursor is no longer valid.

But there is one problem, with file renaming; it's seen by the app. as the pair (deleted, new) and, unless my test code is buggy, I don't see any way to "reconstruct" the rename infos. , especially in case of several renames (I didn't test yet the "move" which also must work..).

Yes, renames (and moves) are reported as a deletion/addition pair. You can track files across renames and moves using the file ID, which doesn't change for renames/moves.

Second question: what are the alternative to long term poll? , knowing that solutions based on "webhooks like" techniques are not an option.

In general, longpolling is meant for client-side apps, and webhooks are meant for server-side apps. If neither work for your use case, the alternative is to just call listFolderContinue ocassionally to check for changes.

View solution in original post

Greg-DB · ‎03-18-2019

My (Java) app. needs to know how a Dropbox filesystem (or part of it) changes and also has changed while she's not running.

The listFolderLongpoll API seems to be well suited for this purpose and my first tests are promissing.

You can use the listFolder[*] methods to see what exists in the account, and what has changed. The listFolderLongpoll method in particular is only meant for when apps are running, not for when they're not running. It allows the app to hold open a long running connection to receive low latency notification of when changes occur, which doesn't work if the app isn't running.

The basic flow should work like this:

The app uses listFolder to begin listing items, and saves the returned cursor.
If 'has_more' was returned as true, the app uses the resulting cursor with listFolderContinue to continue listing more items, and saves the latest returned cursor, and repeats if 'has_more' is still true.
Once caught up (i.e., 'has_more' is false), the app can use the latest cursor with listFolderLongpoll to monitor for changes.
If any changes occur, the apps calls listFolderContinue again to get the changes, and repeats.
The app is quit by the user at any point.
The app is re-launched by the user at any point.
The app uses the latest cursor with listFolderContinue again to get any changes that occurred when it wasn't running, and repeats the process.

First question: how long will a cursor be valid ?, for a few hours? , days ?, more ?

Cursors don't expire automatically, however there are some scenarios where they can become invalid. You should catch ListFolderContinueErrorException when calling listFolderContinue accordingly. A ListFolderContinueError.RESET indicates that the cursor is no longer valid.

But there is one problem, with file renaming; it's seen by the app. as the pair (deleted, new) and, unless my test code is buggy, I don't see any way to "reconstruct" the rename infos. , especially in case of several renames (I didn't test yet the "move" which also must work..).

Yes, renames (and moves) are reported as a deletion/addition pair. You can track files across renames and moves using the file ID, which doesn't change for renames/moves.

Second question: what are the alternative to long term poll? , knowing that solutions based on "webhooks like" techniques are not an option.

In general, longpolling is meant for client-side apps, and webhooks are meant for server-side apps. If neither work for your use case, the alternative is to just call listFolderContinue ocassionally to check for changes.

jacqkl · ‎03-19-2019

Hello Greg,

I was please to read your reply, and accepted it, maybe a little bit early.

Made an attempt to solve the rename issue with the suggested FileMetadata.id, unfortunatelly, there is no such id for DeletedMetadata, means I will have to keep all id's in an app. cache, too bad ...

An other suggestion ?

Greg-DB · ‎03-19-2019

That's correct, unfortunately DeletedMetadata doesn't include the file ID, but I'll pass this along as a feature request. I can't promise if or when that might be implemented though.

For files, you can call /2/files/list_revisions with the path from the DeletedMetadata to get the history of the file at that path though, including the file ID(s).

Alternatively, you can store the path and id for the each last (File|Folder)Metadata you saw, so that when you get a DeletedMetadata, you can check what the last known file ID was for that path.