Downloading large amount of files to my web application

daynightdev  and all other interested,

There are many undocumented Dropbox calls. Some of them may be useful in different cases (like the current one). One workaround, I know about, is usage of a call that deals with shared links (unfortunately that is inapplicable to temporary link). To use it, reorganization of files that need to be accessed may be needed - all of them have to be accessible in one folder and a link to this folder should exist or will be created. The workaround implies the Dropbox class update (named here 'DropboxLinkList' and implemented in Python) as follows:

#!/bin/python3
###############################################################################
# Addition of undocumented link listing functionality to regular Dropbox client
# -----------------------------------------------------------------------------
# file: dropbox_client_link_list.py​
# Author: Здравко
#   www.dropboxforum.com/t5/user/viewprofilepage/user-id/422790
###############################################################################

from dropbox import Dropbox
from dropbox.exceptions import BadInputError, HttpError, ApiError
from urllib.parse import urlparse, parse_qs
import json

class DropboxLinkList(Dropbox):
  """
  Use this class to make requests to the Dropbox API using a user's access
  token. Methods of this class are meant to act on the corresponding user's
  Dropbox. In addition :meth:`sharing_list_shared_link_folder` may be called.
  """
  def sharing_list_shared_link_folder(self, path_link):
    """
    Returns list of links (and other data) pointing to all direct entries
    (files/folders) residing in the pointed folder. When ``path_link`` refers
    to the target using path in the current context existing link is used or
    new one is created if there is no existing.
    
    Route attributes:
      No specific scopes are needed for this method usage,
      but :meth:`sharing_create_shared_link_with_settings` will be needed when
      path is in use to refer target. (i.e. sharing.write) and scope of
      :meth:`sharing_list_shared_links` may be needed too (i.e. sharing.read).
    
    :param str path_link: Either path pointing folder (in current context)
      containing all entries that need to be listed or Dropbox shared link to
      folder which content needs to be listed.
    
    The type of return result is not documented by Dropbox, but usable links
      are referred in '.shared_link_infos[].url' of the response.
    """
    if not path_link.startswith("https://"):
      try:
        metadata = self.sharing_create_shared_link_with_settings(path_link)
      except ApiError as e:
        er = e.error
        if not er.is_shared_link_already_exists():
          raise
        er = er.get_shared_link_already_exists()
        if er != None and er.is_metadata():
          metadata = er.get_metadata()
        else:
          metadata = sharing_list_shared_links(path_link, direct_only=True
                                               ).links[0]
        path_link = metadata.url
    if path_link.startswith("https://www.dropbox.com/sh/"):
      link_type = 's'
    elif path_link.startswith("https://www.dropbox.com/scl/fo/"):
      link_type = 'c'
    else:
      raise BadInputError('', f"Unsupported link type: {path_link}")
    url = urlparse(path_link)
    path = url.path.split('/')
    body = {'t': 'T', 'link_type': link_type}
    if link_type == 's':
      body['link_key'] = path[2]
      body['secure_hash'] = path[3]
      body['sub_path'] = "/".join(path[4:])
    else:
      body['link_key'] = path[3]
      body['secure_hash'] = path[4]
      body['sub_path'] = "/".join(path[5:])
      rlkey = parse_qs(url.query)
      if 'rlkey' not in rlkey:
        raise BadInputError('',
          f"Expected 'rlkey' query param, but is missing: {path_link}")
      body['rlkey'] = rlkey['rlkey'][0]
    url = 'https://www.dropbox.com/list_shared_link_folder_entries'
    headers = {'Content-Type': 'application/x-www-form-urlencoded',
               'Cookie': 't=T'}
    body = "&".join(f"{k}={v}" for (k,v) in body.items())
    r = self._session.post(url, headers=headers, data=body, stream=False,
                           verify=True, timeout=self._timeout)
    if r.status_code != 200:
      raise HttpError('', r.status_code, r.content.decode())
    assert r.headers.get('content-type') == 'application/json', (
                'Expected content-type to be application/json, got %r' %
                r.headers.get('content-type'))
    return json.loads(r.content.decode())

As mentioned within the code, existing link of a folder may be listed or directly referring to the folder keeping needed files (with path, id, etc.). Let's say somebody keeps lot of files in a folder with path '/ForListing' that need to be pointed with a link (every one). Instead of looping through all of them and get links one by one (generating lot of API calls), the mentioned folder may be enumerated directly with just few calls (2 or 3 at most):

import .dropbox_client_link_list

# Proper init here - the same like original Dropbox class
dbx = dropbox_client_link_list.DropboxLinkList(...)
container_folder = '/ForListing​'

...

for lm in dbx.sharing_list_shared_link_folder(container_folder)['shared_link_infos']:
    print(lm['url'])

...

 The above will list links for all entries (files/folders) inside 'ForListing' folder. They will not be generated - they exist (pre-generated) while files reside within pointed folder. If needed files are hierarchically spread in directory subtree of the pointed folder (not just flat in a single folder), recursive enumeration may be needed, as follows:

...

def linkList(path_link, prefix=''):
    entries = dbx.sharing_list_shared_link_folder(path_link)
    for lm, tm in itertools.zip_longest(entries['shared_link_infos'], entries['entries']):
        url = lm['url']
        is_dir = tm['is_dir']
        print(f"{'D: ' if is_dir else 'F: '}{prefix}{url}")
        if is_dir:
            linkList(url, prefix + "  ")

linkList(container_folder)

Here one more call for each subfolder will be needed. The chance for API call rate overrun to be meet here is negligible. The above code-snippets are for illustration only. The idea is simple enough, so easy enough to be implemented/'translated' into any other programming language. 😉

Good luck.