Some files not syncing from Mac computer to Ubuntu but others do

---

BrunoJ wrote:

...

Hopeful this trend and your work will help the dropbox developers with fixing this issue?

...

---

🙂🤔 Hm.. Good wish, but better don't rely on. My experience shows - if the application doesn't stop completely and some part of the UI doesn't become too 'ugly', Dropbox development doesn't take care. 🤷 Linux is NOT first class citizen here, so everybody should save himself, unfortunately.

My script is definitely not a final solution; it's rather workaround using 'brute force' file fixing (cosa there's not know what exactly force wrong behavior). Nowhere can be seen any documentation describing the possible file attributes leading to such thing. That's why, in my script, I'm clearing all files attributes (accessible or not - in fact files regeneration).

---

BrunoJ wrote:

...,  but it seems to be me (a basic user) it feels too much of a hassle to do all that for I service that I pay a considerable amount of money per year. ...

---

Fully agree that once a user's paying (and not only), the service should be the same everywhere (on every supported platform). 😡 Unfortunately, it's not and, as I said, everybody is looking for solution outside Dropbox (or directly another service - there are competitors).

---

BrunoJ wrote:

... Frankly, if the problem was much bigger than just a few photoshop files ...

---

Especially for you I made some changes in above script and I'm posting it here:

#!/usr/bin/bash
##############################################################################
#                  Workaround for Dropbox broken sync
#                  ----------------------------------
# Verified on Dropbox v159.4.5870
# Just make it executable (if need, using
#   $ chmod a+x dropbox_syncfix
# ) and run it.
# Author: Здравко
#   www.dropboxforum.com/t5/user/viewprofilepage/user-id/422790
##############################################################################

# General command line check.
if [ $# -eq 0 ]
then
  echo "Command line should be 'dropbox_syncfix <access token> [<folder>]'."
  echo "The access token seems missing!"
  echo "You can get it here:"
  echo "https://dropbox.github.io/dropbox-api-v2-explorer/#check_user"
  exit 1
fi
if [ $# -gt 2 ]
then
  echo "Command line should be 'dropbox_syncfix <access token> [<folder>]'."
  echo "Unexpected number of arguments passed!"
  exit 1
fi

##############################################################################
# Access token validity confirmation.
randstr=$(
tmpbase='0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'
for ((i=0; i<16; ++i)); do echo -n "${tmpbase:RANDOM%${#tmpbase}:1}"; done
)
res=`curl -X POST https://api.dropboxapi.com/2/check/user \
  -H "Authorization: Bearer $1" -H 'Content-Type: application/json' \
  -d "{\"query\":\"$randstr\"}" 2> /dev/null | jq -r .result`
if [[ $res != $randstr ]]
then
  echo "👆 The first argument ($1) is NOT valid access token!"
  exit 1
fi
access_token="$1"

##############################################################################
# Figuring out local Dropbox folder location.
dbx_dir=`jq -r .personal.path ~/.dropbox/info.json`
if [ $? -ne 0 ]
then
  echo "Cannot detect Dropbox application installed and worked local."
  exit 1
fi
if [ ! -d "$dbx_dir" ]
then
  echo "Incorrect Dropbox folder path ($dbx_dir)!"
  echo "Only individual Dropbox account is supported."
  exit 1
fi

##############################################################################
# Dealing with start folder path.

# Setting default path - the account root (empty string).
start_folder=''

# Checking for explicit start folder path.
if [ $# -eq 2 ]
then
  if [ ! -d "$2" ]
  then
    echo "😯 The second argument ($2) does NOT point a folder."
    exit 1
  fi
  start_folder=`realpath "$2"`
  if [[ "${start_folder:0:${#dbx_dir}}" != "$dbx_dir" ]]
  then
    echo -n "🤦 The pinted folder ($2) does NOT reside within your Dropbox "
    echo "folder ($dbx_dir)."
    exit 1
  fi
  start_folder="${start_folder:${#dbx_dir}}"
fi

##############################################################################
# Single file sunc.
# $1 - relative file path in your account
sync_to_local() {
  local path="$dbx_dir$1"
  echo "Processing file: '$path'..."
  local error=`curl -o /tmp/dropbox_tmp.file -D /tmp/dropbox_tmp.dump \
    -X POST https://content.dropboxapi.com/2/files/download \
    -H "Authorization: Bearer $access_token" \
    -H "Dropbox-API-Arg: {\"path\":\"$1\"}" 2> /dev/null`
  if [ $? -ne 0 ]
  then
    echo "👺 Trouble fetching '$1' from Dropbox"
    return
  fi
  local file_date=`grep 'Dropbox-Api-Result' /tmp/dropbox_tmp.dump | \
              sed 's/^Dropbox-Api-Result *\: *//' | jq -r .client_modified`
  if [[ "$file_date" == "null" ]]
  then
    echo "👹 Download '$1' error: $error"
    return
  fi
  touch -d "$file_date" /tmp/dropbox_tmp.file
  local res=`curl -X POST https://api.dropboxapi.com/2/files/delete_v2 \
    -H "Authorization: Bearer $access_token" \
    -H 'Content-Type: application/json' -d "{\"path\":\"$1\"}" 2> /dev/null`
  if [[ `echo "$res" | jq -r '.metadata[".tag"]'` != "file" ]]
  then
    echo "🤷 Cannot remove '$1', error: $res"
    return
  fi
  sleep 1
  while [[ `curl -X POST https://api.dropboxapi.com/2/files/get_metadata \
    -H "Authorization: Bearer $access_token" \
    -H 'Content-Type: application/json' -d "{\"path\":\"$1\"}" 2> /dev/null \
    | jq .error_summary` == "null" ]]
  do
    sleep 1
  done
  mv -u /tmp/dropbox_tmp.file "$path"
  echo "'$path' processed."
}

##############################################################################
# Processing single enumeration step.
# $1 - current listing result
# $2 - current folder relative path
enumerate_files() {
  local oldIFS="$IFS"
  IFS=$'\n'
  for path in `jq -r '.entries[] | select(.[".tag"] == "file").name' <<< "$1"`
  do
    if [ ! -f "$dbx_dir$2/$path" ]
    then
      sync_to_local "$2/$path"
    fi
  done
  for path in `jq -r '.entries[] | select(.[".tag"] == "folder").name' <<< "$1"`
  do
    parse_folder "$2/$path"
  done
  IFS="$oldIFS"
}

##############################################################################
# Processing single folder (indirect recursion).
# $1 - relative folder path
parse_folder() {
  local res=`curl -X POST https://api.dropboxapi.com/2/files/list_folder \
    -H "Authorization: Bearer $access_token" \
    -H 'Content-Type: application/json' \
    -d "{\"path\":\"$1\",\"include_non_downloadable_files\":false}" 2> /dev/null`
  echo "$1"
  enumerate_files "$res" "$1"
  while [[ `jq .has_more <<< "$res"` == "true" ]]
  do
    res=`curl -X POST https://api.dropboxapi.com/2/files/list_folder/continue \
      -H "Authorization: Bearer $access_token" \
      -H 'Content-Type: application/json' \
      -d "{\"cursor\":$(jq .cursor <<< "$res")}" 2> /dev/null`
    echo "$1"
    enumerate_files "$res" "$1"
  done
}

##############################################################################
# Starting enumeration  with the start folder.
parse_folder "$start_folder"

##############################################################################
# Wiping the 'dust'.
rm -f /tmp/dropbox_tmp.file /tmp/dropbox_tmp.dump

echo
echo "Complete! 😉"
echo

 The primary change (addition in fact) is the ability to provide folder path in your Dropbox folder. So if your troublesome files are located on limited part of the account folders tree and you know exactly where, providing path (second argument) can speed up script significant. The time needed can be just few minutes or, on very few files, less than a minute. 😉 Now you can use a call like:

./dropbox_syncfix PUT_YOUR_TOKEN_HERE "Dropbox/path/to/the/troubles"

Everything else is like on my previous post above.

PS: I may have forgotten to mention - in the script is used 'jq' tool (able handle JSON format used by Dropbox). The tool is relatively specific and that's why usually it's not preinstalled. You may need to install it explicitly, using your preferable package manager or using following command on Ubuntu (and compatible; other distributions have equivalent commands):

sudo apt-get install jq

Здравко 

thank you for this script! Could you briefly explain what it does exactly?

---

FesT wrote:

... Could you briefly explain what it does exactly?

---

As I said already, main trick is recreation of every problematic file in the Dropbox namespace, so any undesirable attributes (assigned to the former file) get gone. This is the most brief variant.

Within the script itself are many comments, describing what exactly each part of the script does. You need only basic shell scripting knowledge to understand them. About the algorithm - enumeration starts from a target folder (by default the account root) and when non matching file comes up - passed for 'refresh and undressing' (if I can say so). After that the nested folders are processed in recursion, following the same rule of course. The enumeration (and other things, like download, etc) gets performed using corresponding Dropbox API access point. You can follow the links for more information. This is less brief variant.

The same is valid for both script variants. 👍 If something isn't clear enough, clarify your question.

Hope this helps.

Hi Здравко,

thank you for the short explanation. So far I had already understood it, what I do not understand exactly is what for "undesirable attributes" are removed, and in what way they change the files.

---

FesT wrote:

..., what I do not understand exactly is what for "undesirable attributes" are removed, ...

---

Hm... 🤔🤷 To be honest I'm not sure what actually you are asking for. The attributes are undesirable because they break the syncing and that why they are removed! If these attributes are Ok for you, don't take a look on any of the scripts any more (you don't need them). As far as I understand, sync breaking is an issue. Am I right, or I might have misunderstanding something? 🧐

---

FesT wrote:

..., and in what way they change the files.

---

The files themself are not changed in any way. File attributes are things "around" the file (things associated to a file, not file's integral part). Seems some Dropbox application instances (the one on Mac) attach to some files such attributes on upload (don't ask me why; seems nobody knows - including Dropbox staff). Removing these undesired attributes (hope you know already why 'undesired') seems resume syncing.

HAHA

---

Здравко wrote:
...(don't ask me why; seems nobody knows - including Dropbox staff).

this is exactly what I some how tried to figure out... where these attributes are coming from, why there are there, and what happens if there are not there anymore (except that syncing is working again – which obviously is good 😁)

Anyhow thank you a lot for this great script!