How to fix being stuck on Indexing forever loop of death on multi million file DBs after db crash

Ok so I've used Dropbox for almost as long as its existed and recently due to frustration with the never finishing indexing bug I was forced to find out why this kept happening so I could prevent it.

Bear with me on this long post but trust me its worth it, what I found was mind blowing and game changing.

So our business Dropbox is more than 9 million files strong, I've noticed REALISTICALLY any machine handling over 2 million will just enter an indexing loop at some point from which it will never recover, after its happened 5 times in the last week I was pissed enough to that I decided I was going to find out why this is happening, I know I'm pushing the limits but we've had machines with 2.5m files running fine for years, why some work fine and some don't was a mystery, one I was determined to find out.

When you add stuff to your Dropbox, Dropbox has to index it so  it can know what to do with it.

If you add "too much stuff"  (copying 200,000 files of small size in one go, coders know what I'm talking about)

or

do it "too fast" (changing access permissions on 1.5 million files located within Dropbox in one go in less than 5 minutes)

on a computer with too many files (1-2+m)

this causes Dropbox to start indexing them all at once causing the system to slow to a crawl

however if you don't let Dropbox finish before doing something else (like adding more files or using the computer for other tasks)

or

someone else adds a bunch of files on another machine

its almost for sure going to cause Dropbox to crash and restart during this process (it happens quite frequently),

this crashing and restarting triggers a full reindexing of the ENTIRE database, ALL files, and since the machine is already trying to download or upload some of the new files while trying to reindex the current millions of files, doing both at once overtaxes it, which causes it to crash, and were back to square one with the infinite indexing crash loop.

This kept happening to us all the time, the only solution was to unlink and relink the Dropbox account so all pending changes were lost, we got a bunch of conflicted copies and spent days sorting out the mess.

So I figured I needed to see what was going on with Dropbox, what it was doing when it was "indexing" to find out what was causing the crashes.

So after looking for a while I found using Microsoft's sysinternals process monitor (https://docs.microsoft.com/en-us/sysinternals/downloads/procmon) configured in a certain way allowed me to look at EXACTLY what dropbox was doing and I found out all I just told you and another very important piece of info.

(if you want to see how awesome it looks in action check out THIS video:)

When you click PAUSE FILE SYNCING you would think Dropbox pauses and ceases all operations, but you are wrong!, it pauses all incoming and outgoing transfers but any INDEXING tasks keep going!

This is an absolute game changer! now if I see a machine that says "Indexing" for a long time, I turn on the process monitor, hit pause on the file syncing and watch as the machine does the indexing at super high speed (5-10 times faster than doing it while downloading), it usually finishes doing the full file reindexing in a couple of minutes once its done I can hit back resume and keep going, I've never had the app crash while doing this "offline" or "paused" indexing, thus avoiding the inevitable crash, and reindex loop.

I have been successful in recovering 4 machines from the indexing loop of death using this method, where before I was screwed and had to eat the duplicate files and cleanup for a week and a TON of annoyed users in the office.

Basically if your machine is taking too long indexing or is stuck indexing after a crash just hit "pause for 1 hour" and forget about it, it will still be working on the indexing in the background, and when it restarts it should have finished the re-index avoiding a crash when trying to download/upload the new files.

Id wish Dropbox would have told us this, I never expected it to keep indexing while paused, I assumed pause was PAUSE, as in, cease all operations, it would have saved me so many headaches.

All they need to do now is let us have a "log viewer" or something so we can tell when its done doing its thing and we can hit resume, also show us, even when in pause, when its indexing and when its not, so when its done we know we can restart it safely, or the better yet, set it to where if Dropbox has to index a large volume of files (say over 100), it will automatically pause all other disk operations until the indexing is complete, then restart the downloads, trying to do both does not work, i know you want it to but it just doesn't, and just causes the whole thing to explode non stop in a loop of death, maybe enable this on a setting somewhere? or auto enable it on machines with over 500k files? something has to be able to be done.

TLDR:

If your Dropbox is stuck indexing, hit pause 30 mins, and let it do its thing until its done, it will keep on doing it even when paused, you wont know if its doing anything or working unless you use procmon, but its working, and try avoid using the hard drive or the machine until its done, (usually less than 30 mins), and your indexing/crashing problem will be fixed. 

Message to Dropbox:

Dear Dropbox, Please give us a way to view this info without having to resort to third party programs, this way we can help auto troubleshoot our Dropbox issues and take a lot of load off your customer service guys.

Something like: Enabling a setting  somewhere saying "activate/enable troubleshoot/server mode" or something that allows us to turn on an always shown (ALWAYS, NOT ONLY WHEN MOUSE OVER, BUT ALWAYS!!!) 3 tab little window,  containing:

Indexing files. (with a current list of the exact files being indexed and their speed (x files per sec)/paths)

Downloading Files.(with a current list of the exact files being downloaded and their speed/paths)

Uploading Files (with a current list of the exact files being uploaded and their speed/paths)

There's another issue with slow uploads due to Dropbox connections stuck in a "stagnant state" (force closing the TCP socket connection using netmon restarts the download/upload and speed goes back up again) but that's another problem for another time.

I hope this was helpful to some other sysadmin and sorry for the long message but it needed explaining.