How to make Dropbox ignore node_modules folder with symbolic links (aliases)

[sebastian_]

PROBLEM

Dropbox does not currently provide an ignore file/folder feature that enables a locally-unique instance to exist on each machine. One important use of this is the “node_moduels" folder used for node javascript development. The node_modules folder sees thousands of file changes during the development cycle. If this folder is allowed to sync, the important files you need (outside of it), are so far down in the sync queue that you never have the important files when you need them on your other systems.

SOLUTION

A better way to sync projects with Dropbox while ignoring the node_modules folder is to use a “properly” formed symlink (alias). At this time npm and yarn support a property formed symlink. HOWEVER, IT IS CRITICAL THAT THE LINK (ALIAS) AND TARGET HAVE THE SAME NAME!

From shell prompt (terminal):

1. cd YOUR_PROJECT_NAME
2. mkdir -p ~/Downloads/node_modules_for_YOUR_PROJECT_NAME
3. mv node_modules ~/Downloads/node_modules_for_YOUR_PROJECT_NAME
4. ln -s ~/Downloads/node_modules_for_YOUR_PROJECT_NAME/node_modules

Now create the same target node_moduels folder on each of your other systems **AFTER** Dropbox syncs the ENTIRE project direcotry. The old node_modules folder should be removed and the symlink (alias) added before running these steps.

1. cd YOUR_PROJECT_NAME
2. mkdir -p ~/Downloads/node_modules_for_YOUR_PROJECT_NAME/node_modules
3. yarn install

SUCCESS

Dropbox WILL sync the symlink/alias file itself, but NOT the contents it points to. This seems to work well for file syncing across several of your own development systems, and then use git to push to the repo for others on the team.

CAVEAT

The tiny symlink (alias) file itself IS sync’d on all systems, but will point to a non-existent / empty folder on the other systems. This is most applicable for a single user, but can also be useful for teams if an individual variant of the ignored data is appropriate, such as for node_moduels.

Hope this helps.

🙂

..sebastian

[miromannino]

Thank you for posting this idea!

The only thing is... where is this working? In OSX Dropbox is still syncing the symlinks.

[sebastian_ · Accepted solution]

NOTE: The Initial Post on this thread IS the Solution. While the technique specifically targets the node_modules folder in Javascript / TypeScript projects, it can also be applied to any "folders" that need to be ignored (see bottom of this post for a drag-n-drop technique).

HOW IT WORKS

Because Dropbox does not synchronize folders referened by symlinks (aliases), you can move those folders into a NON-DROPBOX path (outside of dropbox), then create a symlink (alias) in the original Dropbox location.

RESULT

The ignored folder in Dropbox looks and behaves as normal folder, but in reality the data lives in a non-synchronized location on the storage device.

HOW TO IGNORE ANY FOLDER (THE DRAG-N-DROP WAY)

1. Move the folder to ignore "Outside of Dropbox" into another location (e.g. ~/Documents)
2. In Finder (Mac OS), Command+Option Drag the folder back into Dropbox to create an alias. The cursor should turn into a little curved arrow while dragging if you have the right keys pressed. Once dropped, an alias will be created in the original Dropbox location pointing to the moved folder.

NOTE: Using symlinks/aliases on Windows and Linux should behave the same way, but I have not verified it. Also, I've updated the original post to state that the tiny symlink (alias) file itself "IS" copied to all systems, but not the content of the folder.

..sebastian

[sebastian_]

I received a response from Tech Support explaining how to use the xattr command in terminal to ignore a file/folder (link below).

There is a an upside and a downside to each approach (xattr and symlink). The xattr method ignores the target file/folder content and does not upload it, HOWEVER, upon EVERY change within node_modules, Dropbox reindexes and pegs the processor for long periods. The symlink method avoids indexing entirely, so it has a zero-impact when changes within node_modules occur. The symlink/alias file may point to a non-existent folder on a system that doesn't need it, so it's ideal for node_modules, but may not for all others.

You can find the xattr ignore Dropbox article here, plus ongoing conversation about the ignore topic here (pages 87, 88, etc...). Hopefully the upcoming xattr ignore method (currently in beta) will support a hard-ignore without reindexing before it's released.

[lkljsjjjsjj]

Are you a god?

I've been searching for a solution for several years! I wouldn't expect it to be as simple as a symlink. Kudos!

[Arne1]

It does not work for me (on Windows). The first npm install is fine, the second run will basically delete the symlink, recreate the directory, add some packages and then fail with a file not found error 😞

[sebastian_]

@lkljsjjjsjj, Glad to hear it works for you!! I too pulled my hair out for years before figuring this configuration out.

@Arne1, Sorry to hear it isn't easily translating to Windows. Is the "second run" failure happening on the same system, or on a secondary system? I no longer have access to a Window platform or I'd be happy to troubleshoot it.

1. Windows Secondary System: “Theoretically", Dropbox should sync the symlink/alias file itself to all clients subscribed to the same Dropbox/share. Once the symlink/alias file itself is sync'd to your other system(s), you need to create the exact same target folder path (e.g. "C:\users\your_username\downloads\node_modules_for_projectname\node_modules").
2. Windows Secondary Run (on same system): "Theoretically" NPM should behave the same, regardless of platform, but maybe not. Have you tried yarn? What do the commands do after creating the node_modules symlink=>downloads_folder?

• yarn install
• yarn upgrade --latest

Unfortunately, I do not know the behavior of symlinks in cross-platform implementations (e.g. Mac Dropbox syncing w/ Windows and/or Linux Dropboxes).

** Can someone working with NPM on Windows help with a “Working and Tested” set of steps for Windows and post back? **

[Arne1]

Hi,

Thanks for the reply! It is the same machine. 

It seems to work with yarn though, but I can't switch easily since other people work with the code (and lock file) too.

[Arne1]

After switching to yarn, I've noticed that composer (the PHP package manager) also doesn't like if its directory ("vendor" in this case) is a symlink. 

So at the moment, this trick does not work with NPM or composer.

[sebastian_]

@Arne1 , That's a bummer. I’ve definitely seen various commands on the toolchain not respecting symlinks and takes a bit of effort to figure it out. You might consider posting a question to the tool's project page asking for help getting symlinks working. If/When you find a workaround for your toolchain, please post it back here to help the rest of the developer community.

Until we have a .dropboxignore (a “true ignore”, without reindexing), sharing workarounds (hacks) are the only course of action. If a .dropboxignore feature existed, I’d put far more of my projects into Dropbox.