You might see that the Dropbox Community team have been busy working on some major updates to the Community itself! So, here is some info on what’s changed, what’s staying the same and what you can expect from the Dropbox Community overall.
Forum Discussion
TheFlodge
4 years agoHelpful | Level 6
Dropbox CLI support is atrocious.
I am not a fan of the uploading tools available via API (evidently the only option for CLI users). I'm writing this after bashing my head against them for a while, so I'm having to work hard to restrain my frustration. The REAL TL;DR of this post is "Why isn't FTP or SCP supported for a FILE STORAGE solution?", but let's pretend it's "API Upload tools are convoluted, opaque, and poorly documented".
----- Disclaimer: I searched high and low for alternate solutions for CLI users and came up dry, but if there IS a more appropriate option, then perhaps my problem is using the wrong solution all along - I'm certainly willing to entertain that possibility, so if there's a better way to do this, please feel free to let me know. -----
- You know what might be helpful? Something in the documentation (or anywhere) that says "hey, you're going to need to chop up anything you want to upload into 150MB chunks beforehand". The current explanation of "Upload sessions allow you to upload a single file in one or more requests" led me to believe the exact opposite. In fact, the word "chunk" appears exactly once in the entire documentation: "For concurrent upload sessions, only chunks with size multiples of 4194304 bytes can be uploaded." The words "segment" or "partial" do not appear at all, and the word "split" only appears in reference to docs - "The newly created Paper doc would be too large. Please split the content into multiple docs" - Gee, it seems like this might be a helpful message in regards to FILE uploads too, maybe you should include that in the UPLOAD documentation for a FILE STORAGE service. Hell, I'm STILL not even totally sure if uploading the file in chunks is the right way to do this, the documentation tells me NOTHING, and there is no useful error output. In fact, there's not really error output of ANY kind (see next point).
- You know what ELSE might be helpful? If someone goes over that limit, returning an error telling them "hey, that's over the limit", instead of just returning HTML for a generic 404 page (sorry, it's actually not even that specific - it's a generic 4xx page, though it does helpfully include "We can't find the page you're looking for" if you comb through the HTML response to your POST). Returning a generic 4xx response might cause that person to waste a bunch of time trying to figure out what's up with their syntax, or cause them to wonder why on earth this URL seems to work for file1.mp4 but returns a 4xx for file2.mp4, which might piss that person off a whole lot.
Chopping up my entire library into 150MB chunks just to upload it sure seems like a major pain in the ass. Dropbox is a file storage service - you'd think it wouldn't require me to chunk every single large file I want to upload (I'm usually working with video, which often has filesizes over 150MB) just to utilize the most basic service it provides. Now, that may be related to the fact that uploads are done via HTTP POST - it's almost like there should be some kind of protocol JUST for file transfers, like a ... "file transfer protocol" or something (crazy idea, I know). The only reason I'm even screwing with the API is that I have some files on an old headless VPS that I want to store before wiping and repurposing the box - I don't have a ton of extra space, which seems like it might be a fairly common situation for people transferring files away to the cloud. So imagine I'm down to just 5GB of free space, so I need to store some assets. Unfortunately, all of my assets are 6GB or larger - how can I split them without running out of space, since splitting a file basically doubles the usage? Or, to look at it another way, let's say I have a particularly large file, say 200GB. In order to upload that file, I'll need to split it into 1334 150MB chunks which, already not loving THAT, but more importantly, that's another 200GB of space I had to use to do it, unless I use a method that destroys the original file while splitting it, which seems pointlessly risky.
----------
I can appreciate that some of these difficulties are inherent to uploading via HTTP - there's not really a defined standard for large file transfer over HTTP, as far as the RFCs are concerned, so if your workflow/application requires you to use HTTP, these are pains you have to live with. What I can't figure out, however, is why a FILE STORAGE service doesn't support FILE TRANSFER protocol as an upload method (or even SCP). Instead of "negotiate connection, send upload, disconnect", I have to do "split into hundreds or THOUSANDS of files locally, POST to start session, 1334 separate POSTs to upload 150MB at a time, POST to finish session and define storage location". Sure, I can write a Python script to do all that (which is what I HAVE done), but my point is that I shouldn't need to do that just to upload files from CLI to Dropbox. It certainly makes this an AWFUL business solution to any company that uses CLI for their web assets (in other words, basically everyone). I don't understand why a file storage company would make it so hard to store files, that seems like it should be the number one function, and ease of use plus available features should be the first priority - you don't even offer an error message for files over the size limit, just a generic 4xx page! Madness. Even a SUCCESSFUL "/upload_session/append" returns "null" instead of any useful information!
I searched for a long time for alternate solutions or a guide of some kind, certain that I must just be missing something obvious, that a company that lives or dies based on the ability to upload and download files wouldn't make this so pointlessly opaque, but all the results I encountered were similar to this forum post (which, hilariously, is marked as "solved").
The answer, and overall attitude here, seems to be "we don't support using Dropbox from CLI". I guess that's your choice, as a business, but leaving a HUGE chunk of your market out in the cold doesn't seem like a very intelligent business strategy to me. You have a "Dropbox Business" offering, but even THAT is GUI based. This may surprise you, but most businesses don't manage their content solely via GUI - basically any company running a web server of any kind uses CLI on at least SOME level, even if they're using mostly cloud solutions.
The end result of this little journey is that I'll be cancelling my account and finding a file storage solution that supports both CLI and GUI - offering your services to just one of these groups means you're ignoring a huge chunk of your market, and since the health of the company that's storing my files is important to me (so they don't go belly up and require me to migrate or just shut the servers off one day due to insolvency), and I evidently can't trust your company to make intelligent business decisions regarding the ONE SERVICE it provides, I'll need to find a company that doesn't alienate/ignore a major portion of their market.
----------
Useful things you could do today, right now, to help:
- Update your documentation to give better direction on how to upload large files. Some usage of the words "chunk", "segment", or "split" might be helpful here.
- Change the way you handle POSTs above the size limit - instead of returning a generic 4xx HTML page, how about a JSON format error message, like you do for a bad token? Something like what you use for "/paper/create": "doc_length_exceeded Void The Paper doc would be too large, split the content into multiple docs." You have this for docs, why on earth do you not have it for FILES??
- Change the response message for SUCCESSFUL append requests - right now they just kick back "null", which is aggressively unhelpful. Maybe put something like "here's the offset you should use for the next request" or something, or even just "ok", returning two characters would be better than what you're doing now.
- Consider linking your params to the definitions, or even providing a generalized info link like you do for "Request and Response formats" - for example, I see "Dropbox-API-Arg" used a lot, but where is it defined? What arguments can it accept?
- Update your documentation to describe use cases for each call. For example, look at "/create": "Creates a file request for this user." How does this differ from "/upload"? The syntax sure looks similar, what's the difference between uploading a file and creating a file request? Another example - "Dropbox-API-Arg" is passed in a header in the examples, but only after digging around did I discover that it can also be passed as a URL parameter (I'm assuming this means QS param): "...passed as JSON in the Dropbox-API-Arg request header or arg URL parameter" - that is incredibly useful information, but 99% of users will never know because it's buried so deep it might as well be a comment in the page's source code!
Something I've done in the past when designing APIs and handling the documentation for them is to have someone not familiar with my service come in and try to use them, instead of having my devs decide what info is and isn't relevant - it's like using cURL, if you're familiar with cURL, you don't need much info to learn how to do a POST, for example, instead of a GET, however if you've never used cURL before, you'd need more context to understand how and why it does what it does. Hire a documentation specialist to use your API and write docs that are actually useful for people OUTSIDE of your dev team.
Alternately, offer support for FTP or SCP so people can interact with your service like a normal file repository instead of trying to FORCE everyone to use GUI. We get it, you built a cool GUI and you want people to use it, but some people have use cases where that's not practical, and you are alienating everyone in that group, which includes some lucrative customer segments. Offering support for CLI users can and will generate additional revenue - failing to do so is just leaving money on the table.
- Greg-DBDropbox Staff
Thanks for writing up and sharing all of this feedback! I'm sending this along to the relevant people.
About Dropbox API Support & Feedback
Find help with the Dropbox API from other developers.
5,915 PostsLatest Activity: 5 hours agoIf you need more help you can view your support options (expected response time for an email or ticket is 24 hours), or contact us on X or Facebook.
For more info on available support options for your Dropbox plan, see this article.
If you found the answer to your question in this Community thread, please 'like' the post to say thanks and to let us know it was useful!