Problem downloading file using Dropbox-provided URL

[HairyWombat]

Hi

1. I'm developing an app for iOS and Android. Users can enter the url of a file they would like to download. They might choose a Dropbox url like https://www.dropbox.com/s/fzv989hz5mazhqw/ReplacementDatabase.dbf?dl=0. My app does not use Dropbox API and successfully downloads files from other websites but, if my app changes ?dl=0 to ?dl=1 in the url (as suggested by Dropbox docs), the download does not succeed and instead my app receives a small html file saying there is a problem at Dropbox and they will get back to me. Shouldn't this work? It works in a web browser if I make this substitution.

2. If this is not supposed to work, is there a way in the Dropbox API to download a file from a url like https://www.dropbox.com/s/fzv989hz5mazhqw/ReplacementDatabase.dbf?dl=1 ?  That is, when the filename is unknown.

Thanks

Bill

[HairyWombat]

I mean, when the path to the file is unknown.

[DB-Des]

Yes, you should be able to download with ?dl=1.

I was unable to reproduce the reported issue, by using cURL.

Could you provide detailed steps on how to reproduce the issue you are experiencing? For example, the code snippet you are using to reproduce the issue.

[Здравко]

---

@HairyWombat wrote:

... My app does not use Dropbox API and successfully downloads files from other websites but, if my app changes ?dl=0 to ?dl=1 in the url (as suggested by Dropbox docs), the download does not succeed and instead my app receives a small html file saying there is a problem at Dropbox and they will get back to me. Shouldn't this work? ...

---

Hi @HairyWombat,

This should work, if your web client support redirection and the redirection is not forbidden. Dropbox links don't point the target file directly, but make 2 redirects before that. That's why if your client doesn't support redirection, your download is likely to finish in such a way. Make sure redirects are supported properly.

---

@HairyWombat wrote:

..., is there a way in the Dropbox API to download a file from a url like https://www.dropbox.com/s/fzv989hz5mazhqw/ReplacementDatabase.dbf?dl=1 ?  That is, when the filename is unknown.

...

---

Yes, there is such a way. The API access point /2/sharing/get_shared_link_file gives you the file metadata and downloads the content. Using this access point doesn't suppose redirection support and you can provide the link as is (without replacing ?dl=0 to ?dl=1 etc.). The drawback is that application authentication is not yet supported here (i.e. your application has to be user authenticated - explicitly signed in user and OAuth flow - doesn't matter that it's meaningless).

Hope this helps.

[Здравко]

---

@DB-Des wrote:

...

I was unable to reproduce the reported issue, by using cURL.

...

---

Don't let your cURL to redirect (remove the L flag). Does it work correctly still? 😉

[HairyWombat]

Thanks, iMonkia, for the very prompt and helpful response! Love your wolf icon btw.

When I use curl -L, I get 'no matches found'. However this same url downloads correctly in a web browser. You might try it yourself with my sample url https://www.dropbox.com/s/fzv989hz5mazhqw/ReplacementDatabase.dbf?dl=1

The command line transcript is below. I've also attached my C# .NET 7.0.1 code. It is allowing redirection as best I can see.

My test file ReplacementDatabase.dbf is in a Dropbox subdirectory, not at top level, if that matters. Any suggestion appreciated.   🙂

Cheers

Bill

---------------------------------

Last login: Tue Jun 13 11:14:00 on ttys000

bill@Bills-MBP-10 ~ % cd desktop

bill@Bills-MBP-10 desktop % ls Test.dbf

ls: Test.dbf: No such file or directory

bill@Bills-MBP-10 desktop % curl https://billfulton.github.io/ReplacementDatabase.dbf > Test.dbf

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

100  671k  100  671k    0     0  1045k      0 --:--:-- --:--:-- --:--:-- 1064k

bill@Bills-MBP-10 desktop % ls -l Test.dbf

-rw-r--r--  1 bill  staff  687752 13 Jun 11:31 Test.dbf

bill@Bills-MBP-10 desktop % 

bill@Bills-MBP-10 desktop % 

bill@Bills-MBP-10 desktop % curl -L https://www.dropbox.com/s/fzv989hz5mazhqw/ReplacementDatabase.dbf?dl=1 > Test1.dbf

zsh: no matches found: https://www.dropbox.com/s/fzv989hz5mazhqw/ReplacementDatabase.dbf?dl=1

bill@Bills-MBP-10 desktop % curl -L https://www.dropbox.com/s/fzv989hz5mazhqw/ReplacementDatabase.dbf?dl=1

zsh: no matches found: https://www.dropbox.com/s/fzv989hz5mazhqw/ReplacementDatabase.dbf?dl=1

bill@Bills-MBP-10 desktop % curl https://www.dropbox.com/s/fzv989hz5mazhqw/ReplacementDatabase.dbf?dl=1

zsh: no matches found: https://www.dropbox.com/s/fzv989hz5mazhqw/ReplacementDatabase.dbf?dl=1

bill@Bills-MBP-10 desktop % curl -L --max-redirs 20 https://www.dropbox.com/s/fzv989hz5mazhqw/ReplacementDatabase.dbf?dl=1 > Test2.dbf

zsh: no matches found: https://www.dropbox.com/s/fzv989hz5mazhqw/ReplacementDatabase.dbf?dl=1

bill@Bills-MBP-10 desktop % 

---------------------------------

// Initialise httpClient

httpClientHandler = new HttpClientHandler

{

    AllowAutoRedirect = true,// Allegedly the default, but to make sure

    MaxAutomaticRedirections = 20

};

httpClient = new HttpClient ( httpClientHandler );

httpClient.Timeout = TimeSpan.FromSeconds ( 60 );

// Accept all languages

httpClient.DefaultRequestHeaders.AcceptLanguage.Clear();

httpClient.DefaultRequestHeaders.AcceptLanguage.Add ( new System.Net.Http.Headers.StringWithQualityHeaderValue ( "*" ) );

// Accept binary data

httpClient.DefaultRequestHeaders.Accept.Add ( new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue ( "application/octet-stream" ) );

---------------------------------

public static async Task<byte []> ReadWebFileBinaryAsync ( HttpClient httpClient, string webUrl )

// Returns byte array from given file URL

{

    byte [] bytes;

    try

    {

        // Force Dropbox url for testing purposes - to be removed

        webUrl = "https://www.dropbox.com/s/fzv989hz5mazhqw/ReplacementDatabase.dbf?dl=1"

        HttpResponseMessage response = await httpClient.GetAsync ( webUrl );

        using ( MemoryStream memoryStream = new MemoryStream () )

        {

            using ( Stream stream = await response.Content.ReadAsStreamAsync () )

            {

                stream.CopyTo ( memoryStream );

            }

            memoryStream.Seek ( 0, SeekOrigin.Begin );

            bytes = memoryStream.ToArray ();

        }

        return bytes;

    }

    catch ( Exception ex )

    {

        // ...

    }

}

---------------------------------

[HairyWombat]

Further info:

I tried with a file that sin the Dropbox root folder, but it didn't help.

Here is the html string that I am receiving (Error 400):

<!DOCTYPE html>
<html>
<head><meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Dropbox - 400</title>
<link href="https://cfl.dropboxstatic.com/static/metaserver/static/css/error.css" rel="stylesheet" type="text/css"/>
<link rel="shortcut icon" href="https://cfl.dropboxstatic.com/static/images/favicon.ico"/>
</head>
<body>
<div class="figure">
<img src="https://assets.dropbox.com/www/en-us/illustrations/spot/target-miss.svg" alt="Error: 400"/>
</div>
<div id="errorbox">
<h1>Error (400)</h1>Something went wrong. Don't worry, your files are still safe and the Dropbox team has been notified. Check out our <a href="https://status.dropbox.com">Status Page</a> to see if there is a known incident, our <a href="https://www.dropbox.com/help">Help Center</a> and <a href="https://forums.dropbox.com">forums</a> for help, or head back to <a href="https://www.dropbox.com/home">home</a>.
</div>

</body>
</html>

[Здравко]

Hi @HairyWombat,

On my side it works as expected:

curl -L https://www.dropbox.com/s/fzv989hz5mazhqw/ReplacementDatabase.dbf?dl=1 > Test1.dbf
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    17    0    17    0     0      2      0 --:--:--  0:00:06 --:--:--     4
100   363  100   363    0     0     52      0  0:00:06  0:00:06 --:--:--    52
100  671k  100  671k    0     0  68064      0  0:00:10  0:00:10 --:--:--  267k

As expected, there are 3 transactions - 2 redirections and the last one is actual download. You can try verbose output (flag v) for more info:

curl -Lv https://www.dropbox.com/s/fzv989hz5mazhqw/ReplacementDatabase.dbf?dl=1 > Test1.dbf

Here - in the detailed output, you should be able see what's going wrong on your side.

Good luck.

Add: A trick code that works for me:

namespace Dropbox
{
  using System;
  using System.IO;
  using System.Net;
  
   internal class Program
   {
      static void Main(string[] args)
      {
         if (args.Length != 1) {
           Console.WriteLine(
             "Expected one Dropbox URL as a single command parameter!");
           return;
         }
         //"https://www.dropbox.com/s/fzv989hz5mazhqw/ReplacementDatabase.dbf?dl=1";
         string webUrl = args[0];
         var request = WebRequest.Create(webUrl);
         var response = request.GetResponse();
         var ContentDisposition = response.Headers["Content-Disposition"];
         var dispositions = ContentDisposition.Split(';');
         string fileName = "Test.bin"; // default name
         string disposeName = "filename";
         var disposeLength = disposeName.Length;
         for(int i = 0; i < dispositions.Length; ++i) {
           var disposition = dispositions[i].Trim();
           if (string.Compare(disposition, 0,
                              disposeName, 0, disposeLength, true) == 0) {
              if (string.Compare(disposition, disposeLength, "=\"", 0, 2) == 0
                && disposition[disposition.Length-1] == '"')
                fileName = disposition.Substring(disposeLength + 2,
                              disposition.Length - disposeLength - 3);
              else if (string.Compare(disposition, disposeLength,
                                      "*=UTF-8''", 0, 9) == 0)
                fileName = disposition.Substring(disposeLength + 9);
           }
         }
         Console.WriteLine("fileName: " + fileName);
         var ContentLength = response.ContentLength;
         Console.WriteLine("ContentLength: " + ContentLength);
         FileStream fs = new FileStream(fileName, FileMode.Create,
                                        FileAccess.Write);
         int bufLength = Math.Min((int)ContentLength, 0x10000);
         var buffer = new byte[bufLength];
         var readStream = response.GetResponseStream();
         int readed = readStream.Read(buffer, 0, bufLength);
         int consumed = 0;
         while(readed > 0) {
           consumed += readed;
           fs.Write(buffer, 0, readed);
           readed = readStream.Read(buffer, 0, bufLength);
         }
         readStream.Close();
         fs.Close();
         Console.WriteLine("Downloaded: " + consumed);
      }
   }
}

[Greg-DB]

@HairyWombat The "zsh: no matches found" error you're getting is because the URL contains the "?" character which has a special meaning in that shell. To avoid that, quote the URL, like this:

curl -L "https://www.dropbox.com/s/fzv989hz5mazhqw/ReplacementDatabase.dbf?dl=1" > Test1.dbf

I also put together your C# code and tried downloading the file from the shared link with that, and I did successfully get the expected 687752 bytes of the file back.

If you're getting back that HTML 400 error page, that may indicate that the HTTP request was malformed, causing the Dropbox servers to be unable to understand the request. Is there anything on your .NET environment or your network connection itself, such as a proxy, VPN, firewall, etc., that may be interfering with or modifying these requests?

[HairyWombat · Accepted solution]

Hi @Greg-DB

Hey I really appreciate the effort you and the team have put in.

What I found was:

1. Yes, curl works with the url - I switched from Mac to Windows and curl worked. Bash both times.

2. Fiddler also works with the URL.

3. My published code fails using automatic redirect. It works for you - the difference may be that I'm using Microsoft Xamarin on Mac.

4. In the second redirect, Dropbox proposes https://uc012d2e25383f12de25c9db8029.dl.dropboxusercontent.com/cd/0/get/B-Dfp2veRgmAiBVuB2GBdOb2SPrj...    -- which strictly speaking is not a valid url because of the trailing #.

5. I solved the problem by handling the redirects myself and url-encoding the #.

So, Xamarin could do better. But also, Dropbox really should not be sending invalid urls.  🙂

Code below fyi.

Cheers

Bill

----------------------

httpClientHandler = new HttpClientHandler

{

AllowAutoRedirect = false

};

----------------------

try

{

    // Here change ?dl=0 to ?dl=1   *****

    // webUrl = "https://www.howsmyssl.com/a/check";    // Debug - to check TLS version. It is TLS 1.2. (June 2023)

    HttpResponseMessage response = await httpClient.GetAsync ( webUrl );

    int redirectCount = 0;

    while ( redirectCount < 20 )

    {

        if ( response.StatusCode == HttpStatusCode.Redirect )

        {

            string redirectUrl;

            if ( response.Headers != null && response.Headers.Location != null )

            {

                if ( response.Headers.Location.IsAbsoluteUri )

                    redirectUrl = response.Headers.Location.AbsoluteUri;

                else

               {

                    string[] splits = webUrl.Split ('/' );

                    redirectUrl = "https://" + splits [2] + response.Headers.Location.OriginalString;

               }

               redirectUrl = redirectUrl.Replace ( "#", "#".UrlEncode () );

               response = await httpClient.GetAsync ( redirectUrl );

               if ( response.StatusCode != HttpStatusCode.Redirect )

                   break;

            }

            redirectCount++;

        }

    }

    using ( MemoryStream memoryStream = new MemoryStream () )

    {

        using ( Stream stream = await response.Content.ReadAsStreamAsync () )

        {

            stream.CopyTo ( memoryStream );

        }

        memoryStream.Seek ( 0, SeekOrigin.Begin );

        bytes = memoryStream.ToArray ();

    }

    return bytes;

}

catch ( Exception ex )

{

    // ...

    }

}

----------------------