Uploading large HTTP multipart request with System.Net.HttpWebRequest in C#

My previous post described a method of sending a file and some data via HTTP multipart post by constructing the HTTP request with the System.IO.MemoryStream class before writing the contents to the System.Net.HttpWebRequest class. This method of sending our HTTP request will work only if we can restrict the total size of our file and data.

Although the MemoryStream class reduces programming effort, using it to hold a large amount of data will result in a System.OutOfMemoryException being thrown. Hence, to send large amount of data, we will need to write our contents to the HttpWebRequest instance directly. Before doing so, there are several properties in the HttpWebRequest instance that we will need to set.

HttpWebRequest requestToServer = (HttpWebRequest)
    WebRequest.Create("http://localhost/fileReceiver.php");

// Define a boundary string
string boundaryString = "----SomeRandomText";

// Turn off the buffering of data to be written, to prevent
// OutOfMemoryException when sending data
requestToServer.AllowWriteStreamBuffering = false;
// Specify that request is a HTTP post
requestToServer.Method = WebRequestMethods.Http.Post;
// Specify that the content type is a multipart request
requestToServer.ContentType 
    = "multipart/form-data; boundary=" + boundaryString;
// Turn off keep alive
requestToServer.KeepAlive = false;

Calculating the total size of the upload content

Recall that a HTTP multipart post request resembles the following form:

POST http://127.0.0.1/GetPostRequest.php HTTP/1.1
Host: 127.0.0.1
Connection: keep-alive
Referer: http://localhost/GetPostRequest.php
Content-Length: 1611568
Cache-Control: max-age=0
Origin: http://localhost
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.91 Safari/534.30
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryX6nBO7q27yQ1JNbb
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3

------WebKitFormBoundaryX6nBO7q27yQ1JNbb
Content-Disposition: form-data; name="myFileDescription"

 My sample file description.
------WebKitFormBoundaryX6nBO7q27yQ1JNbb
Content-Disposition: form-data; name="myFile"; filename="SomeRandomFile.pdf"
Content-Type: application/pdf

file contents...
------WebKitFormBoundaryX6nBO7q27yQ1JNbb--

From the HTTP request created by the browser, we see that the upload content spans from the first boundary string Β to the last boundary string. The total size of this block of content need to be set to the ContentLength property of the HttpWebRequest instance, before we write any data out to the request stream. To calculate the total size of the HTTP request, we need to add the byte sizes of the string values and the file that we are going to upload. We can convert the strings in the HTTP request into byte arrays with the System.Text.ASCIIEncoding class and get the size of the strings with the Length property of the byte arrays. The size of the file can be retrieved via the Length property of a System.IO.FileInfo instance.

ASCIIEncoding ascii = new ASCIIEncoding();
string boundaryStringLine = "\r\n--" + boundaryString + "\r\n";
byte[] boundaryStringLineBytes = ascii.GetBytes(boundaryStringLine);

string lastBoundaryStringLine = "\r\n--" + boundaryString + "--\r\n";
byte[] lastBoundaryStringLineBytes = ascii.GetBytes(lastBoundaryStringLine);

// Get the byte array of the myFileDescription content disposition
string myFileDescriptionContentDisposition = String.Format(
    "Content-Disposition: form-data; name=\"{0}\"\r\n\r\n{1}",
    "myFileDescription",
    "A sample file description");
byte[] myFileDescriptionContentDispositionBytes 
    = ascii.GetBytes(myFileDescriptionContentDisposition);

string fileUrl = @"C:\SomeRandomFile.pdf";
// Get the byte array of the string part of the myFile content
// disposition
string myFileContentDisposition = String.Format(
    "Content-Disposition: form-data;name=\"{0}\"; "
     + "filename=\"{1}\"\r\nContent-Type: {2}\r\n\r\n",
    "myFile", Path.GetFileName(fileUrl), Path.GetExtension(fileUrl));
byte[] myFileContentDispositionBytes =
    ascii.GetBytes(myFileContentDisposition);

FileInfo fileInfo = new FileInfo(fileUrl);

// Calculate the total size of the HTTP request
long totalRequestBodySize = boundaryStringLineBytes.Length * 2
    + lastBoundaryStringLineBytes.Length
    + myFileDescriptionContentDispositionBytes.Length
    + myFileContentDispositionBytes.Length
    + fileInfo.Length;
// And indicate the value as the HTTP request content length
requestToServer.ContentLength = totalRequestBodySize;

Sending the HTTP request content

After calculating the content length, we can write the byte arrays that we have generated previously to the stream returned via the HttpWebRequest.GetRequestStream() method.

// Write the http request body directly to the server
using (Stream s = requestToServer.GetRequestStream())
{
    // Send the file description content disposition over to the server
    s.Write(boundaryStringLineBytes, 0, boundaryStringLineBytes.Length);
    s.Write(myFileDescriptionContentDisposition , 0,
        myFileDescriptionContentDisposition.Length);

    // Send the file content disposition over to the server
    s.Write(boundaryStringLineBytes, 0, boundaryStringLineBytes.Length);
    s.Write(myFileContentDispositionBytes, 0,
        myFileContentDispositionBytes.Length);

    // Send the file binaries over to the server, in 1024 bytes chunk
    FileStream fileStream = new FileStream(fileUrl, FileMode.Open,
        FileAccess.Read);
    byte[] buffer = new byte[1024];
    int bytesRead = 0;
    while ((bytesRead = fileStream.Read(buffer, 0, buffer.Length)) != 0)
    {
        s.Write(buffer, 0, bytesRead);
    } // end while
    fileStream.Close();

    // Send the last part of the HTTP request body
    s.Write(lastBoundaryStringLineBytes, 0, lastBoundaryStringLineBytes.Length);
} // end using

Getting the response from the server

We get the server response by reading from the System.Net.WebResponse instance, that can be retrieved via the HttpWebRequest.GetResponseStream() method.

// Grab the response from the server. WebException will be thrown
// when a HTTP OK status is not returned
WebResponse response = requestToServer.GetResponse();
StreamReader responseReader = new StreamReader(response.GetResponseStream());
string replyFromServer = responseReader.ReadToEnd();

Related posts

Advertisements

About Clivant

Clivant a.k.a Chai Heng enjoys composing software and building systems to serve people. He owns techcoil.com and hopes that whatever he had written and built so far had benefited people.

10 Comments

  • jeremy
    November 9, 2012 at 10:22 pm

    few things needed to be corrected but great code. thx a lot

    ######################################################
    instead of that:
    string myFile = String.Format(
    “Content-Disposition: form-data;name=\”{0}\”; ”
    + “filename=\”{1}\”\r\nContent-Type: {2}\r\n\r\n”,
    “myFile”, Path.GetFileName(fileUrl), Path.GetExtension(fileUrl));
    byte[] myFileContentDispositionBytes =
    ascii.GetBytes(myFileDescriptionContentDisposition);

    it is that:
    string myFileContentDisposition = String.Format(
    “Content-Disposition: form-data;name=\”{0}\”; ”
    + “filename=\”{1}\”\r\nContent-Type: {2}\r\n\r\n”,
    “myFile”, Path.GetFileName(fileUrl), Path.GetExtension(fileUrl));
    byte[] myFileContentDispositionBytes =
    ascii.GetBytes(myFileContentDisposition);

    ########################################################
    And in the “Sending the HTTP request content” block:
    instead of that:
    s.Write(myFileDescriptionContentDisposition , 0,
    myFileDescriptionContentDisposition.Length);

    it is that:
    s.Write(myFileDescriptionContentDispositionBytes, 0,
    myFileDescriptionContentDispositionBytes.Length);

    • Clivant
      November 12, 2012 at 11:27 am

      Hi Jeremy,

      Thank you for your visit and fixes. πŸ™‚ Had updated the post for the benefit of others.

      • Yougen
        January 22, 2013 at 7:25 am

        Nice sample and thanks for sharing! We have been using same code as your example, it only can upload a single file < 2GB, otherwise the server couldn't find the ending boundary. How large the single file "SomeRandomFile.pdf" could be?

        • Clivant
          January 23, 2013 at 7:24 am

          Hi Yougen,

          Thanks for coming by! πŸ™‚

          Never tried more than 2GB, but I think the code should be able to send more than 2GB if the server write the file bytes to file as it reads from the HTTP multipart request and the server is using a long to store the content length. A signed int can only store up to 2 ^ 31 = 2147483648 bytes.

          Some workarounds could be compressing your file before you send it out to the server or “chopping” the files into smaller sizes and having the server piece them back when it receives them.

          • Yougen
            January 28, 2013 at 5:47 am

            Thanks Clivant! It is the way to handle large file upload through HTTP request as you and I both thought. The file we upload to server is always in zip file, App server will unzip it. There is an Apache server between client and App server, it is running on a 64-bit Linux OS box, according the Apache 2.2 release document http://httpd.apache.org/docs/2.2/new_features_2_2.html, the large file (>2GB) has been resolved on 32-bit Unix box, but it didn’t mention the same fix in Linux box, however there is a directive called EnableSendfile discussed http://demon.yekt.com/manual/mod/core.html, someone has it turned off and that resolves the large file upload issue, we tried and App server still couldn’t find the ending boundary.

          • Clivant
            January 28, 2013 at 10:30 am

            You are welcomed Yougen. πŸ™‚

            Sounds like it is the app server’s end that need tweaking.

            Hope u can resolve your app server problem soon!

  • Mat
    August 23, 2013 at 10:47 pm

    Hi Clivant,

    Very useful post. One question -why do you set the keep alive to false here?

    • Clivant
      August 23, 2013 at 11:31 pm

      Hi Mat,

      Thanks for dropping by. πŸ™‚

      I guess I had left keep alive to false because I was not trying to send multiple requests with the same HttpWebRequest instance.

  • Sara
    July 3, 2014 at 3:57 pm

    Hey, just to inform you that the following link:
    “Downloading a file from a HTTP server with System.Net.HttpWebRequest in C#”, doesn’t work. Tnx!

    • Clivant
      July 8, 2014 at 11:06 am

      Hi Sara,

      Thanks for dropping by with the update. I had updated the link accordingly. πŸ™‚

Advertisements
Advertisements