a

Javascript Uploads & Downloads

• 0 views

Problem Statement

Given two files of MIME type, PDF with sizes 18MB and 3MB, we want to upload to and download from a service using Javascript. The service accepts small files (<3MB) in base64 format and upload is straight forward. However, for large files (>3MB), we are required to create an upload session, break the file (18MB) into smaller chunks (maybe 2-3MB) and then upload each chunk "in binary format" via the session.

Want a real-life example ? Check out Microsoft Graph API.

Solution

The use-case I chose to cover is using Microsoft Graph API for Outlook, but this solution will help for any other service that you might intend to use.

Uploading Small File

Uploading the small file (3 MB) is straightforward. First add an HTML input element to select the files. This input shall pick both the files.

<input style="display: none;" type="file" id="uploadattachments" multiple="multiple" />

We can store the file JSON objects in two arrays, smallAttachments and largeAttachments. Then we iterate on the smallAttachments array, read each file using FileReader API, convert bytes to base64String, create the Microsoft Outlook Attachment object and make a POST request to upload the file to the message.

const getBase64Content = file => {
  return new Promise((resolve, reject) => {
    let reader = new FileReader();
    reader.onload = e => {
      let bytes = Array.from(new Uint8Array(e.target.result));
      let base64String = btoa(bytes.map(byte => String.fromCharCode(byte)).join(" "));
      resolve({ base64String: base64String });
    }
    reader.onerror = reject;
    reader.readAsArrayBuffer(file);
  })
}

const smallAttachmentUploadObjArr = 
Array.from(smallAttachments).map(async file => {
  const base64Content = await getBase64Content(file);
  if (base64Content) {
    return ({
      "@odata.type": "#microsoft.graph.fileAttachment",
      name: file.name,
      contentType: file.type,
      contentBytes: base64Content.base64String,
    });
  }
})

const uploadSmallAttachments = msgId => {
  smallAttachmentUploadObjArr.forEach(async attachment => {
    const attachmentUploadRes = 
      await graphClient.api(`/me/messages/${msgId}/attachments`)
            .post(attachment);
    if (attachmentUploadRes) 
      console.log("Small attachment upload successful");
    else console.log("Small attachment upload failed");
                                  
  })
}

Uploading Large File

The file should be uploaded successfully after this. Now what about the large file (> 3MB). The file has to be divided into smaller chunks of byteranges and uploaded "in binary format". Most people are confused what binary format means in Javascript. Binary Format is nothing but the raw bytes of the file content. Basically, we use the FileReader to get an ArrayBuffer containing the file content, which is then placed into an Uint8Array Typed array. Then, we divide this Uint8Array into small chunks with an offset and upload those chunks one after the other.

const getBinaryContent = file => {
  return new Promise((resolve, reject) => {
    let reader = new FileReader();
    reader.onload = e => {
      const bytes = Array.from(new Uint8Array(e.target.result));
      resolve({ byteArray: bytes });
    }
    reader.onerror = reject;
    reader.readAsArrayBuffer(file);
  })
}

const largeAttachmentUploadObjArr = 
  Array.from(largeAttachments).map(async file => {
    const attachmentBytes = await getBinaryContent(file);
    if (attachmentBytes) {
      return ({
        name: file.name,
        size: file.size,
        type: file.type,
        contentBytes: attachmentBytes.byteArray
      });
    }
  })

const uploadLargeAttachments = msgId => {
  largeAttachmentUploadObjArr.forEach(async attachment => {
    let uploadBinaryData = attachment.contentBytes;
    let attachmentItemObject = {
      AttachmentItem: {
        attachmentType: "file",
        name: file.name,
        size: uploadBinaryData.length
      }
    };
    /* Step 1: Create session & retrieve URL */
    let createSession = 
      await graphClient.api(`/me/messages/${msgId}/attachments/createUploadSession`)
      .post(attachmentItemObject);

    const uploadUrl = createSession.uploadUrl;
    const offset = 2097152; // 2MB
    const fullsize = uploadBinaryData.length;
    let beg = 0, end = offset - 1;
    let config, uploadData, uploadReq;
    
    /* Step 2-3: Upload small chunks of byte ranges */
    while (beg <= end && end <= fullsize-1) {
      uploadData = uploadBinaryData.slice(beg, end-1);
      config = {
        headers: {
          "Content-Type": "application/octet-stream",
          "Content-Range": `bytes ${beg}-${end}/${fullsize}`
        }
      }
      uploadReq = await axios.put(uploadUrl, uploadData, config);
      beg = end + 1;
      end = end + offset;
    }

    /* Step 4: Upload final byte range */
    end = end - offset + 1;
    uploadData = file.contentBytes.slice(end, fullsize);
    config = {
      headers: {
        "Content-Type": "application/octet-stream",
        "Content-Range": `bytes ${end}-${fullsize-1}/${fullsize}`
      }
    }
    uploadReq = await axios.put(uploadUrl, uploadData, config);
    if (uploadReq.status === 201) 
      console.log("Large Attachment Upload Successful");
    else console.log("Large Attachment Upload Failed");
  })
}

This should upload the large attachment successfully.

Downloading small and large files

We require a HTML Anchor Element, which when clicked by an user, will initiate a file download. Also, let us consider we have an attachment list array, which contains file name, file size, file contents and other metadata.

const attachmentList = getAttachmentList();
const attachmentListDOM = document.getElementById("AttachmentList");

attachmentList.value.forEach((attachment) => {
  const binaryArr = base64ToArrayBuffer(attachment.contentBytes);
  const blob = new Blob([binaryArr], { type: attachment.contentType });
  const fileNameElement = document.createElement('a');
  fileNameElement.textContent = attachment.name;
  fileNameElement.download = attachment.name;
  fileNameElement.href = window.URL.createObjectURL(blob);
  attachmentListDOM.appendChild(fileNameElement);
  attachmentListDOM.appendChild(document.createElement("br"));
})

const base64ToArrayBuffer = base64 => {
  const binaryString = window.atob(base64);
  const binaryLen = binaryString.length;
  const bytes = new Uint8Array(binaryLen);
  binaryString.forEach(bit => bytes.push(String.fromCharCode(bit)));
  return bytes;
}

The following section explains some concepts revolving around the problem statement.

ArrayBuffers, Uint8Arrays, Blobs, Arrays

  • The ArrayBuffer object represents generic, fixed length raw binary data buffer. It's similar to byte array and this buffer can not be manipulated. It contains "raw binary data". If you read a file with 2MB of data, it shall take up 2MB contiguous space in ArrayBuffer.
  • The Uint8Array is a typed array. It represents 8 bit unsigned integers. The ArrayBuffer is fed to an Uint8Array, to store data in binary format, since binary digits have 8 bits of 0s and 1s. The number of bytes are same as 2MB.
  • The Blob object represents a blob, which is a file-like object of immutable, raw data; they can be read as text or binary data, or converted into a ReadableStream so its methods can be used for processing the data. This is similar to ArrayBuffer.
  • The Array is a javascript list like object. The length of the array is not fixed. Data can be stored in non-contiguous locations in the array. When I tried binding the Uint8Array using Array.from, and added to request body while uploading, the content length increased (> 2MB). FYI, content length is automatically generated by Axios or Fetch API.

References

  • https://developer.mozilla.org/en-US/docs/Web/API/Blob
  • https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/Sending_and_Receiving_Binary_Data
  • https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Uint8Array
  • https://stackoverflow.com/questions/3195865/converting-byte-array-to-string-in-javascript