CSES problemset editorial

Javascript Uploads & Downloads

GSoC'20 Final Evaluation Report 

GSoC[chapel].insert (@ankingcodes)

### Problem Statement

Given two files of MIME type, PDF with sizes 18MB and 3MB, we want to upload to and download from a service using Javascript. The service accepts small files (<3MB) in base64 format and upload is straight forward. However, for large files (>3MB), we are required to create an upload session, break the file (18MB) into smaller chunks (maybe 2-3MB) and then upload each chunk "in binary format" via the session.

Want a real-life example ? Check out [Microsoft Graph API](https://docs.microsoft.com/en-us/graph/outlook-large-attachments?tabs=http).

### Solution

The use-case I chose to cover is using Microsoft Graph API for Outlook, but this solution will help for any other service that you might intend to use. 

#### Uploading Small File

Uploading the small file (3 MB) is straightforward. First add an HTML input element to select the files. This input shall pick both the files.

```html
<input style="display: none;" type="file" id="uploadattachments" multiple="multiple" />
```
We can store the file JSON objects in two arrays, `smallAttachments` and `largeAttachments`. 
Then we iterate on the `smallAttachments` array, read each file using [FileReader API](https://developer.mozilla.org/en-US/docs/Web/API/FileReader), convert bytes to base64String, create the Microsoft Outlook Attachment object and make a POST request to upload the file to the message.


```js
const getBase64Content = file => {
  return new Promise((resolve, reject) => {
    let reader = new FileReader();
    reader.onload = e => {
      let bytes = Array.from(new Uint8Array(e.target.result));
      let base64String = btoa(bytes.map(byte => String.fromCharCode(byte)).join(" "));
      resolve({ base64String: base64String });
    }
    reader.onerror = reject;
    reader.readAsArrayBuffer(file);
  })
}

const smallAttachmentUploadObjArr = 
Array.from(smallAttachments).map(async file => {
  const base64Content = await getBase64Content(file);
  if (base64Content) {
    return ({
      "@odata.type": "#microsoft.graph.fileAttachment",
      name: file.name,
      contentType: file.type,
      contentBytes: base64Content.base64String,
    });
  }
})

const uploadSmallAttachments = msgId => {
  smallAttachmentUploadObjArr.forEach(async attachment => {
    const attachmentUploadRes = 
      await graphClient.api(`/me/messages/${msgId}/attachments`)
            .post(attachment);
    if (attachmentUploadRes) 
      console.log("Small attachment upload successful");
    else console.log("Small attachment upload failed");
                                  
  })
}
```


#### Uploading Large File

The file should be uploaded successfully after this. Now what about the large file (> 3MB). The file has to be divided into smaller chunks of byteranges and uploaded "in binary format". Most people are confused what binary format means in Javascript. Binary Format is nothing but the raw bytes of the file content. Basically, we use the FileReader to get an ArrayBuffer containing the file content, which is then placed into an Uint8Array Typed array. Then, we divide this Uint8Array into small chunks with an offset and upload those chunks one after the other.


```js
const getBinaryContent = file => {
  return new Promise((resolve, reject) => {
    let reader = new FileReader();
    reader.onload = e => {
      const bytes = Array.from(new Uint8Array(e.target.result));
      resolve({ byteArray: bytes });
    }
    reader.onerror = reject;
    reader.readAsArrayBuffer(file);
  })
}

const largeAttachmentUploadObjArr = 
  Array.from(largeAttachments).map(async file => {
    const attachmentBytes = await getBinaryContent(file);
    if (attachmentBytes) {
      return ({
        name: file.name,
        size: file.size,
        type: file.type,
        contentBytes: attachmentBytes.byteArray
      });
    }
  })

const uploadLargeAttachments = msgId => {
  largeAttachmentUploadObjArr.forEach(async attachment => {
    let uploadBinaryData = attachment.contentBytes;
    let attachmentItemObject = {
      AttachmentItem: {
        attachmentType: "file",
        name: file.name,
        size: uploadBinaryData.length
      }
    };
    /* Step 1: Create session & retrieve URL */
    let createSession = 
      await graphClient.api(`/me/messages/${msgId}/attachments/createUploadSession`)
      .post(attachmentItemObject);

    const uploadUrl = createSession.uploadUrl;
    const offset = 2097152; // 2MB
    const fullsize = uploadBinaryData.length;
    let beg = 0, end = offset - 1;
    let config, uploadData, uploadReq;
    
    /* Step 2-3: Upload small chunks of byte ranges */
    while (beg <= end && end <= fullsize-1) {
      uploadData = uploadBinaryData.slice(beg, end-1);
      config = {
        headers: {
          "Content-Type": "application/octet-stream",
          "Content-Range": `bytes ${beg}-${end}/${fullsize}`
        }
      }
      uploadReq = await axios.put(uploadUrl, uploadData, config);
      beg = end + 1;
      end = end + offset;
    }

    /* Step 4: Upload final byte range */
    end = end - offset + 1;
    uploadData = file.contentBytes.slice(end, fullsize);
    config = {
      headers: {
        "Content-Type": "application/octet-stream",
        "Content-Range": `bytes ${end}-${fullsize-1}/${fullsize}`
      }
    }
    uploadReq = await axios.put(uploadUrl, uploadData, config);
    if (uploadReq.status === 201) 
      console.log("Large Attachment Upload Successful");
    else console.log("Large Attachment Upload Failed");
  })
}
```

This should upload the large attachment successfully. 

#### Downloading small and large files

We require a HTML Anchor Element, which when clicked by an user, will initiate a file download. Also, let us consider we have an attachment list array, which contains file name, file size, file contents and other metadata.



```js
const attachmentList = getAttachmentList();
const attachmentListDOM = document.getElementById("AttachmentList");

attachmentList.value.forEach((attachment) => {
  const binaryArr = base64ToArrayBuffer(attachment.contentBytes);
  const blob = new Blob([binaryArr], { type: attachment.contentType });
  const fileNameElement = document.createElement('a');
  fileNameElement.textContent = attachment.name;
  fileNameElement.download = attachment.name;
  fileNameElement.href = window.URL.createObjectURL(blob);
  attachmentListDOM.appendChild(fileNameElement);
  attachmentListDOM.appendChild(document.createElement("br"));
})

const base64ToArrayBuffer = base64 => {
  const binaryString = window.atob(base64);
  const binaryLen = binaryString.length;
  const bytes = new Uint8Array(binaryLen);
  binaryString.forEach(bit => bytes.push(String.fromCharCode(bit)));
  return bytes;
}
```


The following section explains some concepts revolving around the problem statement.

#### ArrayBuffers, Uint8Arrays, Blobs, Arrays

- The `ArrayBuffer` object represents generic, fixed length raw binary data buffer. It's similar to byte array and this buffer can not be manipulated. It contains "raw binary data". If you read a file with 2MB of data, it shall take up 2MB contiguous space in ArrayBuffer. 
- The `Uint8Array` is a typed array. It represents 8 bit unsigned integers. The ArrayBuffer is fed to an Uint8Array, to store data in binary format, since binary digits have 8 bits of 0s and 1s. The number of bytes are same as 2MB.
- The `Blob` object represents a blob, which is a file-like object of immutable, raw data; they can be read as text or binary data, or converted into a ReadableStream so its methods can be used for processing the data. This is similar to ArrayBuffer.
- The `Array` is a javascript list like object. The length of the array is not fixed. Data can be stored in non-contiguous locations in the array. When I tried binding the `Uint8Array` using `Array.from`, and added to request body while uploading, the content length increased (> 2MB). FYI, content length is automatically generated by Axios or Fetch API.

### References
- https://developer.mozilla.org/en-US/docs/Web/API/Blob
- https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/Sending_and_Receiving_Binary_Data 
- https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Uint8Array
- https://stackoverflow.com/questions/3195865/converting-byte-array-to-string-in-javascript

## In real open source, you have the right to control your own destiny

*by Linus Torvalds*

#### 21 January 2020

 I had already won NWOC'19 and one of my closest mentors there texted me :
_"You're very creative and strong in Javascript. Go for an org with full-stack web development projects, and you'll definitely get selected in GSoC"_. 
I had two choices for GSoC - go for a project I am comfortable to work with(the *easy* choice) or go for a project which would challenge me and am more likely to fail(the *hard* choice). 

*I am glad that I chose the latter.* 

I had opened the Chapel repository 5 times earlier and was intimidated by the huge codebase. I joined the gitter channels and enjoyed reading the long discussions amongst the community members, although I couldn't understand most of the things. Reading through the "Getting started" guide, I started to make a simple project using Mason. I typed `mkdir new`, `cd new` and then `mason init`, out of habit, being a javascript developer(using `npm`) and got an error. And there it was, my first idea to contribute to Chapel, a `mason init` command to initialise a Mason project.

#### 26 February 2020
I worked with [Ben Albrecht](https://www.github.com/ben-albrecht) and finally got `mason init` merged ! Ben had been very patient and helpful to me and that's one of the primary reasons I was motivated to work on Chapel everyday after a tiring college day. I made multiple new feature requests, and had also found a long list of pending tasks for Mason. Today, again, I had two choices for GSoC - go for a project from the ideas list of Chapel, or present my own project idea and risk getting selected in GSoC. 

*Once again I'm glad I chose the latter*. 

#### 16 March - 31 March 2020 (Application period)
I made multiple PRs on small to intermediate tasks and created new issues I found on the way. I had also presented a doc with my project idea and started discussing it with Ben. Ben had also invited [Sam Partee](https://www.github.com/spartee) (author of Mason), to take a look at my idea. Both of them were equally excited about the idea and encouraged me to go forward with the proposal. I was extremely happy with my proposal because I had no clue about majority of the tasks I had included in it. I knew I am going to learn a lot in the summer. 

#### 4 May, 2020 - 1 June, 2020 (Community Bonding Period)
My [project](https://summerofcode.withgoogle.com/projects/#6454194829000704) "Improve Mason Package Manager" mentored by Ben Albrecht, Sam Partee and Krishna Kumar Dey was selected in GSOC !! Being from a small university where students pay for an internship certificate, and have absolutely no ecosystem to code or to be a good engineer, I felt it was time things change. I let my closest friends know about my achievement and they couldn't believe it. 

Mostly, students get a head start on their project during the community bonding period. But I spent the month, taking up issues outside my proposal. One of the major issues was "Improving `mason external --setup`". This command sets up the Spack package repository, to allow users to use dependencies from Spack in their project. However, installing it took an extremely long time, about 15-20 minutes. 

After some design discussions with Ben, and a PR, Spack now installs within 9-10 seconds which was a very shocking outcome for me. The trick was that we did a "shallow clone" of the Spack repository, which would get only the latest revision of the Spack repository stripping off the entire history of commits. 

Also, I got to attend CHIUW 2020, which was fortunately online due to Covid-19 situation. Surprisingly, my work was also recognized by [Brad Chamberlain](https://www.github.com/bradcray)(core designer, creator of Chapel) during his introductory talk.

## Conclusion
I am grateful to have been given this opportunity to work with Chapel in Google Summer of Code 2020. I congratulate my fellow GSoCers [Rahul](https://www.github.com/rahulghangas), [Aniket](https://www.github.com/Aniket21mathur) and [Yujia](https://www.github.com/Rapiz1) for getting selected, and wish them good luck to deliver the tasks proposed by them successfully. Most importantly, I hope everyone grow their skills during the summer, and continue to contribute to Chapel post GSoC. 
 


A brief recap of pre-GSoC work and Community Bonding Period

Chronicles. Buzz. Playbooks.

Featured Articles

Javascript Uploads & Downloads

GSoC[chapel].insert (@ankingcodes)

All Articles