Uploading Files to Object Storage using Multipart in VBCS

Object Storage has become a go to alternative to store large amounts of data, hence uploading huge chunks of file has become a common requirement in Cloud. Anyone who has tried to upload a huge file to server knows the drawbacks and shortcomings of such approach, this is where the MultiPart Upload approach comes in handy.

Below I try to explain the steps to achieve the same

MultiPart upload to Object Storage is done in 3 phases

Phase 1: Create a multi part upload request

Phase 2: Upload Parts

Phase 3: Commit Upload

Note: Before you start building the VBCS application, make sure your object storage instance is setup and you have the bucket name for the same

Let’s start into building the VBCS application

 1. Build a VBCS application and drag and drop a File upload component to the same

          

2. Set up below properties for File Picker component

    a.      Selection Mode : Single

3. Create a Service Connection for Object storage with below tested endpoints

a.      REST endpoint for object storage looks like below :

       https://objectstorage.eu-frankfurt-1.oraclecloud.com/{version}/{account}

4. Create a New “Selected Files” event for File Picker and assign the file data to a page variable. Also add a Reset variable to make sure every time a new file is loaded the page variable gets updated with new file details

5. Once done create a Process button to start with the Multipart upload process

6. Create a ojAction event on Button, this is where all the Multipart code will be written and executed

7. Create a JS function “getFileChunks” which will break the file into multiple Base 64 parts and return the array for the same

PageModule.prototype.getFileChunks = function(file,byteSize){

    let startPointer = 0;

    let byteSizeNum = parseInt(byteSize,10);

    let endPointer = file.size;

    console.log("File Size : "+endPointer);

    console.log("byteSizeNum : "+byteSizeNum+" is type of "+typeof (byteSizeNum));

    let b64Chunks = [];

    while(startPointer<endPointer){

      let newStartPointer = 0;

      newStartPointer = startPointer+byteSizeNum;

      b64Chunks.push(convertBlobToB64(file.slice(startPointer,newStartPointer)));

      startPointer = newStartPointer;

    }

    return b64Chunks;

  };

 

async function convertBlobToB64 (file){

    let fileContentPromise = new Promise(function(resolve){

      if (file){

        console.log("File exists");

        var reader = new FileReader();

        reader.onloadend = function (fileReadEvent){

          var fileDataBase64 = fileReadEvent.target.result;

          resolve(fileDataBase64);

        };

        reader.readAsDataURL(file);

      }

    });

  return await fileContentPromise;

  };

 8.       Call JS function to create body for “initializeMultiPartRest” Rest service

PageModule.prototype.initMultiPartBody = function(objectName){

    let body = '{ "object": "'+objectName+'" }';

    console.log("Body : "+body);

    return body;  

};

9. Call “objectStorage/initializeMultiPart” Rest with below data :

         a.  Parameters :

                i. Bucket_name : <bucket name in Object storage>

          b. Body

                i. Pass above function output

10. On Successful execution of Rest it returns uploadId which will be used further in the process, hence store the same in a page variable

11. Open a loop to process and upload parts into object storage

        a. Call “objectStorage/uploadMultiPartFile” Rest to upload parts with below data

                i. Parameters :

1.       bucketName : < bucket name in Object storage >

2.      finalFileName : <file name>

3.      uploadPartNum : <loop index starting with 1-10000>

4.      uploadid: <derived from init Rest service> 

                          i.      Body :

                                               1. Pass the base64 file parts

12. Once all the parts have been uploaded successfully call “objectStorage/getMultiPartDetails” Rest service to check and collect data for commiting the parts

        a.      Parameters :

                i. bucketName : < bucket name in Object storage >

                 ii. finalFileName : <file name>

                 iii. uploadid: <derived from init Rest service>

13. Once the Rest service is run successfully write a JS function to process getMultiPart Rest Service output and form the body for calling commitMultiPart

PageModule.prototype.commitMultiPartBody = function(multiPartDetails){

    let multiPartJson = JSON.stringify(multiPartDetails);

    console.log("multiPartJson - "+multiPartJson);

    let multiPartJson1 = multiPartDetails;

    multiPartJson1.map(jsonObj => {

      delete jsonObj.md5;

      delete jsonObj.size;

      delete jsonObj.lastModified;

      jsonObj.partNum = jsonObj.partNumber;

      delete jsonObj.partNumber;

      return jsonObj;

    });

    console.log("multiPartJson1 - After delete : "+JSON.stringify(multiPartJson1));

    let body = '{ "partsToCommit": '+JSON.stringify(multiPartJson1)+' }';

    console.log("Commit Body : "+body);

    return body;

  };

14. Call “objectStorage/commitMultiPartUpload” Rest to commit the operation

        a. Parameters :

                i. bucketName : < bucket name in Object storage>

                ii. finalFileName : <file name>

                iii. uploadid: <derived from init Rest service>

        b. Body

                i.   Pass above function output

15. Once successful executed file is uploaded successfully to Object Storage, this can either be checked offline in objectstorage or run another get service on object storage to check the same

References :

https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/usingmultipartuploads.htm

https://soavijaya.blogspot.com/2021/05/oci-object-storage-multi-part-upload.html