A plethora of banks employ JSON for data transfer between the bank and the client servers. With the help of Amazon S3 data lakes, the banks use the data to train AI/ML models and perform advanced analytics. JSON is an open standard file format and data interchange format that stores and transmits data objects composed of attribute-value pairs and array data types using human-readable text (or any other serializable value).
It’s extensively used on the internet for practically every API you’ll access, as well as config files and other things like games and text editors. Almost every major programming language provides a library or built-in functionality for parsing JSON texts into objects or classes. This makes working with JSON data in a computer language extremely simple.
In this blog, we will discuss the step-by-step implementation of how to fetch the contents of a JSON File stored in Amazon S3 using Express.js and AWS SDK.
In this blog, we will cover:
- What is Express.js?
- Advantages of Express.js
- Why should you use Express.js?
- Features of Express.js
- What is AWS SDK?
- What is Amazon S3?
- Need for Amazon S3 integration in an application
- Required installations for the process
- Hands-on
- Conclusion
What is Express.js?
Express is a simple web framework for Node.js that is quick and unopinionated. Express.js is free and open-source. It is used to quickly and easily design and create web apps. It includes a number of features that enable developing web applications faster and easier than if only Node.js was utilized.
The link between Express.js is built on the Node.js gateway module, which uses the HTTP module. As a result, any connect-based middleware will function with Express.js.
It’s a backend web application framework that’s available as free and open-source software under the MIT License.
Advantages of Express.js
- Develops Node.js web applications quickly and easily.
- It’s simple to set up and personalize.
- Allows you to define application routes using HTTP methods and URLs.
- Includes a number of middleware modules that can be used to execute additional requests and responses activities.
- Simple to interface with a variety of template engines, including Jade, Vash, and EJS.
- Allows you to specify a middleware for handling errors.
- It’s simple to provide your application’s static files and resources.
- You can use it to make a REST API server.
- Databases such as MongoDB, Redis, and MySQL are simple to connect to.
Why should you use Express.js?
- JavaScript, a widely used language that is simple to use and widely supported, is supported by express js.
- If you are familiar with JavaScript, you will find it very simple to use express. js
- You can quickly create any web program with the help of Express.js. Express.js provides a simple route for client requests.
- To create a routing component without Express.js, you have to write your own code, which is a time-consuming and challenging task.
- Express.js provides programmers with simplicity, efficiency, flexibility, and scalability.
- It has a strong performance because it is a Node.js framework.
- Because Express.js is written in JavaScript, it can be used for websites and web applications, and even mobile applications.
Features of Express.js
Usability: Many commonly used Node.js features are provided by Express.js in the form of functions that can be used anywhere in the programme.
Connectivity: It is a program component that has a database, client request, and other middleware access. Its primary responsibility is to ensure the systematic organization of express js functions.
Path: Express Js provides an advanced pathway system that aids in the preservation of the state of web pages via their URLs.
Templating: Expressjs includes templating engines that enable developers to create dynamic content for web pages by building HTML templates on the server-side.
Debugging: Debugging is critical for web page development. Debugging has the ability to pinpoint the exact section of an application that contains bugs.
What is AWS SDK?
The AWS SDK for JavaScript is a set of software tools for creating apps and libraries which use the AWS resources. There are individual software development kits (SDKs) for developing browser-based and server-side JavaScript applications.
The AWS SDK for browser-based development enables developers to access AWS from JavaScript code running directly in the browser. User authentication via Facebook, Google, or Amazon login is possible with a web identity federation.
Developers use the AWS SDK for JavaScript in Node.js for server-side applications. The SDK includes JavaScript objects for connecting to AWS components like Amazon S3, Amazon SQS, Amazon SNS, and DynamoDB.
To use the AWS SDK for JavaScript, developers should only download the appropriate SDK – no additional software is required.
What is Amazon S3?
Amazon S3 is a Simple Storage Service with industry-leading scalability, data availability, security, and performance.
Amazon S3 allows customers of all sizes and sectors to store and safeguard any amount of data for a variety of use cases, including data lakes, websites, mobile applications, backup and restore, archive, business applications, IoT devices, and big data analytics.
You may use Amazon S3’s administration tools to optimize, organize, and configure data access to meet your specific business, organizational, and compliance needs.
It delivers 99.999999999% durability and 99.99% availability of items while being built for developers to make web-scale computing easier. It can also hold up to 5 terabytes of computer data.
Need for Amazon S3 integration in an application
Every organization requires to handle huge amounts of data – structured and unstructured, at shifting pace, volume, and variety. This calls for the need of having a robust cloud strategy.
Amazon Simple Storage Service (Amazon S3) is designed to store and retrieve any quantity of data from any location for millions of applications and is utilized by businesses of all sizes. Users may utilize Amazon S3 to seamlessly migrate and integrate vital business and application data for highly-scalable cloud storage by integrating Amazon S3 into their applications.
The most prominent benefits of Amazon S3 include reliability, high security, all-time availability, low cost, ease of Migration, and simplicity of management.
Required installations for the process
- AWS SDK: It is a collection of software tools that allows developers to connect to the various AWS services via the Javascript code to be able to perform the required operations.
- Express: It is basically a back-end web application framework for Node.js designed for building various web applications and APIs.
- Express Generator: It is an all-in-one tool that provides us with the flexibility to create a node.js application with ease and quickly using the express library.
Hands-on
In this hands-on, we will have a look at the steps and the procedure to fetch and parse the contents of a JSON file uploaded on Amazon S3 using Express.js and AWS SDK. We will first set up an Express application into a new directory onto the local machine. Then we will perform the required package installations. Post that, we will navigate to the AWS console, create a new bucket in the Amazon S3 service and add a sample JSON file to it as an object. We will then navigate to the Amazon IAM dashboard to create access keys for the IAM user that we will have to store in the codebase to be able to connect to Amazon S3. We will then install the AWS-SDK package to connect to the service. Altering the code base, we will first connect to the Amazon S3 service via Express.js hitting the URL on port 3000 fetching the unparsed contents of the object uploaded in S3. Then we will alter the codebase to get the parsed object on the command terminal and finally, we will make changes in the codebase in a way to paste the contents of the JSON file on the UI.
Create a new directory on your local machine. Ensure that git bash is installed and if not then you can make use of cmd to perform the operations. Right-click into the new directory and click on Git Bash Here.
A new command prompt will open up. Execute npm init. Keep hitting enter selecting the default options.
On creation of the object, confirm the configuration and hit enter.
In the same directory, a new file package.json will be created.
Open the directory into a code editor.
Navigate back to the command prompt and execute npm i express.
Open the package.json file and confirm the creation of the package.
Now, navigate back to the terminal and execute the below command.
Post that, run express onto the terminal. A new folder structure will be created for you in the directory.
You will get to see the folder structure in the code editor.
Onto the command terminal, execute npm install.
Execute npm start to start the server and test it out.
Onto a browser, hit the below URL.
On success, you will see the screen as shown below.
Now, open the app.js file. The file will have some already written codes. You may change the routes if needed.
Login to your AWS console and navigate to the dashboard.
Navigate to the Amazon S3 dashboard and click on create a bucket.
Enter a name for the bucket and leave the other settings to default.
Enable block all public access.
Disable versioning and add tags if needed.
Scroll to the bottom and click on Create bucket.
On success, you will see the message as shown in the image below.
Now in the same directory create a new JSON file and add some JSON formatted content to it and save it.
Navigate to the newly created bucket page and click on upload.
Upload the JSON file into the bucket.
On success, you will see the file in the bucket as shown below.
Now, search for IAM and navigate to the IAM dashboard.
Click on Users in the left navigation pane.
Select your user and choose the Security credentials tab. Click on create an access key.
Download the access key id and secret access key.
Navigate back to the code terminal and execute the command:
npm i aws-sdk
Confirm the package installation into the package.json file.
Open the index.js file that is to be altered.
In the folder structure, right-click on the javascripts folder in the public directory and click on New File.
Enter a name for the file and hit enter. This file will have the code to fetch the object from S3.
Open the newly created file and add the below code into the same file. Add the access key id, secret access key, and the region that we fetched above.
Add the following code to the same file. Thus code will fetch the object from the S3 bucket and provide us with an output. Note that this function will return a Uint8 Array buffer that will be unparsed.
To get the bucket name and the key name, navigate to the S3 bucket and click on the uploaded object. In the S3 URN, you will find the bucket name and the key name.
Replace the below variables with the bucket name and the object key name.
Now, open the index.js file under the routes directory in the folder structure. Remove the default code from the router.get section.
Import the newly created javascript file into the routes file as shown in the image below.
Paste the code into the router.get section. Console.log will print the output on success.
The index.js file looks like as shown below with the final code.
Stop the server and restart it by using the npm start command in the terminal.
Hit the below api url again.
On success, you will see the message as shown in the image below. That means the object has been successfully fetched from the S3 bucket.
Navigate back to the command terminal on which the server is running. On success, you will see the Body parameter with the Uint8 Array that is unparsed. To make it in a readable format, we need to parse the same.
To parse the output and make it into a readable format, alter the code in the index.js file as shown in the image below. Parse the output using json.parse.
Stop and restart the server.
Hit the URL again to check out the result.
Now, you will get to see the parse JSON object in the file that exists in the S3 bucket.
Now, alter the code in the index.js file with the code below in case you wish to print the content of the file that’s stored in S3 on the UI.
Restart the server and hit the URL on port 3000.
On success, you will see the JSON content of the file on the UI as shown in the image below.
In case you are following this implementation for learning purposes, ensure to delete the created bucket and also delete the generated access keys to avoid unnecessary costs if not needed later on.
Conclusion
In this blog, we had a look at the steps and the procedure to fetch and parse the contents of a JSON file uploaded on Amazon S3 using Express.js and AWS SDK. We first set up an Express application into a new directory onto the local machine. Then we performed the required package installations. After that, we navigated to the AWS console, created a new bucket in the Amazon S3 service, and added a sample JSON file to it as an object. We then navigated to the Amazon IAM dashboard and created access keys for the IAM user that we stored in the codebase to connect to Amazon S3. We then installed the AWS-SDK package to connect to the service.
Altering the code base, we first connected to the Amazon S3 service via Express.js hitting the URL on port 3000 and fetching the unparsed contents of the object uploaded in S3. Then we altered the codebase to get the parsed object on the command terminal and finally, we made changes in the codebase in a way to paste the contents of the JSON file on the UI. We will discuss more such use cases in our upcoming blogs. Stay tuned to keep getting all updates about our upcoming new blogs on AWS and relevant technologies.
Meanwhile …
Keep Exploring -> Keep Learning -> Keep Mastering
This blog is part of our effort towards building a knowledgeable and kick-ass tech community. At Workfall, we strive to provide the best tech and pay opportunities to AWS-certified talents. If you’re looking to work with global clients, and build kick-ass products while making big bucks doing so, give it a shot at workfall.com/partner today.
