How to set up AWS CLI, Create S3 Bucket and Copy Dataset
In this tutorial, we'll cover how to set up AWS CLI, then we will create an S3 bucket and copy the dataset into the bucket.
Introduction
We can confidently say that no single developer who has not interacted with any command-line interface (CLI), this is because CLI is a must-encounter tool when we talk about software development.
Most developers and data scientists prefer to use CLI when they are looking for performance and speed. This is because with Graphical User Interface (GUI) one has to navigate through different icons which makes it slow when compared to CLI where one utilizes the keyboard to navigate through the interface.
For any AWS user, AWS CLI is an essential tool. It is a tool that will help you manage your AWS services from the command line and automate them through scripts with just a few commands.
Setting up AWS CLI
Prerequisites :
First, we need to check if we have Python (version 3.9 and above) and PIP (version 8.1.1 and above) installed.
To check if Python and PIP are installed, run the following commands:
# Checking python version
sudo python3 --version or $ python3 -V
# Checking the version of pip
sudo pip --version or $ pip -V
Now after we have verified that we have Python and pip installed, we set to install AWS CLI.
Installation:
To install AWS CLI, run the following command:
# Installing AWS CLI using pip
sudo pip install awscli
After installation is complete, you can check the version of AWS CLI, run the following command:
# Checking the version of installed AWS CLI
sudo aws --version
NOTE: There is the old way to install AWS CLI and it’s not recommended because you won’t get the latest version here, which is using sudo apt-get install awscli.
Configuring AWS CLI
After installing the CLI, we need to configure the AWS CLI with the following command. We'll need AWS Access Key ID, and AWS Secret Access Key to configure the same.
To configure, we will use the following command:
# Configuring AWS CLI
$ aws configure
AWS Access Key ID [None]:
AWS Secret Access Key [None]:
Default region name [None]:
Default output format [None]:
Now you have configured AWS CLI on your system and you are ready to start using the CLI.
You can run the aws command to start.
Creating S3 Bucket
Now, after setting up AWS CLI, with a few commands we will be able to create an s3 bucket and copy data from our local machine.
There are two commands and most of the time, many developers get confused about which option they will use.
s3api vs s3
Both commands provide different functionality. Whereas s3api supports all of the functionality provided by S3, s3 supports a limited and higher-level set of functionality.
Generally, the s3 command is easier to use and if your use-case is solved by using the s3 command, then you should go ahead and use that.
If your use case isn’t supported by the s3 command, then you need to use the s3api subcommand.
If you need a lot more flexibility, then s3api might be the better option for you.
We will use s3api to create our s3 bucket.
# Creating s3 bucket called test-s3-bucket
aws s3api create-bucket --bucket test-s3-bucket --region us-east-1
Output:
{
"Location": "http://test-s3-bucket.s3.amazonaws.com/"
}
Copying data to the S3 bucket
Now after creating our s3 bucket, we can now perform operations like copying local files to s3, copying an s3 object from one bucket to another and many more.
Copying a file or data to an s3 bucket, you can use the following command:
aws s3 cp file.txt s3://bucket-name
In file.txt, you will replace it with the directory of the data or file you want to upload and also on bucket-name, replace it with the name of your bucket.
We can also perform several operations like list, get, put, and delete objects and buckets depending upon our requirements.
Conclusion
AWS CLI is an important tool for developers who daily use AWS services. It makes work easier for them to manage the services.
In this tutorial, we have learned how to set up AWS CLI on your system, and configure it so that you can start using it. Then we also learned how to create an s3 bucket and copy data from your local machine using AWS CLI.