Atlas on AWS¶
The beauty of the cloud is that everyone can have easy access to really powerful machines.
Being able to run Atlas on AWS means access to GPUs and faster results.
In this tutorial we will cover the following:
- Setting up an AWS instance for DeepLearning
- Setting up Atlas
- Configuring VSCode to be able to run remote jobs
- Running our first job remotely!
- An AWS account
First, let's go to the AWS developer console. Sign in, and then we'll create a new instance: AWS console
Go to Services > EC2, then click the "Launch Instance" button to start setup.
Step 1: Choose an Amazon Machine Image (AMI)¶
We will be using AWS's DeepLearning AMI for this tutorial. We recommend using this to begin with unless you have heavy custom requirements. AWS's DeepLearning AMI already includes Python setup using anaconda and comes pre-built with maority of the DeepLearning libraries like Tensorflow.
You are welcome to use your own custom instances, just make sure to install Anaconda here is a great tutorial..
Let's use the AWS Deep Learning AMI, to do this:
- Select "AWS Marketplace" from the sidebar
- Search for "AWS Deep Learning AMI"
- Select the AWS AMI with the latest version of Ubuntu.
- In our case we have selected
AWS Deep Learning AMI (Ubuntu 18.04)
- In our case we have selected
Step 2: Configure an Instance Type¶
For this page we'll select a p2.xlarge instance type, as we want some a GPU to train with. We recommend a P2 instance as it a good balance between performance and cost, but if you are looking for more power you can select a P3 instance.
Make sure to click "Next: Configure Instance Details" to continue provisioning the instance.
Step 3: Configure Instance Details¶
- If your instance does require additional configuration adjust as needed
- Otherwise there are no changes needed here, just click "Next: Add Storage"
Step 4: Add Storage¶
We recommend increasing the size of your storage to at least 250GB, as the base AMI image is already quite big. Adjust as needed for your expected data.
Step 5: Add Tags¶
- No changes, just click Next: Configure Security Group
Step 6: Configure Security Group¶
Next, we'll create a new security group to allow for Atlas to securely use a few different ports on our instance. Specifically to allow the GUI, REST API, and archive server.
To allow for this:
- Add 3 new rules, with ports for 5555, 5557, and 5959
- Adjust the "Source" to My IP for all the rules including the SSH rule
Changing the source to "My IP" means that the field will automatically be populated with the public IPv4 address of your local computer. This acts as a firewall for the instance, ensuring that the only inbound traffic comes from your personal computer. Learn more from the official AWS guide.
Now, click Review and Launch to go review our instance before launching.
Step 7: Review Instance Launch¶
Our instance setup should now look similar to the below.
Step 8: Launch¶
- Click "Launch" which should then ask you to create a new key pair
- Select "Create a new key pair" and give it the name "atlas" (note: if you've already used this key name before you can either re-use of, or create another unique key name)
- Download the key pair to a folder
- Click "Launch Instances"
- It should then redirect you to the Launch Status page
- Beside "The following instance launches have been initiated" you should see your new instance ID. Click the ID to go to the Instances page
- You can find the IPv4 address of the instance in the lower Description panel.
💰 Once the instance has been finalized and spun up, costs will be incurred. Please follow the tear down steps at the bottom of this guide to minimize this.
Step 9: SSH¶
Make sure you have the
.pem key from Step 7.
We should now be able to use your downloaded
.pem key to then SSH into the instance to check that everything is setup.
chmod 400 <key_name>.pemon your key before SSH'ing to give it the proper permissions.
- [Optional] We also recommend that you move the key to your
ssh -i <path_to_your_key>.pem ubuntu@<instance_ipv4>replacing "<>" with your key file and IPv4 in your terminal. This will connect you to the remote instance.
You should expect to wait ~30 seconds when you first SSH in.
- You should now be within your AWS instance.
- Create a new directory for Atlas:
- Find a version of Atlas from https://github.com/dessa-oss/atlas/releases that you want to install and run
wget <link_to_installer_file>to download the installer file to this instance.
- Create and activate a Python >=3.6 virtual environment using Conda or venv to minimize dependency issues.
conda create --name=atlas python=3.7
- Ensure you are in the right conda environment, then run the install script
python atlas_installer.pyand follow the instructions.
python atlas_installer.py --help will give you troubleshooting advice if the script isn't working as expected.
The longest part of the script is pulling the Atlas docker images, if the script fails at this point,
you can re-run it using
python atlas_installer.py -dp to skip over the download and unpacking and go directly to the image pull.
Run our first Atlas job¶
- Validate that you are in the same Python environment that was used to run the installation script.
If your instance comes with a GPU, you can ensure Atlas runs with GPU enabled by running
atlas-server start -g
You should now be able to see the GUI. In your browser go
<ipv4.of.ec2.instance>:5555 and you should see the Atlas project page with two projects that have each been run once. These projects were baked into the AMI to help get started.
If this is your first time using Atlas, you can try to run a simple job with the following steps or you can just skip ahead to setting up AWS to remotely work with VSCode below.
- Navigate to where you'd like to create your Atlas project directory.
foundations init hello-atlasto create an example project.
- Navigate into the newly created
- Run the sample code provided by running
- Head to the GUI to see your experiment!
For more information on the Foundations CLI read our CLI section, or run
We've now successfully setup a P2 instance with Atlas! In the next section we'll go over running jobs on AWS with VSCode.
Running jobs remotely via VSCode¶
Configuring VSCode to work with a remote instance allows us to code on our own machines, while having the power of a cloud GPU. While we also love PyCharm's remote deployment, it's not free, so we'll just use VSCode. This will allow for our code to be synced with our instance, and simply run
foundations submit ... commands to run jobs.
- Install VSCode
First, open up VSCode, and we'll install the Remote - SSH plugin that will allow us to open code from the remote instance in VSCode:
- Open extensions via the left bar menu (or by pressing
shift + cmd + X)
- Search for
- install the plugin
Before we connect to the host in VSCode, let's open the SSH config file, to add this instance to our SSH Config, to easily allows VSCode to connect to our instance.
~/.ssh/config(Create it if it does not exist)
Add the following lines:
Host <my.aws.ip.address> IdentityFile ~/.ssh/<key_name>.pem User ubuntu
Let's also give the right permission to our
pem file using the following command:
chmod 400 ~/.ssh/<key_name>.pem
Once the plugin is installed and the SSH config is updated, open the Command Palette and select "Remote-SSH: Connect to Host".
- Open command prompt in VSCode:
Cmd + Shift + P(or click green "Open Remote Window" botton, bottom left of VSCode)
- Search for
Remote-SSH: Connect to Hostfrom the menu
- Select your instance ip address
It should open a new VSCode window where we'll be able to select "File" > "Open..." and then select a directory from our instance to open. In our case it will be the sample project we made in the "Running your jobs on Atlas" section above.
At the bottom the VSCode window we'll select the "Terminal" tab, and click the "+" symbol to open a new shell tab. Choose "bash" from the drop down that will give us bash shell access to AWS.
cdinto your code's repository
- Let's test that we can run a job with
foundations submit scheduler . driver.py
- You should see this job in the Atlas GUI
We're now set! You can now open the files and adjust as you wish, or you can start on your own project.
If you're looking for more detailed docs on setting up VSCode with AWS, Microsoft has a good setup guide with much more detail.
Tear down AWS instance¶
It's important to be aware that after the instance has been spun up, you will incur costs on your account. There are two options: either stop the instance (which will mean costs are minimal to keep your storage around), or throw away the instance completely.
To stop or terminate the instance:
- Go to your AWS console
- Right click on our P2 instance > "Instance State" > either Terminate or Stop
That's it! We've successfully spun up a GPU instance, and run jobs remotely, from VSCode.
You can now either start your own projects, or look at some of our more advanced tutorials to explore more of Atlas.
If you have any thoughts or feedback about setting up Atlas on AWS, we're always happy to help answer questions on our Dessa's Community Slack!