gaspersic/freeCodeCamp
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

docs: update flight manuals (#39361)

Browse Source
This commit is contained in:
Mrugesh Mohapatra
2020-08-04 01:42:50 +05:30
committed by GitHub
parent f52baa51be
commit a3c1e25d78
1 changed files with 240 additions and 119 deletions
Download Patch File Download Diff File Expand all files Collapse all files

359
docs/flight-manuals/working-on-virtual-machines.md
View File

@ -1,17 +1,20 @@
# Flight Manual for working on Virtual Machines # Flight Manual for working on Virtual Machines
As a member of the staff or the dev-team, you may have been given access to our cloud service providers like Azure, Digital Ocean, etc. As a member of the staff or the dev-team, you may have been given access to our
cloud service providers like Azure, Digital Ocean, etc.
Here are some handy commands that you can use to work on the Virtual Machines (VM), for instance performing maintenance updates or doing general houeskeeping. Here are some handy commands that you can use to work on the Virtual Machines
(VM), for instance performing maintenance updates or doing general houeskeeping.
# Get a list of the VMs # Get a list of the VMs
> [!NOTE] > [!NOTE] While you may already have SSH access to the VMs, that alone will not
> While you may already have SSH access to the VMs, that alone will not let you list VMs unless you been granted access to the cloud portals as well. > let you list VMs unless you been granted access to the cloud portals as well.
## Azure ## Azure
Install Azure CLI `az`: https://docs.microsoft.com/en-us/cli/azure/install-azure-cli Install Azure CLI `az`:
https://docs.microsoft.com/en-us/cli/azure/install-azure-cli
> **(One-time) Install on macOS with [`homebrew`](https://brew.sh):** > **(One-time) Install on macOS with [`homebrew`](https://brew.sh):**
@ -33,7 +36,8 @@ az vm list-ip-addresses --output table
## Digital Ocean ## Digital Ocean
Install Digital Ocean CLI `doctl`: https://github.com/digitalocean/doctl#installing-doctl Install Digital Ocean CLI `doctl`:
https://github.com/digitalocean/doctl#installing-doctl
> **(One-time) Install on macOS with [`homebrew`](https://brew.sh):** > **(One-time) Install on macOS with [`homebrew`](https://brew.sh):**
@ -43,7 +47,8 @@ brew install doctl
> **(One-time) Login:** > **(One-time) Login:**
Authentication and context switching: https://github.com/digitalocean/doctl#authenticating-with-digitalocean Authentication and context switching:
https://github.com/digitalocean/doctl#authenticating-with-digitalocean
``` ```
doctl auth init doctl auth init
@ -55,15 +60,109 @@ doctl auth init
doctl compute droplet list --format "ID,Name,PublicIPv4" doctl compute droplet list --format "ID,Name,PublicIPv4"
``` ```
# Keeping VMs Updated # Spin a VM (or VM Scale Set)
You should keep the VMs up to date by performing updates and upgrades. This will ensure that the virtual machine is patched with latest security fixes. ### 0. Prerequisites (workspace Setup) for Staff
> [!WARNING] Get a login session on `azure cli`, and clone the
> Before you run these commands: [`infra`](https://github.com/freeCodeCamp/infra) for setting up template
workspace.
```console
az login
git clone https://github.com/freeCodeCamp/infra
cd infra
```
Use the Scratchpad subdirectory for temporary files, and making one-off edits.
The contents in this subdirectory are intentionally ignored from source control.
### 1. Provision VMs on Azure.
List all Resource Groups
```console
az group list --output table
```
```console
Name Location Status
--------------------------------- ------------- ---------
tools-rg eastus Succeeded
```
Create a Resource Group
```
az group create --location eastus --name stg-rg
```
```console
az group list --output table
```
```console
Name Location Status
--------------------------------- ------------- ---------
tools-rg eastus Succeeded
stg-rg eastus Succeeded
```
Next per the need, provision a single VM or a scaleset.
#### A. provision single instances
```console
az vm create \
--resource-group stg-rg-eastus \
--name <VIRTUAL_MACHINE_NAME> \
--image UbuntuLTS \
--size <VIRTUAL_MACHINE_SKU>
--custom-data cloud-init/nginx-cloud-init.yaml \
--admin-username <USERNAME> \
--ssh-key-values <SSH_KEYS>.pub
```
#### B. provision scaleset instance
```console
az vmss create \
--resource-group stg-rg-eastus \
--name <VIRTUAL_MACHINE_SCALESET_NAME> \
--image UbuntuLTS \
--size <VIRTUAL_MACHINE_SKU>
--upgrade-policy-mode automatic \
--custom-data cloud-init/nginx-cloud-init.yaml \
--admin-username <USERNAME> \
--ssh-key-values <SSH_KEYS>.pub
```
> [!NOTE]
> >
> - Make sure that the VM has been provisioned completely and there is no post-install steps running. > - The custom-data config should allow you to configure and add SSH keys,
> - If you are updating packages on a VM that is already serving an application, make sure the app has been stopped / saved. Package updates will cause network bandwidth, memory and/or CPU usage spikes leading to outages on running applications. > install packages etc. via the `cloud-init` templates in your local
> workspace. Tweak the files in your local workspace as needed. The cloud-init
> config is optional and you can omit it completely to do setups manually as
> well.
>
> - The virtual machine SKU is something like: **Standard_B2s** which can be
> retrived by executing something like
> `az vm list-sizes -l eastus --output table` or checking the Azure portal
> pricing.
# Keep VMs updated
You should keep the VMs up to date by performing updates and upgrades. This will
ensure that the virtual machine is patched with latest security fixes.
> [!WARNING] Before you run these commands:
>
> - Make sure that the VM has been provisioned completely and there is no
> post-install steps running.
> - If you are updating packages on a VM that is already serving an application,
> make sure the app has been stopped / saved. Package updates will cause
> network bandwidth, memory and/or CPU usage spikes leading to outages on
> running applications.
Update package information Update package information
@ -85,88 +184,25 @@ sudo apt autoremove -y
# Work on Web Servers (Proxy) # Work on Web Servers (Proxy)
We are running load balanced (Azure Load Balancer) instances for our web servers. These servers are running NGINX which reverse proxy all of the traffic to freeCodeCamp.org from various applications running on their own infrastructures. We are running load balanced (Azure Load Balancer) instances for our web
servers. These servers are running NGINX which reverse proxy all of the traffic
to freeCodeCamp.org from various applications running on their own
infrastructures.
The NGINX config is available on [this repository](https://github.com/freeCodeCamp/nginx-config). The NGINX config is available on
[this repository](https://github.com/freeCodeCamp/nginx-config).
## First install ## First Install
### 0. Prerequisites (workspace Setup) for Staff Provisioning VMs with the Code
Get a login session on azure cli, and clone the `cloud-setup` (private repo) for setting up template workspace. ### 1. (Optional) Install NGINX and configure from repository.
```console The basic setup should be ready OOTB, via the cloud-init configuration. SSH and
az login make changes as necessary for the particular instance(s).
git clone cloud-setup
cd cloud-setup
```
### 1. Provision VMs on Azure. If you did not use the cloud-init config previously use the below for manual
setup of NGINX and error pages:
List all Resource Groups
```console
az group list --output table
```
```console
Name Location Status
--------------------------------- ------------- ---------
tools-rg eastus Succeeded
```
Create a Resource Group
```
az group create --location eastus --name stg-rg-eastus
```
```console
az group list --output table
```
```console
Name Location Status
--------------------------------- ------------- ---------
tools-rg eastus Succeeded
stg-rg-eastus eastus Succeeded
```
Next per the need, provision a single VM or a scaleset.
#### A. provision single instances
```console
az vm create \
--resource-group stg-rg-eastus \
--name <VIRTUAL_MACHINE_NAME> \
--image UbuntuLTS \
--custom-data cloud-init/nginx-cloud-init.yaml \
--admin-username <USERNAME> \
--ssh-key-values <SSH_KEYS>.pub
```
#### B. provision scaleset instance
```console
az vmss create \
--resource-group stg-rg-eastus \
--name <VIRTUAL_MACHINE_SCALESET_NAME> \
--image UbuntuLTS \
--upgrade-policy-mode automatic \
--custom-data cloud-init/nginx-cloud-init.yaml \
--admin-username <USERNAME> \
--ssh-key-values <SSH_KEYS>.pub
```
> [!NOTE]
> The custom-data config should allow you to configure and add SSH keys, install packages etc. via the cloud-init templates in your local workspace. Tweak the files in your local workspace as needed. The cloud-init config is optional and you can omit it completely to do setups manually as well.
### 2. (Optional) Install NGINX and configure from repository.
The basic setup should be ready OOTB, via the cloud-init configuration. SSH and make changes as necessary for the particular instance(s).
If you did not use the cloud-init config previously use the below for manual setup of NGINX and error pages:
```console ```console
sudo su sudo su
@ -181,9 +217,10 @@ git clone https://github.com/freeCodeCamp/nginx-config nginx
cd /etc/nginx cd /etc/nginx
``` ```
### 3. Install Cloudflare origin certificates and upstream application config. ### 2. Install Cloudflare origin certificates and upstream application config.
Get the Cloudflare origin certificates from the secure storage and install at required locations. Get the Cloudflare origin certificates from the secure storage and install at
required locations.
**OR** **OR**
@ -199,21 +236,6 @@ rm -rf ./ssl
mv /tmp/ssl ./ mv /tmp/ssl ./
``` ```
<details>
<summary> Custom workflow with managed keys and hosts (Mrugesh) </summary>
```console
# Local
scp -r -i ~/.ssh/id_rsa_fcc source-server-hostname:/etc/nginx/ssl ./
scp -pr -i ~/.ssh/id_rsa_fcc ./ssl target-server-hostname:/tmp/
# Remote
rm -rf ./ssl
mv /tmp/ssl ./
```
</details>
Update Upstream Configurations: Update Upstream Configurations:
```console ```console
@ -222,13 +244,14 @@ vi configs/upstreams.conf
Add/update the source/origin application IP addresses. Add/update the source/origin application IP addresses.
### 4. Setup networking and firewalls. ### 3. Setup networking and firewalls.
Configure Azure firewalls and `ufw` as needed for ingress origin addresses. Configure Azure firewalls and `ufw` as needed for ingress origin addresses.
### 5. Add the VM to the load balancer backend pool. ### 4. Add the VM to the load balancer backend pool.
Configure and add rules to load balancer if needed. You may also need to add the VMs to load balancer backend pool if needed. Configure and add rules to load balancer if needed. You may also need to add the
VMs to load balancer backend pool if needed.
## Logging and Monitoring ## Logging and Monitoring
@ -244,7 +267,8 @@ sudo systemctl status nginx
## Updating Instances (Maintenance) ## Updating Instances (Maintenance)
Config changes to our NGINX instances are maintained on GitHub, these should be deployed on each instance like so: Config changes to our NGINX instances are maintained on GitHub, these should be
deployed on each instance like so:
1. SSH into the instance and enter sudo 1. SSH into the instance and enter sudo
@ -260,7 +284,8 @@ git fetch --all --prune
git reset --hard origin/master git reset --hard origin/master
``` ```
3. Test and reload the config [with Signals](https://docs.nginx.com/nginx/admin-guide/basic-functionality/runtime-control/#controlling-nginx). 3. Test and reload the config
[with Signals](https://docs.nginx.com/nginx/admin-guide/basic-functionality/runtime-control/#controlling-nginx).
```console ```console
nginx -t nginx -t
@ -269,8 +294,6 @@ nginx -s reload
# Work on API Instances # Work on API Instances
> **Todo: Add VM setup and installation details**
1. Install build tools for node binaries (`node-gyp`) etc. 1. Install build tools for node binaries (`node-gyp`) etc.
```console ```console
@ -330,15 +353,17 @@ pm2 logs
``` ```
```console ```console
pm2 monitor pm2 monit
``` ```
## Updating Instances (Maintenance) ## Updating Instances (Maintenance)
Code changes need to be deployed to the API instances from time to time. It can be a rolling update or a manual update. The later is essential when changing dependencies or adding enviroment variables. Code changes need to be deployed to the API instances from time to time. It can
be a rolling update or a manual update. The later is essential when changing
dependencies or adding enviroment variables.
> [!DANGER] > [!DANGER] The automated pipelines are not handling dependencies updates at the
> The automated pipelines are not handling dependencies updates at the minute. We need to do a manual update before any deployment pipeline runs. > minute. We need to do a manual update before any deployment pipeline runs.
### 1. Manual Updates - Used for updating dependencies, env variables. ### 1. Manual Updates - Used for updating dependencies, env variables.
@ -372,5 +397,101 @@ pm2 start all --update-env && pm2 logs
pm2 reload all --update-env && pm2 logs pm2 reload all --update-env && pm2 logs
``` ```
> [!NOTE] > [!NOTE] We are handling rolling updates to code, logic, via pipelines. You
> We are handling rolling updates to code, logic, via pipelines. You do not need to run these commands. These are here for documentation. > should not need to run these commands. These are here for documentation.
# Work on Client Instances
1. Install build tools for node binaries (`node-gyp`) etc.
```console
sudo apt install build-essential
```
## First Install
Provisioning VMs with the Code
1. Install Node LTS.
2. Update `npm` and install PM2 and setup logrotate and startup on boot
```console
npm i -g npm
npm i -g pm2
npm install -g serve
pm2 install pm2-logrotate
pm2 startup
```
3. Clone client config, setup env and keys.
```console
git clone https://github.com/freeCodeCamp/client-config.git client
cd client
```
```console
git clone https://github.com/freeCodeCamp/client-config.git client
cd client
```
Start placeholder instances for the web client, these will be updated with
artifacts from the Azure pipline.
> Todo: This setup needs to move to S3 or Azure Blob storage
```console
echo "serve -c ../../serve.json www -p 50505" >> client-start-primary.sh
chmod +x client-start-primary.sh
pm2 delete client-primary
pm2 start ./client-start-primary.sh --name client-primary
echo "serve -c ../../serve.json www -p 52525" >> client-start-secondary.sh
chmod +x client-start-secondary.sh
pm2 delete client-secondary
pm2 start ./client-start-secondary.sh --name client-secondary
```
## Logging and Monitoring
```console
pm2 logs
```
```console
pm2 monit
```
## Updating Instances (Maintenance)
Code changes need to be deployed to the API instances from time to time. It can
be a rolling update or a manual update. The later is essential when changing
dependencies or adding enviroment variables.
> [!DANGER] The automated pipelines are not handling dependencies updates at the
> minute. We need to do a manual update before any deployment pipeline runs.
### 1. Manual Updates - Used for updating dependencies, env variables.
1. Stop all instances
```console
pm2 stop all
```
2. Install or update dependencies
3. Start Instances
```console
pm2 start all --update-env && pm2 logs
```
### 2. Rolling updates - Used for logical changes to code.
```console
pm2 reload all --update-env && pm2 logs
```
> [!NOTE] We are handling rolling updates to code, logic, via pipelines. You
> should not need to run these commands. These are here for documentation.