From 4291c03c1a43420fcd49e48a972dce2f5de6e683 Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra <1884376+raisedadead@users.noreply.github.com> Date: Mon, 13 Jul 2020 18:50:00 +0530 Subject: [PATCH] docs: update flight manuals (#39240) Updated and unified workflows for working with various application VMs --- docs/_sidebar.md | 6 +- docs/flight-manuals/README.md | 1 - docs/flight-manuals/_sidebar.md | 7 - .../working-on-api-instances.md | 107 ------ .../working-on-virtual-machines.md | 355 ++++++++++++++++-- .../working-on-web-server-instances.md | 57 --- 6 files changed, 325 insertions(+), 208 deletions(-) delete mode 100644 docs/flight-manuals/README.md delete mode 100644 docs/flight-manuals/_sidebar.md delete mode 100644 docs/flight-manuals/working-on-api-instances.md delete mode 100644 docs/flight-manuals/working-on-web-server-instances.md diff --git a/docs/_sidebar.md b/docs/_sidebar.md index 924bd9ce8f..05b09e7f2e 100644 --- a/docs/_sidebar.md +++ b/docs/_sidebar.md @@ -14,9 +14,9 @@ - [文章翻译计划](/chinese-guides/news-translations.md) - [视频翻译计划](/chinese-guides/video-translations.md) - ** Flight Manuals** (for Staff & Mods) - - [Our DevOps Guide](/devops.md) - - [Using Reply Templates](/flight-manuals/using-reply-templates.md) - - [Other flight manuals](/flight-manuals/) + - [DevOps Overview](/devops.md) + - [Reply Templates](/flight-manuals/using-reply-templates.md) + - [Working on Servers](/flight-manuals/working-on-virtual-machines.md) --- diff --git a/docs/flight-manuals/README.md b/docs/flight-manuals/README.md deleted file mode 100644 index ed58a916e6..0000000000 --- a/docs/flight-manuals/README.md +++ /dev/null @@ -1 +0,0 @@ -Flight manuals for our staff and mods. \ No newline at end of file diff --git a/docs/flight-manuals/_sidebar.md b/docs/flight-manuals/_sidebar.md deleted file mode 100644 index f9276bb33f..0000000000 --- a/docs/flight-manuals/_sidebar.md +++ /dev/null @@ -1,7 +0,0 @@ -- [** Go back to previous guides**](/index.md) - ---- - -- [Work on Virtual Machines](/flight-manuals/working-on-virtual-machines.md) -- [Work on API Instances](/flight-manuals/working-on-api-instances.md) -- [Work on Web Server Instances](/flight-manuals/working-on-web-server-instances.md) diff --git a/docs/flight-manuals/working-on-api-instances.md b/docs/flight-manuals/working-on-api-instances.md deleted file mode 100644 index 1fb0863080..0000000000 --- a/docs/flight-manuals/working-on-api-instances.md +++ /dev/null @@ -1,107 +0,0 @@ -# Work on API Instances - -## Installing pre-requisites - -1. Perform updates to the OS packages by following [this guide](flight-manuals/working-on-virtual-machines). - -2. Install build tools for node binaries (`node-gyp`) etc. - - ```console - sudo apt install build-essential - ``` - -## First Install - -Provisioning VMs with the Code - -1. Install Node LTS. - -2. Update `npm` and install PM2 and setup logrotate and startup on boot - - ```console - npm i -g npm - npm i -g pm2 - pm2 install pm2-logrotate - pm2 startup - ``` - -3. Clone freeCodeCamp, setup env and keys. - - ```console - git clone https://github.com/freeCodeCamp/freeCodeCamp.git - cd freeCodeCamp - ``` - -4. Create the `.env` from the secure credentials storage. - -5. Install dependencies - - ```console - npm ci - ``` - -6. Build the server - - ```console - npm run ensure-env && npm run build:server - ``` - -7. Start Instances - - ```console - cd api-server - pm2 start production-start.js -i max --max-memory-restart 600M --name org - ``` - -## Logging and Monitoring - -```console -pm2 logs -``` - -```console -pm2 monitor -``` - -## Updating Instances (Maintenance) - -Code changes need to be deployed to the API instances from time to time. It can be a rolling update or a manual update. The later is essential when changing dependencies or adding enviroment variables. - -> [!DANGER] -> The automated pipelines are not handling dependencies updates at the minute. We need to do a manual update before any deployment pipeline runs. - -### 1. Manual Updates - Used for updating dependencies, env variables. - -1. Stop all instances - -```console -pm2 stop all -``` - -2. Install dependencies - -```console -npm ci -``` - -3. Build the server - -```console -npm run ensure-env && npm run build:server -``` - -4. Start Instances - -```console -pm2 start all --update-env && pm2 logs -``` - -### 2. Rolling updates - Used for logical changes to code. - -```console -pm2 reload all --update-env && pm2 logs -``` - -> [!NOTE] -> We are handling rolling updates to code, logic, via pipelines. You do not need to run these commands. These are here for documentation. - diff --git a/docs/flight-manuals/working-on-virtual-machines.md b/docs/flight-manuals/working-on-virtual-machines.md index 405d5b8ef5..1660d901e1 100644 --- a/docs/flight-manuals/working-on-virtual-machines.md +++ b/docs/flight-manuals/working-on-virtual-machines.md @@ -1,41 +1,13 @@ -# Working on Virtual Machines +# Flight Manual for working on Virtual Machines +As a member of the staff or the dev-team, you may have been given access to our cloud service providers like Azure, Digital Ocean, etc. -As a member of the staff or the dev-team, you may have been given access to our cloud vendors like Azure, Digital Ocean, etc. - -Here are some handy commands that you can use to work on the VMs, for instance performing maintenance updates or doing general houeskeeping. -# Keeping Virtual Machines Updated - -You should keep the Virtual Machines up to date by performing updates and upgrades. This will ensure that the virtual machine is patched with latest security fixes. - -> [!WARNING] -> Before you run these commands: -> - Make sure that the VM has been provisioned completely and there is no post-install steps running. -> - If you are updating packages on a VM that is already serving an application, make sure the app has been stopped / saved. Package updates will cause network bandwidth, memory and/or CPU usage spikes leading to outages on running applications. - -Update package information - -```console -sudo apt update -``` - -Upgrade installed packages - -```console -sudo apt upgrade -y -``` - -Cleanup unused packages - -```console -sudo apt autoremove -y -``` - -# Getting a list of the Virtual Machines +Here are some handy commands that you can use to work on the Virtual Machines (VM), for instance performing maintenance updates or doing general houeskeeping. +# Get a list of the VMs > [!NOTE] -> You may already have SSH access to virtual machines, but that alone will not let you list VMs unless you also have access to the vendor portals as well. +> While you may already have SSH access to the VMs, that alone will not let you list VMs unless you been granted access to the cloud portals as well. ## Azure @@ -82,3 +54,320 @@ doctl auth init ``` doctl compute droplet list --format "ID,Name,PublicIPv4" ``` + +# Keeping VMs Updated + +You should keep the VMs up to date by performing updates and upgrades. This will ensure that the virtual machine is patched with latest security fixes. + +> [!WARNING] +> Before you run these commands: +> +> - Make sure that the VM has been provisioned completely and there is no post-install steps running. +> - If you are updating packages on a VM that is already serving an application, make sure the app has been stopped / saved. Package updates will cause network bandwidth, memory and/or CPU usage spikes leading to outages on running applications. + +Update package information + +```console +sudo apt update +``` + +Upgrade installed packages + +```console +sudo apt upgrade -y +``` + +Cleanup unused packages + +```console +sudo apt autoremove -y +``` + +# Work on Web Servers (Proxy) + +We are running load balanced (Azure Load Balancer) instances for our web servers. These servers are running NGINX which reverse proxy all of the traffic to freeCodeCamp.org from various applications running on their own infrastructures. + +The NGINX config is available on [this repository](https://github.com/freeCodeCamp/nginx-config). + +## First install + +### 0. Prerequisites (workspace Setup) for Staff + +Get a login session on azure cli, and clone the `cloud-setup` (private repo) for setting up template workspace. + +```console +az login +git clone cloud-setup +cd cloud-setup +``` + +### 1. Provision VMs on Azure. + +List all Resource Groups + +```console +az group list --output table +``` + +```console +Name Location Status +--------------------------------- ------------- --------- +tools-rg eastus Succeeded +``` + +Create a Resource Group + +``` +az group create --location eastus --name stg-rg-eastus +``` + +```console +az group list --output table +``` + +```console +Name Location Status +--------------------------------- ------------- --------- +tools-rg eastus Succeeded +stg-rg-eastus eastus Succeeded +``` + +Next per the need, provision a single VM or a scaleset. + +#### A. provision single instances + +```console +az vm create \ + --resource-group stg-rg-eastus \ + --name \ + --image UbuntuLTS \ + --custom-data cloud-init/nginx-cloud-init.yaml \ + --admin-username \ + --ssh-key-values .pub +``` + +#### B. provision scaleset instance + +```console +az vmss create \ + --resource-group stg-rg-eastus \ + --name \ + --image UbuntuLTS \ + --upgrade-policy-mode automatic \ + --custom-data cloud-init/nginx-cloud-init.yaml \ + --admin-username \ + --ssh-key-values .pub +``` + +> [!NOTE] +> The custom-data config should allow you to configure and add SSH keys, install packages etc. via the cloud-init templates in your local workspace. Tweak the files in your local workspace as needed. The cloud-init config is optional and you can omit it completely to do setups manually as well. + +### 2. (Optional) Install NGINX and configure from repository. + +The basic setup should be ready OOTB, via the cloud-init configuration. SSH and make changes as necessary for the particular instance(s). + +If you did not use the cloud-init config previously use the below for manual setup of NGINX and error pages: + +```console +sudo su + +cd /var/www/html +git clone https://github.com/freeCodeCamp/error-pages + +cd /etc/ +rm -rf nginx +git clone https://github.com/freeCodeCamp/nginx-config nginx + +cd /etc/nginx +``` + +### 3. Install Cloudflare origin certificates and upstream application config. + +Get the Cloudflare origin certificates from the secure storage and install at required locations. + +**OR** + +Move over existing certificates: + +```console +# Local +scp -r username@source-server-public-ip:/etc/nginx/ssl ./ +scp -pr ./ssl username@target-server-public-ip:/tmp/ + +# Remote +rm -rf ./ssl +mv /tmp/ssl ./ +``` + +
+ + Custom workflow with managed keys and hosts (Mrugesh) + +```console +# Local +scp -r -i ~/.ssh/id_rsa_fcc source-server-hostname:/etc/nginx/ssl ./ +scp -pr -i ~/.ssh/id_rsa_fcc ./ssl target-server-hostname:/tmp/ + +# Remote +rm -rf ./ssl +mv /tmp/ssl ./ +``` +
+ +Update Upstream Configurations: + +```console +vi configs/upstreams.conf +``` + +Add/update the source/origin application IP addresses. + +### 4. Setup networking and firewalls. + +Configure Azure firewalls and `ufw` as needed for ingress origin addresses. + +### 5. Add the VM to the load balancer backend pool. + +Configure and add rules to load balancer if needed. You may also need to add the VMs to load balancer backend pool if needed. + +## Logging and Monitoring + +1. Check status for NGINX service using the below command: + +```console +sudo systemctl status nginx +``` + +2. Logging and monitoring for the servers are available at: + +>

https://amplify.nginx.com

+ +## Updating Instances (Maintenance) + +Config changes to our NGINX instances are maintained on GitHub, these should be deployed on each instance like so: + +1. SSH into the instance and enter sudo + +```console +sudo su +``` + +2. Get the latest config code. + +```console +cd /etc/nginx +git fetch --all --prune +git reset --hard origin/master +``` + +3. Test and reload the config [with Signals](https://docs.nginx.com/nginx/admin-guide/basic-functionality/runtime-control/#controlling-nginx). + +```console +nginx -t +nginx -s reload +``` + +# Work on API Instances + +> **Todo: Add VM setup and installation details** + +1. Install build tools for node binaries (`node-gyp`) etc. + +```console +sudo apt install build-essential +``` + +## First Install + +Provisioning VMs with the Code + +1. Install Node LTS. + +2. Update `npm` and install PM2 and setup logrotate and startup on boot + + ```console + npm i -g npm + npm i -g pm2 + pm2 install pm2-logrotate + pm2 startup + ``` + +3. Clone freeCodeCamp, setup env and keys. + + ```console + git clone https://github.com/freeCodeCamp/freeCodeCamp.git + cd freeCodeCamp + ``` + +4. Create the `.env` from the secure credentials storage. + +5. Install dependencies + + ```console + npm ci + ``` + +6. Build the server + + ```console + npm run ensure-env && npm run build:server + ``` + +7. Start Instances + + ```console + cd api-server + pm2 start production-start.js -i max --max-memory-restart 600M --name org + ``` + +## Logging and Monitoring + +```console +pm2 logs +``` + +```console +pm2 monitor +``` + +## Updating Instances (Maintenance) + +Code changes need to be deployed to the API instances from time to time. It can be a rolling update or a manual update. The later is essential when changing dependencies or adding enviroment variables. + +> [!DANGER] +> The automated pipelines are not handling dependencies updates at the minute. We need to do a manual update before any deployment pipeline runs. + +### 1. Manual Updates - Used for updating dependencies, env variables. + +1. Stop all instances + +```console +pm2 stop all +``` + +2. Install dependencies + +```console +npm ci +``` + +3. Build the server + +```console +npm run ensure-env && npm run build:server +``` + +4. Start Instances + +```console +pm2 start all --update-env && pm2 logs +``` + +### 2. Rolling updates - Used for logical changes to code. + +```console +pm2 reload all --update-env && pm2 logs +``` + +> [!NOTE] +> We are handling rolling updates to code, logic, via pipelines. You do not need to run these commands. These are here for documentation. diff --git a/docs/flight-manuals/working-on-web-server-instances.md b/docs/flight-manuals/working-on-web-server-instances.md deleted file mode 100644 index d098dcfb91..0000000000 --- a/docs/flight-manuals/working-on-web-server-instances.md +++ /dev/null @@ -1,57 +0,0 @@ -# Work on Web Server instances - -We are running load balanced (Azure Load Balancer) instances for our web servers. These servers are running NGINX which reverse proxy all of the traffic to freeCodeCamp.org from various applications running on their own infrastructures. - -The NGINX config is available on [this repository](https://github.com/freeCodeCamp/nginx-config). - -## Installing pre-requisites - -Perform updates to the OS packages by following [this guide](flight-manuals/working-on-virtual-machines). - -## First install - -> TODO - Detailed Instructions - -1. Provision a VM on Azure. -2. Install NGINX and configure from repository. -3. Inatall Cloudflare origin certificates and upstream application config. -4. Setup networking and firewalls. -5. Add the VM to the load balancer backend pool. - - -## Logging and Monitoring - -1. Check status for NGINX service using the below command: - -```console -sudo systemctl status nginx -``` - -2. Logging and monitoring for the servers are available at: - ->

https://amplify.nginx.com

- -## Updating Instances (Maintenance) - -Config changes to our NGINX instances are maintained on GitHub, these should be deployed on each instance like so: - -1. SSH into the instance and enter sudo - -```console -sudo su -``` - -2. Get the latest config code. - -```console -cd /etc/nginx -git fetch --all --prune -git reset --hard origin/master -``` - -3. Test and reload the config [with Signals](https://docs.nginx.com/nginx/admin-guide/basic-functionality/runtime-control/#controlling-nginx). - -```console -nginx -t -nginx -s reload -```