gaspersic/freeCodeCamp
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

docs: update links and workflow (#38457)

Browse Source
This commit is contained in:
mrugesh
2020-03-27 13:53:32 +05:30
committed by GitHub
parent 3ee1c39e41
commit a45c2c4f15
1 changed files with 72 additions and 74 deletions
Download Patch File Download Diff File Expand all files Collapse all files

146
docs/devops.md
View File

@ -1,6 +1,9 @@
<!-- do not translate this -->
| [Read these guidelines in other languages](/docs/i18n-languages) |
|-|
| ---------------------------------------------------------------- |
<!-- do not translate this -->
# Developer Operations at freeCodeCamp.org
@ -13,25 +16,27 @@ Let us know, if you have feedback or queries, and we will be happy to clarify.
## How do we build, test and deploy the codebase?
Our codebase is continuously built, tested and deployed to **separate sets of infrastructure (Servers, Databases, CDNs, etc.)**.
Our codebase is continuously built, tested and deployed to **separate sets of infrastructure (Servers, Databases, CDNs, etc.)**.
This involves three steps to be followed in sequence:
First, new changes are merged into our primary development branch (`master`) in form of pull requests. Next, these changes are run through a series of automated tests. And finally, once the tests pass we release the changes (or update them if needed) to deployments on our infrastructure.
1. New changes (both fixes and features) are merged into our primary development branch (`master`) via pull requests.
2. These changes are run through a series of automated tests.
3. Once the tests pass we release the changes (or update them if needed) to deployments on our infrastructure.
### Building the codebase - Mapping Git Branches to Deployments.
#### Building the codebase - Mapping Git Branches to Deployments.
Typically, [`master`](https://github.com/freeCodeCamp/freeCodeCamp/tree/master) (the default development branch) is merged into the [`production-staging`](https://github.com/freeCodeCamp/freeCodeCamp/tree/production-staging) branch once a day and is released into an isolated infrastructure.
This is an intermediate release for our developers and volunteer contributors. It also known as our "staging/beta" application.
This is an intermediate release for our developers and volunteer contributors. It also known as our "staging" or "beta" release.
It is identical to our live production environment at `freeCodeCamp.org`, other than it using a separate set of databases, servers, web-proxies, etc. This isolation lets us test ongoing development and features in a "production" like scenario, without affecting regular users of freeCodeCamp.org's main platforms.
Once the developer team [`@freeCodeCamp/dev-team`](https://github.com/orgs/freeCodeCamp/teams/dev-team/members) is happy with the changes on the staging application, these changes are moved every few days to the [`production-current`](https://github.com/freeCodeCamp/freeCodeCamp/tree/production-current) branch.
Once the developer team [`@freeCodeCamp/dev-team`](https://github.com/orgs/freeCodeCamp/teams/dev-team/members) is happy with the changes on the staging platform, these changes are moved every few days to the [`production-current`](https://github.com/freeCodeCamp/freeCodeCamp/tree/production-current) branch.
This is the final release that moves changes to our production platforms on freeCodeCamp.org.
### Testing changes - Integration and User Acceptance Testing.
#### Testing changes - Integration and User Acceptance Testing.
We employ various levels of integration and acceptance testing to check on the quality of the code. All our tests are done through software like [Travis CI](https://travis-ci.org/freeCodeCamp/freeCodeCamp) and [Azure Pipelines](https://dev.azure.com/freeCodeCamp-org/freeCodeCamp).
@ -39,23 +44,17 @@ We have unit tests for testing our challenge solutions, Server APIs and Client U
> Note: We are also in the process of writing end user tests which will help in replicating real world scenarios like updating an email or making a call to the API or third-party services.
Together these tests help in preventing issues from repeating themselves and ensure we do not introduce a bug while working on another bug or a feature.
Together these tests help in preventing issues from repeating themselves and ensure we do not introduce a bug while working on another bug or a feature.
### Deploying Changes - Pushing changes to servers.
#### Deploying Changes - Pushing changes to servers.
We have configured continuous delivery software to push changes to our development and production servers. Once the changes are pushed to the protected release branches, these should trigger our build and release pipelines:
We have configured continuous delivery software to push changes to our development and production servers.
You can take a look and browse these here:
Once the changes are pushed to the protected release branches, a build pipeline is automatically triggered for the branch. The build pipelines are responsible for building artefacts and keeping them in a cold storage for later use.
| Build Pipeline | Release Pipeline |
| :------------- | :--------------- |
| Setup to build artifacts for deployments. | Setup to deploy artifacts to their destination servers. |
| [Go to builds](https://dev.azure.com/freeCodeCamp-org/freeCodeCamp/_build) | [Go to releases](https://dev.azure.com/freeCodeCamp-org/freeCodeCamp/_release) |
The build pipeline goes on to trigger a corresponding release pipeline if it completes a successful run. The release pipelines are responsible for collecting the build artefacts, moving them to the servers and going live.
The build pipeline triggers the release pipeline after a hold of 5 minutes for our staff to go in and intervene if necessary.
The code/config is publicly accessible on Azure's Dev Dashboard. Write access to this is limited to the freeCodeCamp.org staff team.
We recommend not pushing more than 3-4 builds to the pipelines within a day and not more than one within the hour. This is because our artifacts are quite large and would put a load on our servers when deploying.
Status of builds and releases are [available here](#build-and-deployment-status).
## Triggering a build, test and deployment.
@ -68,8 +67,12 @@ Currently, only members on the developer team can push to the production branche
1. Configure your remotes correctly.
```sh
freeCodeCamp on master is 📦 v0.0.1 via ⬢ v10.16.0
❯ git remote -v
git remote -v
```
**Results:**
```
origin git@github.com:raisedadead/freeCodeCamp.git (fetch)
origin git@github.com:raisedadead/freeCodeCamp.git (push)
upstream git@github.com:freeCodeCamp/freeCodeCamp.git (fetch)
@ -86,7 +89,7 @@ Currently, only members on the developer team can push to the production branche
3. Check that the Travis CI is passing on the `master` branch for upstream.
The [continuous integration](https://travis-ci.org/freeCodeCamp/freeCodeCamp/branches) tests should be green and PASSING for the `master` branch.
The [continuous integration](https://travis-ci.com/github/freeCodeCamp/freeCodeCamp/branches) tests should be green and PASSING for the `master` branch.
<details>
<summary>
@ -112,19 +115,23 @@ Currently, only members on the developer team can push to the production branche
git push upstream
```
You will not be able to force push and if you have re-written the history in anyway these commands will error out. If they do, you may have done something incorrectly and you should just start over.
**Note:** You will not be able to force push and if you have re-written the history in anyway these commands will error out. If they do, you may have done something incorrectly and you should just start over.
And that's it, this will automatically trigger a build on the build pipeline for the `production-staging` branch. Typically this takes ~20-25 minutes for the all the applications. Once the build is complete, it will save the artifacts as `.zip` files in a cold storage to be retrieved and used by the release pipeline.
The above steps will automatically trigger a run on the build pipeline for the `production-staging` branch. Once the build is complete, the artifacts are saved as `.zip` files in a cold storage to be retrieved and used later.
The release pipeline automatically triggers itself when a fresh artifact is available from the connected build pipeline. For the staging applications this is completely automated and the artifacts are pushed to the client CDN and the API servers. They typically take ~15-20 mins for the client, and ~5 mins for the API servers to be available live.
The release pipeline is triggered automatically when a fresh artifact is available from the connected build pipeline. For staging platforms, this process does not involve manual approval and the artifacts are pushed to the Client CDN and API servers.
This makes each release from code push to being available on the staging applications ~60 mins.
> **Estimates:**
>
> Typically the build run takes ~20-25 minutes to complete followed by the release run which takes ~15-20 mins for the client, and ~5-10 mins for the API to be available live. From code push to being live on the staging platforms the whole process takes **~35-45 mins** in total.
### Pushing changes to Production Applications.
The process is mostly the same as the staging applications, with a few extra checks in place. This is just to make sure, we do not break anything on freeCodeCamp.org which can see hundreds of users using it at any moment.
The process is mostly the same as the staging platforms, with a few extra checks in place. This is just to make sure, we do not break anything on freeCodeCamp.org which can see hundreds of users using it at any moment.
| Do NOT execute these commands unless you have verified that everything is working on the staging platform. You should not bypass or skip any testing on staging before proceeding further. |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
> #### DO NOT execute these commands until you have verified that everything is working on the staging application. You should not bypass or skip any testing on staging before proceeding further.
1. Make sure your `production-staging` branch is pristine and in sync with the upstream.
@ -142,54 +149,45 @@ The process is mostly the same as the staging applications, with a few extra che
git push upstream
```
You will not be able to force push and if you have re-written the history in anyway these commands will error out. If they do, you may have done something incorrectly and you should just start over.
**Note:** You will not be able to force push and if you have re-written the history in anyway these commands will error out. If they do, you may have done something incorrectly and you should just start over.
And that's it, this will automatically trigger a build on the build pipeline for the `production-current` branch. Typically this also takes ~20-25 minutes for the all the applications like explained previously.
The above steps will automatically trigger a run on the build pipeline for the `production-current` branch. Once a build artifact is ready, it will trigger a run on the release pipeline.
Here are some additional steps that need to be followed by a freeCodeCamp.org Staff developer. To prevent any accidental pushes we have a couple of manual approval steps configured on the pipelines.
> **Estimates:** Typically the build run takes ~20-25 minutes to complete.
Once a build artifact is ready on the `production-current` branch, it will trigger a release on the release pipeline.
**Additional Steps for Staff Action**
Next, the freeCodeCamp.org developer staff team will receive an email. They can either *approve* or *reject* the release. If the changes are working nicely and have been tested on the staging application, then it can be approved. This must happen within 4 hours of the release being triggered or it will automatically get rejected. If this happens a staff member will need to re-trigger the release pipeline manually.
One a release run is triggered, members of the developer staff team will receive an automated manual intervention email. They can either _approve_ or _reject_ the release run.
If the changes are working nicely and have been tested on the staging platform, then it can be approved. The approval must be given within 4 hours of the release being triggered before getting rejected automatically. A staff can re-trigger the release run manually for rejected runs, or wait for the next cycle of release.
For staff use:
| Approve Release |
| :-------------: |
| Check your email for a direct link or [Open release dashboard](https://dev.azure.com/freeCodeCamp-org/freeCodeCamp/_release?_a=releases&view=mine&definitionId=6) |
Once one of the members approves a release, the pipeline will push the changes live to freeCodeCamp.org's production CDN and API servers. They typically take ~15-20 mins for the client, and ~5 mins for the API servers to be available live.
As a final step, a staff member will also manually click the publish deploy button on Netlify's deployment's dashboard.
For staff use:
| Publish or Rollback on Netlify |
| :----------------------------: |
| [Open Netlify deployments](https://app.netlify.com/sites/freecodecamp-org/deploys) |
| Check your email for a direct link or [go to the release dashboard](https://dev.azure.com/freeCodeCamp-org/freeCodeCamp/_release) after the build run is complete. |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
## Build and Deployment Status
Once one of the staff members approves a release, the pipeline will push the changes live to freeCodeCamp.org's production CDN and API servers. They typically take ~15-20 mins for the client, and ~5 mins for the API servers to be available live.
Here is the current build and deployment status of the codebase.
> **Estimates:**
>
> The release run typically takes ~15-20 mins for each client instance, and ~5-10 mins for each API instance to be available live. From code push to being live on the production platforms the whole process takes **~90-120 mins** in total (not counting the wait time for the staff approval).
### Build Status
## Test, Build and Deployment Status
| Platform | Type | Status |
| :-------------- | :--------- | :---------: |
| Travis CI | Unit Tests | [![Travis CI Build Status](https://travis-ci.org/freeCodeCamp/freeCodeCamp.svg?branch=master)](https://travis-ci.org/freeCodeCamp/freeCodeCamp) |
| Azure Pipelines | Artifacts | [![Azure Pipelines Build Status](https://dev.azure.com/freeCodeCamp-org/freeCodeCamp/_apis/build/status/freeCodeCamp-CI)](https://dev.azure.com/freeCodeCamp-org/freeCodeCamp/_build) |
Here is the current test, build and deployment status of the codebase.
### Deployment Status
| Type | Branch | Status | Dashboard |
| :--------------- | :------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------- |
| CI Tests | [`master`](https://github.com/freeCodeCamp/freeCodeCamp/tree/master) | ![Travis CI Build Status](https://travis-ci.com/freeCodeCamp/freeCodeCamp.svg?branch=master) | [Go to status dashboard](https://travis-ci.com/github/freeCodeCamp/freeCodeCamp/branches) |
| CI Tests | [`production-staging`](https://github.com/freeCodeCamp/freeCodeCamp/tree/production-staging) | ![Travis CI Build Status](https://travis-ci.com/freeCodeCamp/freeCodeCamp.svg?branch=production-staging) | [Go to status dashboard](https://travis-ci.com/github/freeCodeCamp/freeCodeCamp/branches) |
| CI Tests | [`production-current`](https://github.com/freeCodeCamp/freeCodeCamp/tree/production-current) | ![Travis CI Build Status](https://travis-ci.com/freeCodeCamp/freeCodeCamp.svg?branch=production-current) | [Go to status dashboard](https://travis-ci.com/github/freeCodeCamp/freeCodeCamp/branches) |
| Build Pipeline | [`production-staging`](https://github.com/freeCodeCamp/freeCodeCamp/tree/production-staging) | | [Go to status dashboard](https://dev.azure.com/freeCodeCamp-org/freeCodeCamp/_build) |
| Build Pipeline | [`production-current`](https://github.com/freeCodeCamp/freeCodeCamp/tree/production-staging) | | [Go to status dashboard](https://dev.azure.com/freeCodeCamp-org/freeCodeCamp/_build) |
| Release Pipeline | [`production-staging`](https://github.com/freeCodeCamp/freeCodeCamp/tree/production-staging) | | [Go to status dashboard](https://dev.azure.com/freeCodeCamp-org/freeCodeCamp/_release) |
| Release Pipeline | [`production-current`](https://github.com/freeCodeCamp/freeCodeCamp/tree/production-staging) | | [Go to status dashboard](https://dev.azure.com/freeCodeCamp-org/freeCodeCamp/_release) |
| Application | Version | Status |
| :----------- | :--------- | :---------: |
| Client | Beta/Next | [![Azure Pipelines Deployment Status](https://vsrm.dev.azure.com/freeCodeCamp-org/_apis/public/Release/badge/4b80aded-11d9-49ea-9b7d-596e98ff07c4/4/8)](https://dev.azure.com/freeCodeCamp-org/freeCodeCamp/_release) |
| API | Beta/Next | [![Azure Pipelines Deployment Status](https://vsrm.dev.azure.com/freeCodeCamp-org/_apis/public/Release/badge/4b80aded-11d9-49ea-9b7d-596e98ff07c4/4/9)](https://dev.azure.com/freeCodeCamp-org/freeCodeCamp/_release) |
| Client | Production | [![Azure Pipelines Deployment Status](https://vsrm.dev.azure.com/freeCodeCamp-org/_apis/public/Release/badge/4b80aded-11d9-49ea-9b7d-596e98ff07c4/6/22)](https://dev.azure.com/freeCodeCamp-org/freeCodeCamp/_release) |
| API | Production | [![Azure Pipelines Deployment Status](https://vsrm.dev.azure.com/freeCodeCamp-org/_apis/public/Release/badge/4b80aded-11d9-49ea-9b7d-596e98ff07c4/6/23)](https://dev.azure.com/freeCodeCamp-org/freeCodeCamp/_release) |
## Early access and beta testing upcoming versions of freeCodeCamp.org's platform and curriculum
## Early access and beta testing
We welcome you to test these releases in a **"public beta testing"** mode and get early access to upcoming features to the platforms. Sometimes these features/changes are referred to as **next, beta, staging,** etc. interchangeably.
@ -197,31 +195,31 @@ Your contributions via feedback and issue reports will help us in making the pro
We thank you for reporting bugs that you encounter and help in making freeCodeCamp.org better. You rock!
## Identifying the upcoming version of platform
### Identifying the upcoming version of the platforms
The domain name will be different than **`freeCodeCamp.org`**. Currently this public beta testing version is available at:
Currently a public beta testing version is available at:
<h3 align="center"><a href='https://www.freecodecamp.dev' _target='blank'><code>www.freecodecamp.dev</code></a></h4>
<h3 align="center"><a href='https://www.freecodecamp.dev' _target='blank'><code>www.freecodecamp.dev</code></a></h3>
To prevent accidental indexing on search engines and users accidentally using this site (without knowledge of it being a development site) is closed off with a simple password:
>**Note:** The domain name is different than **`freeCodeCamp.org`**. This is intentional to prevent search engine indexing and avoid confusion for regular users of the platform.
<h3 align="center"><code>freecodecamp-is-awesome</code></h4>
## Identifying the current version of platform
### Identifying the current version of the platforms
**The current version of the platform is always available at [`freeCodeCamp.org`](https://www.freecodecamp.org).**
The dev-team merges changes from the `production-staging` branch to `production-current` when they release changes. The top commit should be what you see live on the site. You can identify the exact version deployed by visiting the build and deployment logs available below in the status section.
The dev-team merges changes from the `production-staging` branch to `production-current` when they release changes. The top commit should be what you see live on the site.
## Known Limitations
You can identify the exact version deployed by visiting the build and deployment logs available in the status section. Alternatively you can also ping us in the [contributors chat room](https://gitter.im/FreeCodeCamp/Contributors) for a confirmation.
There will be some known limitations and tradeoffs when using the beta version of the platform.
### Known Limitations
- #### All data / personal progress on these beta applications `will NOT be saved or carried over` to production.
There are some known limitations and tradeoffs when using the beta version of the platform.
- #### All data / personal progress on these beta platforms `will NOT be saved or carried over` to production.
**Users on the beta version will have a separate account from the production.** The beta version uses a physically separate database from production. This gives us the ability to prevent any accidental loss of data or modifications. The dev team may purge the database on this beta version as needed.
- #### There are no guarantees on the uptime and reliability of the beta applications.
- #### There are no guarantees on the uptime and reliability of the beta platforms.
Deployment is expected to be frequent and in rapid iterations, sometimes multiple times a day. As a result there will be unexpected downtime at times or broken functionality on the beta version.