miel.truyen/nx
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

docs(nx-cloud): rewrite dte docs

Browse Source
This commit is contained in:
Victor Savkin 2022-09-07 13:51:13 -04:00
parent 350f71d6ed
commit c334277a93
12 changed files with 207 additions and 87 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
docs/map.json
View File

@ -811,6 +811,11 @@
"name": "Set Up",
"id": "set-up",
"itemList": [
{
"name": "Adding Nx Cloud to an Nx Workspace",
"id": "add-nx-cloud",
"file": "nx-cloud/set-up/add-nx-cloud"
},
{
"name": "Set Up Distributed Caching",
"id": "set-up-caching",

4
docs/nx-cloud/account/google-auth.md
View File

@ -6,11 +6,11 @@ By default, when you invite a member by email, they can create a separate NxClou
If you'd like them to sign-in with Google directly, which ensures they automatically lose access to their NxCloud account if their email gets disabled, you need to enable this option when inviting them:
![Require Google OAuth Sign-In toggle](/nx-cloud/account/require_google-signin.png)
![Require Google OAuth Sign-In toggle](/nx-cloud/account/require_google_signin.png)
They will then only be able to accept the invite if they sign-in with Google directly:
![Log In with Google Button](/nx-cloud/account/google_oath.png)
![Log In with Google Button](/nx-cloud/account/google_oauth.png)
## SAML integration

4
docs/nx-cloud/intro/nx-cloud.md
View File

@ -2,9 +2,9 @@
## Save Time With Distributed Task Execution & Distributed Cache
Nx partitions a command into smaller tasks and runs them in parallel, in the correct order. Nx Cloud takes it one step further and [runs any command across multiple machines](/concepts/dte), while giving you a consolidated view of the command as if it ran locally.
Nx partitions a command into smaller tasks and runs them in parallel, in the correct order. Nx Cloud takes it one step further and [runs any command across multiple machines](/nx-cloud/set-up/set-up-dte), while giving you a consolidated view of the command as if it ran locally.
Nx caches the output of any previously run command such as testing and building, so it can replay the cached results instead of rerunning it. Nx Cloud allows you to [share the computation cache](/core-features/cache-task-results#distributed-computation-caching) across everyone in your team and CI.
Nx caches the output of any previously run command such as testing and building, so it can replay the cached results instead of rerunning it. Nx Cloud allows you to [share the computation cache](/nx-cloud/set-up/set-up-caching) across everyone in your team and CI.
And it takes five minutes to set up.

54
docs/nx-cloud/set-up/add-nx-cloud.md Normal file
View File

@ -0,0 +1,54 @@
# Adding Nx Cloud to an Nx Workspace
## Adding Nx Cloud When Creating a Workspace
You can enable Nx Cloud when creating a new workspace with `npx create-nx-workspace`.
```bash
? Set up distributed caching using Nx Cloud (It's free and doesn't require registration.)
(Use arrow keys)
❯ Yes [Faster builds, run details, Github integration. Learn
more at https://nx.app]
No
```
This will connect your workspace to Nx Cloud, so you can start testing it out right away.
## Adding Nx Cloud to an Existing Workspace
If you are on the latest version of Nx, you can connect your existing Nx workspace to Nx Cloud by
running `nx connect-to-nx-cloud`.
If you are using an older version of Nx (`< 12.0.0`), you can connect your workspace by adding the `@nrwl/nx-cloud`
package to your workspace and running `nx g @nrwl/nx-cloud:init`.
## Connecting Your Workspace to Your Nx Cloud Account
After you have enabled Nx Cloud in your workspace, you will see the following:
```bash
> NX NOTE Nx Cloud has been enabled
Your workspace is currently public. Anybody with code access
can view the workspace on nx.app.
You can connect the workspace to your Nx Cloud account at
https://nx.app/orgs/workspace-setup?accessToken=N2Y3NzcyO...
(You can do this later.)
```
Click on this link to associate the workspace with your Nx Cloud account. If you don't have an Nx Cloud account, you can
create one on the spot.
After you claim your workspace, you will be able to manage permissions, create access tokens, set up billing, and so
forth.
**You will also see an interactive tutorial helping you explore distributed caching and the Nx Cloud user interface.**
If you lose this link, you can still connect your workspace to Nx Cloud. Go to [nx.app](https://nx.app), create an
account, and connect your workspace using the access token from `nx.json`.
## Skipping Cloud
Similar to how `--skip-nx-cache` will instruct Nx not to use the cache, passing `--no-cloud` will tell Nx not to use Nx
Cloud.

49
docs/nx-cloud/set-up/set-up-caching.md
View File

@ -1,42 +1,19 @@
# Adding Nx Cloud to an Nx Workspace
# Enabling Distributed Caching
## Adding Nx Cloud When Creating a Workspace
After connecting to Nx Cloud, you will immediately start taking advantage of the distributed caching. To see this in
action
run:
You can enable Nx Cloud when creating a new workspace with `npx create-nx-workspace`.
- `nx build YOUR_APP_NAME`
- `nx reset` to remove local cache
- `nx build YOUR_APP_NAME` again to fetch the file artifacts and the terminal output from the cloud
```bash
? Set up distributed caching using Nx Cloud (It's free and doesn't require registration.)
(Use arrow keys)
❯ Yes [Faster builds, run details, Github integration. Learn
more at https://nx.app]
No
```
## Skipping Nx Cloud Cache
This will connect your workspace to Nx Cloud, so you can start testing it out right away.
Similar to how `--skip-nx-cache` will instruct Nx not to use the cache, passing `--no-cloud` will tell Nx not to use Nx
Cloud, only local cache will be used.
## Adding Nx Cloud to an Existing Workspace
### Relevant Documentation
If you are on the latest version of Nx, you can connect your existing Nx workspace to Nx Cloud by running `nx connect-to-nx-cloud`.
If you are using an older version of Nx (`< 12.0.0`), you can connect your workspace by adding the `@nrwl/nx-cloud` package to your workspace and running `nx g @nrwl/nx-cloud:init`.
## Connecting Your Workspace to Your Nx Cloud Account
After you have enabled Nx Cloud in your workspace, you will see the following:
```bash
> NX NOTE Nx Cloud has been enabled
Your workspace is currently public. Anybody with code access
can view the workspace on nx.app.
You can connect the workspace to your Nx Cloud account at
https://nx.app/orgs/workspace-setup?accessToken=N2Y3NzcyO...
(You can do this later.)
```
Click on this link to associate the workspace with your Nx Cloud account. If you don't have an Nx Cloud account, you can create one on the spot.
After you claim your workspace, you will be able to manage permissions, create access tokens, set up billing, and so forth.
If you lose this link, you can still connect your workspace to Nx Cloud. Go to [nx.app](https://nx.app), create an account, and connect your workspace using the access token from `nx.json`.
- [Cache Task Results](/core-features/cache-task-results)
- [How Caching Works](/concepts/how-caching-works)

147
docs/nx-cloud/set-up/set-up-dte.md
View File

@ -1,46 +1,131 @@
# Set Up Distributed Task Execution
To enable distributed task execution, the following has to be true:
> Make sure to use the latest version of `@nrwl/nx-cloud`. The latest version works with any version of Nx >= 13.0.
1. `NX_CLOUD_DISTRIBUTED_EXECUTION` has to be set to `true`.
2. Agents must be started using the nx-cloud start-agent command, and must be stopped at the end using `nx-cloud stop-all-agents`.
## How It Works
The examples below use 3 agents to run all builds, tests, and lints.
When using Nx Cloud Distributed Task Execution (DTE), you split your CI setup into two parts:
Note that only cacheable operations can be distributed because they have to be replayed on the main job.
- The main job that controls what is going to be executed
- The agent jobs that actually execute the tasks
You don't have to distribute every single command. For instance, it's pretty common to send test coverage reports from a single machine. You can do it as follows:
The main job execution flow looks like this:
```yaml
- run: npx nx affected --target=test --parallel --max-parallel=2
- run: NX_CLOUD_DISTRIBUTED_EXECUTION=false npx nx affected --target=send-coverage-reports --parallel
# Coordinate the agents to run the tasks
- npx nx-cloud start-ci-run
# Run any commands you want here
- nx affected --target=lint
- nx affected --target=test
- nx affected --target=build
# Stop any run away agents
- npx nx-cloud stop-all-agents
```
Distributed task execution will copy all artifacts from agents to the main job, so after the `affected --target=test` command completes, all the coverage reports will be on the main job.
The agent job execution flow is very simple:
Note, the `--parallel` flag is propagated to Nx Cloud agents.
```yaml
# Wait for tasks to execute
- npx nx-cloud start-agent
```
## Agents
The main job looks more or less the same way as if you haven't used any distribution. The only thing you need to do is
to invoke `npx nx-cloud start-ci-run` at the beginning and optionally invoke `npx nx-cloud stop-all-agents` at the end.
In the examples below, the config creates one main job and three agent jobs. All four jobs have the same environment and the same source code. The four jobs will start around the same time. And, once the main job completes, all the agents will be stopped.
The agent jobs run long-running `start-agent` processes that execute all the tasks associated with a given CI run. The
only thing you need to do to set them up is to invoke `npx nx-cloud start-agent`. This process will keep running until
Nx Cloud tells it to terminate.
It's important to note that an Nx Cloud agent isn't a machine but rather a long-running process that runs on a machine. In the examples below, the config provisions three machines that run `nx-cloud start-agent`. Nx Cloud does not start agents. It simply coordinates work between the agents that you start in your CI. So you are in control of how many agents you want to use and where they are located.
> Note it's important that the main job and the agent jobs have the same environment and the same source code. They start
> around the same time. And, once the main job completes, all the agents
> will be stopped.
## Additional Configuration Options
It's also important to note that an Nx Cloud agent isn't a machine but rather a long-running process that runs on a
machine. I.e., Nx Cloud doesn't manage your agents--you need to do it in your CI config (check out CI examples below).
Setting `NX_CLOUD_DISTRIBUTED_EXECUTION_AGENT_COUNT` to say 8 will tell Nx Cloud assume that there are 8 agents running.
Nx Cloud is an orchestrator. The main job tells Nx Cloud what you want to run, and Nx Cloud will distribute those tasks
across the agents. Nx Cloud will automatically move files from one agent to another, from the agents to the main job.
Setting `NX_CLOUD_DISTRIBUTED_EXECUTION_STOP_AGENTS_ON_FAILURE` to false will tell Nx Cloud not to stop agents if a command fails. You need to make sure to invoke `nx-cloud stop-all-agents` even if CI fails.
The end result is that when say `nx affected --target=build` completes on the main job, all the file artifacts created
on agents are copied over to the main job, as if the main job had built everything locally.
## Distributed Task Execution and Sharding
### Relevant Documentation
The Nx Cloud distributed task execution model is akin to what is used at Google or Facebook. It scales better and provided more flexibility than sharding tasks across agents. [Read more about Distributed Task Execution and how it compares to binning/sharding.](https://blog.nrwl.io/distributing-ci-binning-and-distributed-task-execution-632fe31a8953?source=friends_link&sk=5120b7ff982730854ed22becfe7a640a)
- [Distribute Task Execution](/core-features/distribute-task-execution)
- [Distributed Task Execution Illustrated Guide](/concepts/dte).
## npx nx-cloud start-ci-run
At the beginning of your main job, invoke `npx nx-cloud start-ci-run`. This tells Nx Cloud that the following series of
command correspond to the same CI run.
You can configure your CI run by passing the following flags:
### --use-dte-by-default
By default, invoking `npx nx-cloud start-ci-run` will configure Nx to distribute all commands by default. You can
disable this as follows: `npx nx-cloud start-ci-run --use-dte-by-default=false`.
### --stop-agents-on-failure
By default, a failure in one of the commands is going to terminate the whole CI run and will stop all the
agents. You can disable this as follows: `npx nx-cloud start-ci-run --stop-agents-on-failure=false`.
### --stop-agents-after
By default, Nx Cloud won't terminate any agents until you invoke `npx nx-cloud stop-all-agents` because Nx Cloud
doesn't know if you will need agents to run another command. This can result in agents being idle at the end of a CI
run.
You can fix it by telling Nx Cloud that it can terminate agents after it sees a certain
target: `npx nx-cloud start-ci-run --stop-agents-after=e2e`.
> Earlier versions of `@nrwl/nx-cloud` required you to set the `NX_CLOUD_DISTRIBUTED_EXECUTION` env variable to `true`
> to
> enable DTE, but in the latest version `npx nx-cloud start-ci-run` does it automatically.
## Enabling/Disabling DTE
Invoking `npx nx-cloud start-ci-run` will tell Nx to distribute by default. You can enable/disable distribution for
individual commands as follows:
- `nx affected --target=build --dte` (explicitly enable distribution, Nx >= 14.7)
- `nx affected --target=build --no-dte` (explicitly disable distribution, Nx >= 14.7)
- `NX_CLOUD_DISTRIBUTED_EXECUTION=true nx affected --target=build` (explicitly enable distribution)
- `NX_CLOUD_DISTRIBUTED_EXECUTION=false nx affected --target=build` (explicitly disable distribution)
## npx nx-cloud stop-all-agents
This command tells Nx Cloud to terminate all agents associated with this CI run.
## Running Things in Parallel
`--parallel` is propagated to the agents. E.g., `npx nx affected --target=build --parallel=3 --dte` tells Nx Cloud to run
up to 3 build targets in parallel on each agent. So if you have say 10 agents, you will run up to 30 builds in parallel
across all of them.
You also want to run as many commands in parallel as you can. For instance,
```yaml
- nx affected --target=build
- nx affected --target=test
- nx affected --target=lint
```
is worse than
```yaml
- nx affected --target=build & nx affected --target=test & nx affected --target=lint
```
The latter is going to schedule all the three commands at the same time, so if an agent cannot find anything to build, it will start running tests and lints. The result is better agent utilization and shorter CI time.
## CI/CD Examples
The examples below show how to set up CI using Nx and Nx Cloud using distributed task execution and distributed caching.
Every organization manages their CI/CD pipelines differently, so the examples don't cover org-specific aspects of CI/CD (e.g., deployment). They mainly focus on configuring Nx correctly.
Every organization manages their CI/CD pipelines differently, so the examples don't cover org-specific aspects of
CI/CD (e.g., deployment). They mainly focus on configuring Nx correctly.
Read the guides for more information on how to configure them in CI.
@ -50,18 +135,30 @@ Read the guides for more information on how to configure them in CI.
- [GitHub Actions](/recipe/monorepo-ci-github-actions#distributed-ci-with-nx-cloud)
- [Jenkins](/recipe/monorepo-ci-jenkins#distributed-ci-with-nx-cloud)
Note that only cacheable operations can be distributed because they have to be replayed on the main job.
## Additional Notes
### Env Variables
The `@nrwl/nx-cloud` requires the `NX_BRANCH` environment variables to be set. For many CI providers (e.g., GitHub Actions), the runner is able to set it automatically. For others, the variable will have to be set manually. If you set it manually, note that `NX_BRANCH` has to be set to a PR number for the GitHub integration to work.
- The `@nrwl/nx-cloud` requires the `NX_BRANCH` environment variables to be set. For many CI providers (e.g., GitHub
Actions), the runner is able to set it automatically. For others, the variable will have to be set manually. If you set
it manually, note that `NX_BRANCH` has to be set to a PR number for the GitHub integration to work.
Setting `NX_CLOUD_DISTRIBUTED_EXECUTION` to true enables distributed task execution.
- The `@nrwl/nx-cloud` requires the `NX_RUN_GROUP` environment variables to be set. For many CI providers (e.g., GitHub
Actions), the runner is able to set it automatically. For others, the variable will have to be set manually. If you set
it manually, note that `NX_RUN_GROUP` has to be a unique value associated with a CI run.
Setting `NX_CLOUD_DISTRIBUTED_EXECUTION_AGENT_COUNT` to say 8 will tell Nx Cloud assume that there are 8 agents running.
- Setting `NX_CLOUD_DISTRIBUTED_EXECUTION` to true enables distributed task execution.
Setting `NX_CLOUD_DISTRIBUTED_EXECUTION_STOP_AGENTS_ON_FAILURE` to true will tell Nx Cloud not to stop agents if a command fails. You need to make sure to invoke nx-cloud stop-all-agents even if CI fails.
- Setting `NX_VERBOSE_LOGGING` to true will output the debug information about agents communicating with the main job.
It's a good way to troubleshoot issues.
Setting `NX_VERBOSE_LOGGING` to true will output the debug information about agents communicating with the main job. It's a good way to troubleshoot issues.
- Setting `NX_CLOUD_ENV_NAME` will prefix all your commands so you can easily distinguish them in the UI and in GitHub
comments. For instance, if you run the same set of commands on Windows and Linux machines, you can
set `NX_CLOUD_ENV_NAME` to win on the Windows agent, and linux on Linux agents.
Setting `NX_CLOUD_ENV_NAME` will prefix all your commands so you can easily distinguish them in the UI and in GitHub comments. For instance, if you run the same set of commands on Windows and Linux machines, you can set `NX_CLOUD_ENV_NAME` to win on the Windows agent, and linux on Linux agents.
## Relevant Repositories and Examples
- [Nx: On how to make your CI 16 times faster with a small config change](https://github.com/vsavkin/interstellar)
- ["Lerna & Distributed Task Execution" Example](https://github.com/vsavkin/lerna-dte)

5
docs/shared/core-features/cache-task-results.md
View File

@ -80,10 +80,11 @@ You can connect your workspace to Nx Cloud by running:
npx nx connect-to-nx-cloud
```
Learn more about Nx Cloud at [https://nx.app](https://nx.app).
## Related Documentation
- [Nx Cloud Documentation](/nx-cloud/intro/what-is-nx-cloud)
- [Nx Cloud Main Site](https://nx.app)
### Concepts
- [How Caching Works](/concepts/how-caching-works)

6
docs/shared/monorepo-ci-azure.md
View File

@ -56,7 +56,6 @@ pr:
variables:
CI: 'true'
NX_CLOUD_DISTRIBUTED_EXECUTION: 'true'
${{ if eq(variables['Build.Reason'], 'PullRequest') }}:
NX_BRANCH: $(System.PullRequest.PullRequestNumber)
TARGET_BRANCH: $[replace(variables['System.PullRequest.TargetBranch'],'refs/heads/','origin/')]
@ -83,16 +82,13 @@ jobs:
vmImage: 'ubuntu-latest'
steps:
- script: npm ci
- script: npx nx-cloud start-ci-run
- script: npx nx-cloud start-ci-run --stop-agents-after="build"
- script: npx nx-cloud record -- npx nx workspace-lint
- script: npx nx-cloud record -- npx nx format:check --base=$(BASE_SHA) --head=$(HEAD_SHA)
- script: npx nx affected --base=$(BASE_SHA) --head=$(HEAD_SHA) --target=lint --parallel=3
- script: npx nx affected --base=$(BASE_SHA) --head=$(HEAD_SHA) --target=test --parallel=3 --ci --code-coverage
- script: npx nx affected --base=$(BASE_SHA) --head=$(HEAD_SHA) --target=build --parallel=3
- script: npx nx-cloud stop-all-agents
condition: always()
```
You can also use our [ci-workflow generator](/packages/workspace/generators/ci-workflow) to generate the pipeline file.

7
docs/shared/monorepo-ci-circle-ci.md
View File

@ -67,22 +67,17 @@ jobs:
main:
docker:
- image: cimg/node:lts-browsers
environment:
NX_CLOUD_DISTRIBUTED_EXECUTION: 'true'
steps:
- checkout
- run: npm ci
- nx/set-shas
- run: npx nx-cloud start-ci-run
- run: npx nx-cloud start-ci-run --stop-agents-after="build"
- run: npx nx-cloud record -- npx nx workspace-lint
- run: npx nx-cloud record -- npx nx format:check
- run: npx nx affected --base=$NX_BASE --head=$NX_HEAD --target=lint --parallel=3
- run: npx nx affected --base=$NX_BASE --head=$NX_HEAD --target=test --parallel=3 --ci --code-coverage
- run: npx nx affected --base=$NX_BASE --head=$NX_HEAD --target=build --parallel=3
- run: npx nx-cloud stop-all-agents
when: always
workflows:
build:
jobs:

4
docs/shared/monorepo-ci-gitlab.md
View File

@ -73,7 +73,6 @@ Read more about the [Distributed CI setup with Nx Cloud](/recipes/ci-setup#distr
image: node:18
variables:
CI: 'true'
NX_CLOUD_DISTRIBUTED_EXECUTION: 'true'
# Creating template for DTE agents
.dte-agent:
@ -118,14 +117,13 @@ nx-dte:
stage: affected
extends: .base-pipeline
script:
- yarn nx-cloud start-ci-run
- yarn nx-cloud start-ci-run --stop-agents-after="build"
- yarn nx-cloud record -- yarn nx workspace-lint --base=$NX_BASE --head=$NX_HEAD
- yarn nx-cloud record -- yarn nx format:check --base=$NX_BASE --head=$NX_HEAD
- yarn nx affected --base=$NX_BASE --head=$NX_HEAD --target=lint --parallel=3
- yarn nx affected --base=$NX_BASE --head=$NX_HEAD --target=test --parallel=3 --ci --code-coverage
- yarn nx affected --base=$NX_BASE --head=$NX_HEAD --target=e2e --parallel=3 --ci --code-coverage
- yarn nx affected --base=$NX_BASE --head=$NX_HEAD --target=build --parallel=3
- yarn nx-cloud stop-all-agents
# Create as many agents as you want
nx-dte-agent1:

7
docs/shared/monorepo-ci-jenkins.md
View File

@ -64,7 +64,6 @@ pipeline {
agent none
environment {
NX_BRANCH = env.BRANCH_NAME.replace('PR-', '')
NX_CLOUD_DISTRIBUTED_EXECUTION = true
}
stages {
stage('Pipeline') {
@ -76,13 +75,12 @@ pipeline {
agent any
steps {
sh "npm ci"
sh "npx nx-cloud start-ci-run"
sh "npx nx-cloud start-ci-run --stop-agents-after='build'"
sh "npx nx workspace-lint"
sh "npx nx format:check"
sh "npx nx affected --base=HEAD~1 --target=lint --parallel=3"
sh "npx nx affected --base=HEAD~1 --target=test --parallel=3 --ci --code-coverage"
sh "npx nx affected --base=HEAD~1 --target=build --parallel=3"
sh "npx nx-cloud stop-all-agents"
}
}
stage('PR') {
@ -92,13 +90,12 @@ pipeline {
agent any
steps {
sh "npm ci"
sh "npx nx-cloud start-ci-run"
sh "npx nx-cloud start-ci-run --stop-agents-after='build'"
sh "npx nx workspace-lint"
sh "npx nx format:check"
sh "npx nx affected --base origin/${env.CHANGE_TARGET} --target=lint --parallel=3"
sh "npx nx affected --base origin/${env.CHANGE_TARGET} --target=test --parallel=3 --ci --code-coverage"
sh "npx nx affected --base origin/${env.CHANGE_TARGET} --target=build --parallel=3"
sh "npx nx-cloud stop-all-agents"
}
}

2
nx-dev/nx-dev-e2e/src/integration/nx-cloud-documentation.spec.ts
View File

@ -8,7 +8,7 @@ describe('nx-dev: Nx Cloud section', () => {
(<{ title: string; path: string }[]>[
{ title: 'What is Nx Cloud?', path: '/nx-cloud/intro/what-is-nx-cloud' },
{
title: 'Adding Nx Cloud to an Nx Workspace',
title: 'Enabling Distributed Caching',
path: '/nx-cloud/set-up/set-up-caching',
},
{