miel.truyen/nx
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

docs(core): building blocks of fast CI (#20539)

Browse Source
This commit is contained in:
Isaac Mann 2023-12-07 09:07:18 -05:00 committed by GitHub
parent 308eeb706d
commit be821aa15e
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
8 changed files with 94 additions and 16 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

24
docs/generated/manifests/ci.json
View File

@ -229,9 +229,19 @@
"description": "Learn how to manage Nx Cloud subscriptions and other options.",
"file": "",
"itemList": [
{
"id": "building-blocks-fast-ci",
"name": "The Building Blocks of Fast CI",
"description": "",
"file": "nx-cloud/concepts/building-blocks-fast-ci",
"itemList": [],
"isExternal": false,
"path": "/ci/concepts/building-blocks-fast-ci",
"tags": []
},
{
"id": "reduce-waste",
"name": "Reduce Waste in CI",
"name": "Reduce Wasted Time in CI",
"description": "",
"file": "nx-cloud/concepts/reduce-waste",
"itemList": [],
@ -264,9 +274,19 @@
"path": "/ci/concepts",
"tags": []
},
"/ci/concepts/building-blocks-fast-ci": {
"id": "building-blocks-fast-ci",
"name": "The Building Blocks of Fast CI",
"description": "",
"file": "nx-cloud/concepts/building-blocks-fast-ci",
"itemList": [],
"isExternal": false,
"path": "/ci/concepts/building-blocks-fast-ci",
"tags": []
},
"/ci/concepts/reduce-waste": {
"id": "reduce-waste",
"name": "Reduce Waste in CI",
"name": "Reduce Wasted Time in CI",
"description": "",
"file": "nx-cloud/concepts/reduce-waste",
"itemList": [],

20
docs/generated/manifests/menus.json
View File

@ -5706,7 +5706,15 @@
"isExternal": false,
"children": [
{
"name": "Reduce Waste in CI",
"name": "The Building Blocks of Fast CI",
"path": "/ci/concepts/building-blocks-fast-ci",
"id": "building-blocks-fast-ci",
"isExternal": false,
"children": [],
"disableCollapsible": false
},
{
"name": "Reduce Wasted Time in CI",
"path": "/ci/concepts/reduce-waste",
"id": "reduce-waste",
"isExternal": false,
@ -5733,7 +5741,15 @@
"disableCollapsible": false
},
{
"name": "Reduce Waste in CI",
"name": "The Building Blocks of Fast CI",
"path": "/ci/concepts/building-blocks-fast-ci",
"id": "building-blocks-fast-ci",
"isExternal": false,
"children": [],
"disableCollapsible": false
},
{
"name": "Reduce Wasted Time in CI",
"path": "/ci/concepts/reduce-waste",
"id": "reduce-waste",
"isExternal": false,

8
docs/map.json
View File

@ -1702,7 +1702,13 @@
"description": "Learn how to manage Nx Cloud subscriptions and other options.",
"itemList": [
{
"name": "Reduce Waste in CI",
"name": "The Building Blocks of Fast CI",
"tags": [],
"id": "building-blocks-fast-ci",
"file": "nx-cloud/concepts/building-blocks-fast-ci"
},
{
"name": "Reduce Wasted Time in CI",
"tags": [],
"id": "reduce-waste",
"file": "nx-cloud/concepts/reduce-waste"

35
docs/nx-cloud/concepts/building-blocks-fast-ci.md Normal file
View File

@ -0,0 +1,35 @@
# Building Blocks of Fast CI
Nx has many features that make your CI faster. Each of these features speeds up your CI in a different way, so that enabling an individual feature will have an immediate impact. These features are also designed to complement each other so that you can use them together to create a fully optimized CI pipeline.
## Use Fast Build Tools
The purpose of a CI pipeline is to run tasks like `build`, `test`, `lint` and `e2e`. You use different tools to run these tasks (like Webpack or Vite for you `build` task). If the individual tasks in your CI pipeline are slow, then your overall CI pipeline will be slow. Nx has two ways to help with this.
Nx provides plugins for popular tools that make it easy to update to the latest version of that tool and [automatically updates](/core-features/automate-updating-dependencies) your configuration files to take advantage of enhancements in the tool. The tool authors are always looking for ways to improve their product and the best way to get the most out of the tool you're using is to make sure you're on the latest version. Also, the recommended configuration settings for a tool will change over time so even if you're on the latest version of a tool, you may be using a slower version of it because you don't know about a new configuration setting. [`nx migrate`](/core-features/automate-updating-dependencies) will automatically change the default settings of in your tooling config to use the latest recommended settings so that your repo won't be left behind.
Because Nx plugins have a consistent interface for how they are invoked and how they interact with the codebase, it is easier to try out a different tool to see if it is better than what you're currently using. Newer tools that were created with different technologies or different design decisions can be orders of magnitude faster than your existing tools. Or the new tool might not help your project. Browse through the [list of Nx plugins](/plugin-registry), like [vite](/nx-api/vite) or [rspack](/nx-api/rspack), and try it out on your project with the default settings already configured for you.
## Reduce Wasted Time
In a monorepo, most PRs do not affect the entire codebase, so there's no need to run every test in CI for that PR. Nx provides the [`nx affected`](/ci/features/affected) command to make sure that only the tests that need to be executed are run for a particular PR.
Even if a particular project was affected by a PR, this could be the third time this same PR was run through CI and the build for this project was already run for this same exact set of files twice before. If you enable [remote caching](/ci/features/remote-cache), you can make sure that you never run the same command on the same code twice.
For a more detailed analysis of how these features reduce wasted time in different scenarios, read the [Reduce Wasted Time in CI guide](/ci/concepts/reduce-waste)
## Parallelize and Distribute Tasks Efficiently
Every time you use Nx to run a task, Nx will attempt to run the task and all its dependent tasks in parallel in the most efficient way possible. Because Nx knows about [task pipelines](/concepts/task-pipeline-configuration), it can run all the prerequisite tasks first. Nx will automatically run tasks in parallel processes up to the limit defined in the `parallel` property in `nx.json`.
There's a limit to how many tasks can be run in parallel on the same machine, but the logic that Nx uses to assign tasks to parallel processes can also be used by Nx Cloud to efficiently [distribute tasks across multiple agent machines](/ci/features/distribute-task-execution). Once those tasks are run, the [remote cache](/ci/features/remote-cache) is used to replay those task results on the main machine. After the pipeline is finished, it looks like all the tasks were run on a single machine - but much faster than a single machine could do it.
For a detailed analysis of different strategies for running tasks concurrently, read the [Parallelization and Distribution guide](/ci/concepts/parallelization-distribution)
## More CI Performance Features Coming Soon
The Nx team is continuing to push the limits of how fast CI can be.
Soon, you'll be able to use [Nx Agents](/ci/features/nx-agents) to have Nx Cloud spin up agents for you based on how large your PR is. [Nx Agents](/ci/features/nx-agents) will also allow you to automatically identify and re-run flaky tests in their own agent. Later, Nx Workflows will enable you to use Nx Cloud directly as your CI provider, which will open up the possibility for more optimizations.
{% call-to-action title="Sign Up for Early Access" icon="nxcloud" description="Experience Nx Cloud Agents for yourself" url="https://cloud.nx.app/workflows-early-access" /%}

2
docs/nx-cloud/concepts/cache-security.md
View File

@ -1,4 +1,4 @@
# Remote Cache Security
# Cache Security
{% callout type="warning" title="Use Caution With Read-Write Tokens" %}
Read-write tokens allow full write access to your remote cache. They should only be used in trusted environments. For instance, open source projects should only use read-write tokens as secrets configured for protected branches (e.g, main). Read-only tokens should be used in all other cases.

14
docs/nx-cloud/concepts/reduce-waste.md
View File

@ -1,10 +1,10 @@
# Reduce Waste in CI
# Reduce Wasted Time in CI
This article explores two ways that Nx improves the average speed of your CI pipeline - `nx affected` and remote caching. Using the `nx affected` command speeds up the first CI run for a PR and remote caching speeds up every CI run after that. Both `nx affected` and remote caching provide more benefits to repositories with more projects and a flatter project structure.
For this discussion, we'll assume you understand the Nx [mental model](/concepts/mental-model) and have an understanding of both [what the affected command is](/ci/features/affected) and [how caching works](/concepts/how-caching-works).
## Reduce Waste With Affected
## Reduce Wasted Time With Affected
The `nx affected` command allows you to only run tasks on projects that were affected by a particular PR. This effectively eliminates the wasted time and resources that would have been spent executing tasks on projects that were unrelated to a particular PR.
@ -265,8 +265,8 @@ Note that the 50% chance of any project being modified is an arbitrary number. I
**Stacked:**
ℙa(1) = ℙm(1) + ℙm'(1) _ ℙm(2) + ℙm'(1) _ ℙm'(2) _ ℙm(3) = 0.5 + 0.25 + 0.125 = 0.875
ℙa(2) = ℙm(2) + ℙm'(2) _ ℙm(3) = 0.5 + 0.25 = 0.75
ℙa(1) = ℙm(1) + ℙm'(1) \* ℙm(2) + ℙm'(1) \* ℙm'(2) \* ℙm(3) = 0.5 + 0.25 + 0.125 = 0.875
ℙa(2) = ℙm(2) + ℙm'(2) \* ℙm(3) = 0.5 + 0.25 = 0.75
ℙa(3) = ℙm(3) = 0.5 = 0.5
**Expected Number of Affected Projects:**
@ -274,8 +274,8 @@ Note that the 50% chance of any project being modified is an arbitrary number. I
**Grouped:**
ℙa(1) = ℙm(1) + ℙm'(1) _ ℙm(3) = 0.5 + 0.25 = 0.75
ℙa(2) = ℙm(2) + ℙm'(2) _ ℙm(3) = 0.5 + 0.25 = 0.75
ℙa(1) = ℙm(1) + ℙm'(1) \* ℙm(3) = 0.5 + 0.25 = 0.75
ℙa(2) = ℙm(2) + ℙm'(2) \* ℙm(3) = 0.5 + 0.25 = 0.75
ℙa(3) = ℙm(3) = 0.5 = 0.5
**Expected Number of Affected Projects:**
@ -292,7 +292,7 @@ Note that the 50% chance of any project being modified is an arbitrary number. I
{% /disclosure %}
## Reduce Waste with Remote Caching
## Reduce Wasted Time with Remote Caching
If you use a read/write token on developer machines, CI runs could be dramatically improved because they would be leveraging the work already done on the machine of the developer that pushed the PR. This set up requires you to have full trust in everyone who is capable of viewing the code base, which doesn't make sense for open source projects or for many organizations.

4
docs/nx-cloud/intro/ci-with-nx.md
View File

@ -5,8 +5,8 @@ When implemented well, continuous integration (CI) allows a team of developers t
## Core Features
- Test only the code that might have been [affected](/ci/features/affected) by a PR
- Improve the average CI time with [remote caching](/ci/features/remote-cache)
- Improve the worst case CI time with [distributed task execution](/ci/features/distribute-task-execution)
- Never run the same task on the same code twice with [remote caching](/ci/features/remote-cache)
- Efficiently [distribute task execution across multiple machines](/ci/features/distribute-task-execution)
- Quickly troubleshoot errors that occur in CI
- Create a simple but powerful pipeline configuration that easily scales with your codebase

3
docs/shared/reference/sitemap.md
View File

@ -275,7 +275,8 @@
- [Set up Nx Cloud On-Premise](/ci/features/on-premise)
- [Nx Agents](/ci/features/nx-agents)
- [Concepts](/ci/concepts)
- [Reduce Waste in CI](/ci/concepts/reduce-waste)
- [The Building Blocks of Fast CI](/ci/concepts/building-blocks-fast-ci)
- [Reduce Wasted Time in CI](/ci/concepts/reduce-waste)
- [Parallelization and Distribution](/ci/concepts/parallelization-distribution)
- [Cache Security](/ci/concepts/cache-security)
- [Recipes](/ci/recipes)