Skip to content

Commit a9c1f7f

Browse files
authored
document for incremental processing (#220)
1 parent 8bf72db commit a9c1f7f

File tree

4 files changed

+140
-2
lines changed

4 files changed

+140
-2
lines changed

README.md

+6-2
Original file line numberDiff line numberDiff line change
@@ -370,7 +370,7 @@ currently unavailable:
370370
## For Java Annotation Processor Authors
371371

372372
If you are familiar with the Java Annotation Processing API, see the
373-
[Java annotation processing to KSP reference](reference.md) for how to
373+
[Java annotation processing to KSP reference](docs/reference.md) for how to
374374
implement most functions and data structures that are provided by Java's
375375
API.
376376

@@ -528,8 +528,12 @@ and perhaps also resource directory if your IDE supports them:
528528
build/generated/ksp/src/main/resources
529529
```
530530

531+
## Incremental Processing
532+
533+
See [notes](docs/incremental.md) on incremental processing.
534+
531535
## Migration from old KSP releases
532-
See [Migration from old KSP releases](old-ksp-release.md)
536+
See [Migration from old KSP releases](docs/old-ksp-release.md)
533537

534538
## How to contribute
535539
See [CONTRIBUTING.md](CONTRIBUTING.md)

docs/incremental.md

+134
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
# Incremental Processing
2+
3+
Incremental processing is a way to avoid re-processing of sources as much as possible.
4+
The major goal is to reduce the turn-around time of a typical change-compile-test cycle.
5+
Here is a [wiki link](https://en.wikipedia.org/wiki/Incremental_computing) to the more general
6+
idea of incremental computation.
7+
8+
To be able to know which sources are dirty, i.e., those that need to be reprocessed, KSP needs
9+
processors' help to identify the correspondence of input sources and generated outputs.
10+
Because it is cumbersome and error prone to keep track of which inputs are involved in generating
11+
which outputs, KSP is designed to help with that and only require a minimum set of
12+
**sources that serve as roots that processors start to navigate the code structure**. In other
13+
words, a processor needs to associate an output with sources of those `KSNode`, if obtained from:
14+
* `Resolver.getAllFiles`
15+
* `Resolver.getSymbolsWithAnnotation`
16+
* `Resolver.getClassDesclarationByName`
17+
18+
Currently, only changes in Kotlin and Java sources are tracked. If there is a change in the
19+
classpath, namely in other modules or libraries, a full re-processing will be triggered.
20+
21+
Incremental processing is currently disabled by default. To enable it, set the Gradle property
22+
`ksp.incremental=true`. To enable logs, which dump the dirty set according to dependencies and
23+
outputs, use `ksp.incremental.log=true`. They can be found as `build/*.log`.
24+
25+
## Aggregating v.s. Isolating
26+
The idea is similar but slightly different to the [definition](https://docs.gradle.org/current/userguide/java_plugin.html#sec:incremental_annotation_processing)
27+
in Gradle annotation processing. In KSP,
28+
* *aggregating* / *isolating* is associated with each output, rather than the entire processor
29+
* an isolating output can have several sources
30+
* aggregating means that an output can be affected by any changes
31+
32+
If an output is `aggregating`, any changes may affect it potentially, except removal of files that
33+
don't affect other files.
34+
In other words, if there's a change, all `aggregating` outputs need to be regenerated and therefore
35+
all of their sources will be reprocessed. Note that only registered files and changed / new files
36+
will be re-processed.
37+
38+
For example, an output collecting all symbols with an interesting annotation is `aggregating`.
39+
40+
If an output is not `aggregating`, it only depends on the sources specified. Changes in other
41+
sources do not affect it. Unlike Gradle's java annotation processing, there can be multiple source
42+
files for an output.
43+
44+
For example, a generated class, which is dedicated to an interface it implements, is not
45+
`aggregating`.
46+
47+
In short, if an output may depend on new or any changed sources, it is `aggregating`.
48+
Otherwise it is not.
49+
50+
For readers familiar with Java annotation processing:
51+
* In an *isolating* Java annotation processor, all the outputs are *isolating* in KSP.
52+
* In an *aggregating* Java annotation processor, some outputs can be *isolating* and some be
53+
*aggregating* in KSP.
54+
55+
## Example 1
56+
A processor generates `outputForA` after reading class `A` in `A.kt` and class `B` in `B.kt`,
57+
where `A` extends `B`. The processor got `A` by `Resolver.getSymbolsWithAnnotation` and then got
58+
`B` by `KSClassDeclaration.superTypes` from `A`. Because the inclusion of `B` is due to `A`,
59+
`B.kt` needn't to be specified in `dependencies` for `outputForA`. Note that specifying `B.kt` in this case
60+
doesn't hurt, it is only unnecessary.
61+
62+
```
63+
// A.kt
64+
@Interesting
65+
class A : B()
66+
67+
// B.kt
68+
open class B
69+
70+
// Example1Processor.kt
71+
class Example1Processor : SymbolProcessor {
72+
...
73+
override fun process(resolver: Resolver) {
74+
val declA = resolver.getSymbolsWithAnnotation("Interesting").first() as KSClassDeclaration
75+
val declB = declA.superTypes.first().resolve().declaration
76+
// B.kt isn't required, because it is deducible by KSP.
77+
val dependencies = Dependencies(aggregating = false, declA.containingFile!!)
78+
// outputForA.kt
79+
val outputName = "outputFor${declA.simpleName.asString()}"
80+
// It depends on A.kt and B.kt.
81+
val output = codeGenerator.createNewFile(dependencies, "com.example", outputName, "kt")
82+
output.write("// $declA : $declB\n".toByteArray())
83+
output.close()
84+
}
85+
...
86+
}
87+
```
88+
89+
## Example 2
90+
Consider sourceA -> outputA, sourceB -> outputB.
91+
92+
When sourceA is changed:
93+
* If outputB is aggregating
94+
* sourceA and sourceB are reprocessed
95+
* If outputB is not aggregating
96+
* sourceA is reprocessed.
97+
98+
When sourceC is added:
99+
* If outputB is aggregating
100+
* sourceC and sourceB are reprocessed
101+
* If outputB is not aggregating
102+
* sourceC is reprocessed.
103+
104+
When sourceA is removed:
105+
* nothing has to be done.
106+
107+
When sourceB is removed:
108+
* nothing has to be done.
109+
110+
## How Dirtyness Are Determined
111+
A dirty file is either *changed* by users directly, or *affected* by other dirty files
112+
indirectly. In KSP, propagation of dirtyness is done in 2 steps:
113+
* Propagation by *resolution tracing*:
114+
Resolving a type reference (implicitly or explicitly) is the only way to navigate from one file
115+
to another. When a type reference is resolved by a processor, a *changed* or *affected* file that
116+
contains a change that may potentially affect the resolution result will *affect* the file
117+
containing that reference.
118+
* Propagation by *input-output correspondence*:
119+
If a source file is *changed* or *affected*, all other source files having some output in common
120+
with that file are *affected*.
121+
122+
Note that both of them are transitive and the second forms equivalence classes.
123+
124+
## Reporting Bugs
125+
To report a bug, please set Gradle properties `ksp.incremental=true` and `ksp.incremental.log=true`,
126+
and start with a clean build. There are 4 log files:
127+
128+
* `build/kspDirtySetByDeps.log`
129+
* `build/kspDirtySetByOutputs.log`
130+
* `build/kspDirtySet.log`
131+
* `build/kspSourceToOutputs.log`
132+
133+
They contain file names of sources and outputs, plus the timestamps of the builds.
134+
The first two are only avaiable in successive incremental builds and not available in clean builds.
File renamed without changes.

reference.md docs/reference.md

File renamed without changes.

0 commit comments

Comments
 (0)