|
| 1 | +# Incremental Processing |
| 2 | + |
| 3 | +Incremental processing is a way to avoid re-processing of sources as much as possible. |
| 4 | +The major goal is to reduce the turn-around time of a typical change-compile-test cycle. |
| 5 | +Here is a [wiki link](https://en.wikipedia.org/wiki/Incremental_computing) to the more general |
| 6 | +idea of incremental computation. |
| 7 | + |
| 8 | +To be able to know which sources are dirty, i.e., those that need to be reprocessed, KSP needs |
| 9 | +processors' help to identify the correspondence of input sources and generated outputs. |
| 10 | +Because it is cumbersome and error prone to keep track of which inputs are involved in generating |
| 11 | +which outputs, KSP is designed to help with that and only require a minimum set of |
| 12 | +**sources that serve as roots that processors start to navigate the code structure**. In other |
| 13 | +words, a processor needs to associate an output with sources of those `KSNode`, if obtained from: |
| 14 | +* `Resolver.getAllFiles` |
| 15 | +* `Resolver.getSymbolsWithAnnotation` |
| 16 | +* `Resolver.getClassDesclarationByName` |
| 17 | + |
| 18 | +Currently, only changes in Kotlin and Java sources are tracked. If there is a change in the |
| 19 | +classpath, namely in other modules or libraries, a full re-processing will be triggered. |
| 20 | + |
| 21 | +Incremental processing is currently disabled by default. To enable it, set the Gradle property |
| 22 | +`ksp.incremental=true`. To enable logs, which dump the dirty set according to dependencies and |
| 23 | +outputs, use `ksp.incremental.log=true`. They can be found as `build/*.log`. |
| 24 | + |
| 25 | +## Aggregating v.s. Isolating |
| 26 | +The idea is similar but slightly different to the [definition](https://docs.gradle.org/current/userguide/java_plugin.html#sec:incremental_annotation_processing) |
| 27 | +in Gradle annotation processing. In KSP, |
| 28 | +* *aggregating* / *isolating* is associated with each output, rather than the entire processor |
| 29 | +* an isolating output can have several sources |
| 30 | +* aggregating means that an output can be affected by any changes |
| 31 | + |
| 32 | +If an output is `aggregating`, any changes may affect it potentially, except removal of files that |
| 33 | +don't affect other files. |
| 34 | +In other words, if there's a change, all `aggregating` outputs need to be regenerated and therefore |
| 35 | +all of their sources will be reprocessed. Note that only registered files and changed / new files |
| 36 | +will be re-processed. |
| 37 | + |
| 38 | +For example, an output collecting all symbols with an interesting annotation is `aggregating`. |
| 39 | + |
| 40 | +If an output is not `aggregating`, it only depends on the sources specified. Changes in other |
| 41 | +sources do not affect it. Unlike Gradle's java annotation processing, there can be multiple source |
| 42 | +files for an output. |
| 43 | + |
| 44 | +For example, a generated class, which is dedicated to an interface it implements, is not |
| 45 | +`aggregating`. |
| 46 | + |
| 47 | +In short, if an output may depend on new or any changed sources, it is `aggregating`. |
| 48 | +Otherwise it is not. |
| 49 | + |
| 50 | +For readers familiar with Java annotation processing: |
| 51 | +* In an *isolating* Java annotation processor, all the outputs are *isolating* in KSP. |
| 52 | +* In an *aggregating* Java annotation processor, some outputs can be *isolating* and some be |
| 53 | +*aggregating* in KSP. |
| 54 | + |
| 55 | +## Example 1 |
| 56 | +A processor generates `outputForA` after reading class `A` in `A.kt` and class `B` in `B.kt`, |
| 57 | +where `A` extends `B`. The processor got `A` by `Resolver.getSymbolsWithAnnotation` and then got |
| 58 | +`B` by `KSClassDeclaration.superTypes` from `A`. Because the inclusion of `B` is due to `A`, |
| 59 | +`B.kt` needn't to be specified in `dependencies` for `outputForA`. Note that specifying `B.kt` in this case |
| 60 | +doesn't hurt, it is only unnecessary. |
| 61 | + |
| 62 | +``` |
| 63 | +// A.kt |
| 64 | +@Interesting |
| 65 | +class A : B() |
| 66 | +
|
| 67 | +// B.kt |
| 68 | +open class B |
| 69 | +
|
| 70 | +// Example1Processor.kt |
| 71 | +class Example1Processor : SymbolProcessor { |
| 72 | + ... |
| 73 | + override fun process(resolver: Resolver) { |
| 74 | + val declA = resolver.getSymbolsWithAnnotation("Interesting").first() as KSClassDeclaration |
| 75 | + val declB = declA.superTypes.first().resolve().declaration |
| 76 | + // B.kt isn't required, because it is deducible by KSP. |
| 77 | + val dependencies = Dependencies(aggregating = false, declA.containingFile!!) |
| 78 | + // outputForA.kt |
| 79 | + val outputName = "outputFor${declA.simpleName.asString()}" |
| 80 | + // It depends on A.kt and B.kt. |
| 81 | + val output = codeGenerator.createNewFile(dependencies, "com.example", outputName, "kt") |
| 82 | + output.write("// $declA : $declB\n".toByteArray()) |
| 83 | + output.close() |
| 84 | + } |
| 85 | + ... |
| 86 | +} |
| 87 | +``` |
| 88 | + |
| 89 | +## Example 2 |
| 90 | +Consider sourceA -> outputA, sourceB -> outputB. |
| 91 | + |
| 92 | +When sourceA is changed: |
| 93 | +* If outputB is aggregating |
| 94 | + * sourceA and sourceB are reprocessed |
| 95 | +* If outputB is not aggregating |
| 96 | + * sourceA is reprocessed. |
| 97 | + |
| 98 | +When sourceC is added: |
| 99 | +* If outputB is aggregating |
| 100 | + * sourceC and sourceB are reprocessed |
| 101 | +* If outputB is not aggregating |
| 102 | + * sourceC is reprocessed. |
| 103 | + |
| 104 | +When sourceA is removed: |
| 105 | +* nothing has to be done. |
| 106 | + |
| 107 | +When sourceB is removed: |
| 108 | +* nothing has to be done. |
| 109 | + |
| 110 | +## How Dirtyness Are Determined |
| 111 | +A dirty file is either *changed* by users directly, or *affected* by other dirty files |
| 112 | +indirectly. In KSP, propagation of dirtyness is done in 2 steps: |
| 113 | +* Propagation by *resolution tracing*: |
| 114 | + Resolving a type reference (implicitly or explicitly) is the only way to navigate from one file |
| 115 | + to another. When a type reference is resolved by a processor, a *changed* or *affected* file that |
| 116 | + contains a change that may potentially affect the resolution result will *affect* the file |
| 117 | + containing that reference. |
| 118 | +* Propagation by *input-output correspondence*: |
| 119 | + If a source file is *changed* or *affected*, all other source files having some output in common |
| 120 | + with that file are *affected*. |
| 121 | + |
| 122 | +Note that both of them are transitive and the second forms equivalence classes. |
| 123 | + |
| 124 | +## Reporting Bugs |
| 125 | +To report a bug, please set Gradle properties `ksp.incremental=true` and `ksp.incremental.log=true`, |
| 126 | +and start with a clean build. There are 4 log files: |
| 127 | + |
| 128 | +* `build/kspDirtySetByDeps.log` |
| 129 | +* `build/kspDirtySetByOutputs.log` |
| 130 | +* `build/kspDirtySet.log` |
| 131 | +* `build/kspSourceToOutputs.log` |
| 132 | + |
| 133 | +They contain file names of sources and outputs, plus the timestamps of the builds. |
| 134 | +The first two are only avaiable in successive incremental builds and not available in clean builds. |
0 commit comments