KSP1 breaks compilation avoidance

[ZacSweers]

We've been doing some benchmarking and I was surprised to find that KSP tasks are not compatible with Kotlin's compilation avoidance in incremental compilation because it appears to poke through useClasspathSnapshot to manually depend on packaged jars (ref), which will always be out of date even if the change is non-abi breaking.

The task was not up-to-date because of the following reasons:
Input property 'classpathSnapshotProperties.classpath' file services/navigation/navigation-key/build/intermediates/compile_library_classes_jar/release/bundleLibCompileToJarRelease/classes.jar has changed.

This significantly slows down builds that use KSP extensively because their tasks rerun unnecessarily

[ZacSweers]

To put an example number for this - one benchmark in our monorepo (~1k modules) that uses KSP extensively results in the critical path for a non-abi change for a module in the middle of the graph to go from ~12sec to ~58sec

[ZacSweers]

Confirmed the same issue exists in KSP2

The task was not up-to-date because of the following reasons:
Input property 'kspConfig.libraries' file services/navigation/navigation-key/build/intermediates/compile_library_classes_jar/release/bundleLibCompileToJarRelease/classes.jar has changed.

[ting-yuan]

Some non-ABI changes can invalidate annotation processing results. For example, annotation processors may look into private members of a class in a library and generate outputs accordingly. This seems to be an inherited limitation of annotation processing.

Some other changes can be ignored though. A quick solution is to early return in KSP tasks when it doesn't see any meaningful changes.

The long term solution is to dependOn KGP's finer grained classpath snapshot artifacts instead of compile classpath. The performance would be similar to the above, but with KSP tasks reported as up to date in Gradle.

[ting-yuan]

BTW, there are some changes recently that aggregating processors always re-run in this case. Were you seeing performance changes recently or just discovered this 2-year old behavior?

[ZacSweers]

I see what you mean. That does seem like it could at least be configurable though given that it's not common for processors to process external private symbols

[ZacSweers]

In our case almost all of these projects are using isolating processors. You're right that it's not new behavior, but we also just weren't looking before I suppose.

[lzb6666]

BTW, there are some changes recently that aggregating processors always re-run in this case. Were you seeing performance changes recently or just discovered this 2-year old behavior?

I encountered the same issue when upgrading Kotlin from 1.7 to 2.0, which doubled the incremental compilation time for non-ABI changes in the lower-level modules. Is there any plan to solve this issue?

[ZacSweers]

Excited to see a fix already! Looks like there's no toggle, so will this now work like avoidance in other kotlinc tasks? And if so - does that mean that processing of private symbols from other modules is no longer going to be IC-compatible? (Assuming yes and I think that's ok, just confirming!)

[ZacSweers]

Just tested out the snapshot and worked like a charm 🚀

[ting-yuan]

Changes to signatures of private fields still trigger re-processing. Please file bugs or reopen if you found cases where it is not. The main difference, compared to previous implementation, is that it is no longer sensitive to changes of function body / text. It's basically the difference between @Classpath v.s. signatures of classes and all members.

[ZacSweers]

got it. In that case I still think it may be worth considering an option to ignore private signature changes since they are not part of public ABIs and not common in processors? Otherwise it discourages more mindful API visibility. I'd also expect this to align with Kotlin's handling of internal symbols, but suspect gradle's @Classpath is purely the java API? I'm curious how that would work with KMP projects

[ting-yuan]

The implementation could be non-trivial so I'd like to revisit this after KSP2. Another related performance option is skip-processing-if-no-relavent-changes. Currently, aggregating processors always got re-triggered. Although it should be safe to skip, we do get bugs reports about missing side effects that is not in generated outputs, such as printed logs.

For KMP, compilation avoidance is not supported at the moment, so any changes in libraries will trigger re-processing. Proper support of that is an even bigger effort compared to the above and will only be done after KSP2.