blob: c79e6f016fea69852339dddd6529b50c0d0b91bf [file] [log] [blame]
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Eclipse APT Design Notes</title>
</head>
<body lang="EN-US">
<h1>Overview</h1>
<p>Annotation processors may implement either the Java 5 com.sun.mirror.apt interfaces or the
Java 6 javax.annotation.processing interfaces. The two interfaces are functionally similar but
the details are different. Note that although the Java 5 apt tool has been deprecated by Sun,
it is still included in the Java 6 JDK and developers continue to write processors against it.
However, the com.sun.mirror.apt interface is considered proprietary and there are no Java
compliance tests which validate it.
</p>
<h1>Build Process</h1>
<p>The JDT system performs three kinds of build: full build, incremental build, and reconcile. When
a single file is edited and saved (if autobuild is enabled) or manually built, an incremental
build is performed: only the affected files will be rebuilt. This will include the edited file
as well as any other files with relevant dependencies on the edited file. After a clean, or
when incremental build detects an overwhelming number of files needing to be rebuilt, a full
build will be performed; in this case every file in the project will be rebuilt. During an
editing session, prior to saving, a file will frequently be reconciled, in order to identify
errors while typing. Reconcile operates on only a single file at a time.</p>
<p>
Java 5 and Java 6 processors are handled by different code and interact with the Eclipse Java
compiler in different ways and at different points in the build cycle. This is partly for
historical reasons and partly in order to support Java 6 processors in the context of the
command-line Eclipse compiler (ecj.jar). The Java 5 annotation processing code is built against
the JDT's public API; the Java 6 code uses internal interfaces within the compiler, which was
necessary because the public APIs use platform classes not available in the command-line package.
A pleasant consequence is that the Java 6 annotation processing code is somewhat easier
to understand.</p>
<p>
The main challenge of introducing APT into Eclipse is that annotation processors can contribute
additional Java types, which must in turn be compiled; this compilation can affect previously
compiled types, because a formerly unresolved reference can be resolved. Because of this, it
is necessary for annotation processing to be implemented within the compile process, rather than
as a later build step.</p>
<p>
TODO: overview of build process, i.e., when Java 5 processors are called relative to Java 6 processors
</p>
<h2>Java 5 Build-time Processing</h2>
<p>
The invocation path for Java 5 processors is quite complex, in part because when Java 5 processing
was implemented it was not possible to integrate it closely with the compiler, and in part for
silly historical reasons that serve no current purpose. It is ripe for refactoring.</p>
<p>
A typical call stack when a Java 5 processor is invoked looks like this:</p>
<pre>
SomeProcessor.process() line: xxx
APTDispatchRunnable.dispatchToFileBasedProcessor(AbstractCompilationEnv, boolean, boolean) line: 628
APTDispatchRunnable.runAPTInFileBasedMode(BuildEnv) line: 317
APTDispatchRunnable.build(BuildEnv) line: 655
APTDispatchRunnable.access$1(APTDispatchRunnable, BuildEnv) line: 647
APTDispatchRunnable$1.run(AbstractCompilationEnv) line: 265
BuildEnv$CallbackRequestor.acceptBinding(String, IBinding) line: 611
CompilationUnitResolver.resolve(ICompilationUnit[], String[], ASTRequestor, int, Map, WorkingCopyOwner, int) line: 766
CompilationUnitResolver.resolve(ICompilationUnit[], String[], ASTRequestor, int, Map, IJavaProject, WorkingCopyOwner, int, IProgressMonitor) line: 478
ASTParser.createASTs(ICompilationUnit[], String[], ASTRequestor, IProgressMonitor) line: 737
BaseProcessorEnv.createASTs(IJavaProject, ICompilationUnit[], ASTRequestor) line: 856
BuildEnv.createASTs(BuildContext[]) line: 356
AbstractCompilationEnv.newBuildEnv(BuildContext[], BuildContext[], IJavaProject, AbstractCompilationEnv$EnvCallback) line: 111
APTDispatchRunnable.build() line: 271
APTDispatchRunnable.run(IProgressMonitor) line: 217
Workspace.run(IWorkspaceRunnable, ISchedulingRule, int, IProgressMonitor) line: 1800
APTDispatchRunnable.runAPTDuringBuild(BuildContext[], BuildContext[], Map&lt;IFile,CategorizedProblem[]&gt;, AptProject, Map&lt;AnnotationProcessorFactory,Attributes&gt;, Set&lt;AnnotationProcessorFactory&gt;, boolean) line: 142
AptCompilationParticipant.processAnnotations(BuildContext[]) line: 193
IncrementalImageBuilder(AbstractImageBuilder).processAnnotations(CompilationParticipantResult[]) line: 627
IncrementalImageBuilder(AbstractImageBuilder).compile(SourceFile[]) line: 338
IncrementalImageBuilder.build(SimpleLookupTable) line: 134
JavaBuilder.buildDeltas(SimpleLookupTable) line: 265
JavaBuilder.build(int, Map, IProgressMonitor) line: 193
[higher calls omitted for brevity]
</pre>
<p>
Java 5 processors are invoked via the CompilationParticipant interface. The Java compiler calls
AptCompilationParticipant.processAnnotations(), passing in a list of all the files being built.
Note that the compiler may break large projects into multiple groups of files, resulting in
multiple calls to processAnnotations(). Files are marked with whether or not they contain annotations.</p>
<p>
Within processAnnotations(), annotation processor factories are discovered from the processor
factory path (and cached for subsequent invocations); factories and files are then passed in to
APTDispatchRunnable.runAPTDuringBuild(). This in turn stores the information in a new instance of
APTDispatchRunnable, and invokes its run() method within a workspace.run() operation.</p>
<p>
The run() method invokes BuildEnv.newBuildEnv(), passing in the AptDispatchRunnable as a callback.
The newBuildEnv() method creates a BuildEnv, gets compilation units for each of the files being
compiled, and then requests ASTs for each of the files. The parser is passed an ASTRequestor
callback that is implemented by an inner class of the BuildEnv; in this way, when the ASTs are
ready for consumption, APTDispatchRunnable.build(BuildEnv) is ultimately called, within the
ASTRequestor callback.</p>
<p>
Finally, within build(), processors are selected on the basis of the
annotations they support, and each processor's process() method is called on each appropriate file,
configuring the BuildEnv before each process() invocation so that it will provide the correct data.
The details of this vary depending on whether the processor is normal (file-based) or is marked
as requiring batch mode.</p>
<h3>Batch-mode Processors</h3>
<p>Processors can be marked in the Advanced Factory Path Options dialog as requiring batch mode.
If any of the processors on the project's factory path require batch mode,
AptDispatchRunnable.build(BuildEnv) calls runAPTInMixedMode(); if none of the processors
requires batch mode, runAPTInFileBasedMode() is called instead. The chief difference is that
batch-mode processors are only run during a full build, and that the batch-mode implementation of
AnnotationProcessorEnvironment.getTypeDeclarations() returns all the compilation units in the
build, whereas the normal implementation returns only a single compilation unit at a time.</p>
<p>
Additionally, a separate classloader is used to load batch-mode processors, and it is
discarded after each build; thus the classes are reloaded for each build, and static variables
are reinitialized. This is important because some processors written with command-line
(non-incremental) compilation in mind store dynamic state in static variables and thus cannot
be invoked multiple times.
</p>
<h3>File-mode Processors</h3>
<p>Most processors are designed (or at least should be designed) to support incremental compilation,
meaning that within the AnnotationProcessor.process() invocation, if the processor calls
AnnotationProcessorEnvironment.getTypeDeclarations(), it will only retrieve a single type at a
time; the process() method will be called repeatedly during the course of a build so that all
files are eventually processed.
</p>
<h2>Java 6 Build-time Processing</h2>
<p>
The invocation process for Java 6 processors is more straightforward. A typical call stack looks
like this:</p>
<pre>
ModelTesterProc.process(Set&lt;TypeElement&gt;, RoundEnvironment) line: 107
RoundDispatcher.handleProcessor(ProcessorInfo) line: 139
RoundDispatcher.round() line: 121
IdeAnnotationProcessorManager(BaseAnnotationProcessorManager).processAnnotations(CompilationUnitDeclaration[], ReferenceBinding[], boolean) line: 159
IdeAnnotationProcessorManager.processAnnotations(CompilationUnitDeclaration[], ReferenceBinding[], boolean) line: 134
Compiler.processAnnotations() line: 810
Compiler.compile(ICompilationUnit[]) line: 428
IncrementalImageBuilder(AbstractImageBuilder).compile(SourceFile[], SourceFile[], boolean) line: 364
IncrementalImageBuilder.compile(SourceFile[], SourceFile[], boolean) line: 321
IncrementalImageBuilder(AbstractImageBuilder).compile(SourceFile[]) line: 301
IncrementalImageBuilder.build(SimpleLookupTable) line: 134
JavaBuilder.buildDeltas(SimpleLookupTable) line: 265
JavaBuilder.build(int, Map, IProgressMonitor) line: 193
[higher calls omitted for brevity]
</pre>
<p>
Note that the processing is performed by an IdeAnnotationProcessorManager; this is distinguished
from the BatchAnnotationProcessorManager, which would be used if processing was invoked from a
command line (ecj.jar) compilation. Command-line compilation with ecj.jar uses the same typesystem
implementation as IDE compilation, but somewhat different Messager and Filer implementations.
</p>
<h2>Reconcile-time Processing</h2>
<p>
Reconcile-time processing is only supported for Java 5 processors. This is chiefly because, after
developing the Java 5 processing implementation, it became clear that writing highly-performant
annotation processors requires a level of compiler experience that most programmers do not have,
and if less-performant processors are inserted into the reconcile step, reconcile can become
painfully and confusingly slow. Thus when the Java 6 processing implementation was added it was
decided to not support reconcile-time processing. Eclipse's effectiveness as an IDE depends on
reconcile being an extremely fast operation. In hindsight, edit-time processing should have been
limited to reporting errors, and should have been implemented as a background validation task
rather than within the reconcile step.</p>
<p>
TODO: describe practical differences between reconcile and build.</p>
<p>
The stack trace of a typical reconcile-time invocation looks like this:</p>
<pre>
SomeProcessor.process() line: xxx
APTDispatchRunnable.dispatchToFileBasedProcessor(AbstractCompilationEnv, boolean, boolean) line: 628
APTDispatchRunnable.access$0(APTDispatchRunnable, AbstractCompilationEnv, boolean, boolean) line: 598
APTDispatchRunnable$ReconcileEnvCallback.run(AbstractCompilationEnv) line: 77
ReconcileEnv$CallbackRequestor.acceptBinding(String, IBinding) line: 135
CompilationUnitResolver.resolve(ICompilationUnit[], String[], ASTRequestor, int, Map, WorkingCopyOwner, int) line: 766
CompilationUnitResolver.resolve(ICompilationUnit[], String[], ASTRequestor, int, Map, IJavaProject, WorkingCopyOwner, int, IProgressMonitor) line: 478
ASTParser.createASTs(ICompilationUnit[], String[], ASTRequestor, IProgressMonitor) line: 737
BaseProcessorEnv.createASTs(IJavaProject, ICompilationUnit[], ASTRequestor) line: 856
ReconcileEnv.openPipeline() line: 108
AbstractCompilationEnv.newReconcileEnv(ReconcileContext, AbstractCompilationEnv$EnvCallback) line: 97
APTDispatchRunnable.reconcile(ReconcileContext, IJavaProject) line: 211
APTDispatchRunnable.runAPTDuringReconcile(ReconcileContext, AptProject, Map&lt;AnnotationProcessorFactory,Attributes&gt;) line: 159
AptCompilationParticipant.reconcile(ReconcileContext) line: 223
ReconcileWorkingCopyOperation$1.run() line: 257
SafeRunner.run(ISafeRunnable) line: 42
ReconcileWorkingCopyOperation.notifyParticipants(CompilationUnit) line: 244
ReconcileWorkingCopyOperation.executeOperation() line: 94
ReconcileWorkingCopyOperation(JavaModelOperation).run(IProgressMonitor) line: 728
ReconcileWorkingCopyOperation(JavaModelOperation).runOperation(IProgressMonitor) line: 788
CompilationUnit.reconcile(int, int, WorkingCopyOwner, IProgressMonitor) line: 1242
JavaReconcilingStrategy.reconcile(ICompilationUnit, boolean) line: 126
[higher calls omitted for brevity]
</pre>
<h1>Processor Callbacks</h1>
<p>When annotation processors are invoked, they can in turn call back into the compiler through various
interfaces in order to generate new files and report errors. The interfaces are naturally not the same as
those used within the JDT compiler, so wrappers and proxies are employed to glue the different systems
together. The most significant difference in systems is that, in both the Java 5 and Java 6 processor
interfaces, the typesystem is separated into Type and Element hierarchies, in which a Type object
represents a particular reification of a type while an Element represents a declaration in code. The
distinction is easiest to understand in the context of generics: for example, the following code sample
contains declarations of classes Lister and Test, which contain declarations of method get() and fields
'strings' and 'numbers' respectively. The type of the return of get() is List&lt;T&gt;, the type of strings is
Lister&lt;String&gt;, and the type of numbers is Lister&lt;Number&gt;. Note that the single declaration of Lister&lt;T&gt;
was able to generate multiple Types.
</p>
<pre>
import java.util.List;
class Lister&lt;T&gt; {
List&lt;T&gt; get() { ... }
}
class Test {
Lister&lt;String&gt; strings;
Lister&lt;Number&gt; numbers;
}
</pre>
<h2>Java 5 Type System</h2>
<p>TODO</p>
<h2>Java 5 Filer and Messager</h2>
<p>TODO</p>
<h2>Java 6 Type System</h2>
<p>TODO</p>
<h2>Java 6 Filer and Messager</h2>
<p>TODO</p>
</body>
</html>