| <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" |
| "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> |
| <html xmlns="http://www.w3.org/1999/xhtml"> |
| <head> |
| <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/> |
| <title>Eclipse APT Design Notes</title> |
| </head> |
| <body lang="EN-US"> |
| <h1>Overview</h1> |
| <p>Annotation processors may implement either the Java 5 com.sun.mirror.apt interfaces or the |
| Java 6 javax.annotation.processing interfaces. The two interfaces are functionally similar but |
| the details are different. Note that although the Java 5 apt tool has been deprecated by Sun, |
| it is still included in the Java 6 JDK and developers continue to write processors against it. |
| However, the com.sun.mirror.apt interface is considered proprietary and there are no Java |
| compliance tests which validate it. |
| </p> |
| |
| <h1>Build Process</h1> |
| <p>The JDT system performs three kinds of build: full build, incremental build, and reconcile. When |
| a single file is edited and saved (if autobuild is enabled) or manually built, an incremental |
| build is performed: only the affected files will be rebuilt. This will include the edited file |
| as well as any other files with relevant dependencies on the edited file. After a clean, or |
| when incremental build detects an overwhelming number of files needing to be rebuilt, a full |
| build will be performed; in this case every file in the project will be rebuilt. During an |
| editing session, prior to saving, a file will frequently be reconciled, in order to identify |
| errors while typing. Reconcile operates on only a single file at a time.</p> |
| <p> |
| Java 5 and Java 6 processors are handled by different code and interact with the Eclipse Java |
| compiler in different ways and at different points in the build cycle. This is partly for |
| historical reasons and partly in order to support Java 6 processors in the context of the |
| command-line Eclipse compiler (ecj.jar). The Java 5 annotation processing code is built against |
| the JDT's public API; the Java 6 code uses internal interfaces within the compiler, which was |
| necessary because the public APIs use platform classes not available in the command-line package. |
| A pleasant consequence is that the Java 6 annotation processing code is somewhat easier |
| to understand.</p> |
| <p> |
| The main challenge of introducing APT into Eclipse is that annotation processors can contribute |
| additional Java types, which must in turn be compiled; this compilation can affect previously |
| compiled types, because a formerly unresolved reference can be resolved. Because of this, it |
| is necessary for annotation processing to be implemented within the compile process, rather than |
| as a later build step.</p> |
| <p> |
| TODO: overview of build process, i.e., when Java 5 processors are called relative to Java 6 processors |
| </p> |
| <h2>Java 5 Build-time Processing</h2> |
| <p> |
| The invocation path for Java 5 processors is quite complex, in part because when Java 5 processing |
| was implemented it was not possible to integrate it closely with the compiler, and in part for |
| silly historical reasons that serve no current purpose. It is ripe for refactoring.</p> |
| <p> |
| A typical call stack when a Java 5 processor is invoked looks like this:</p> |
| <pre> |
| SomeProcessor.process() line: xxx |
| APTDispatchRunnable.dispatchToFileBasedProcessor(AbstractCompilationEnv, boolean, boolean) line: 628 |
| APTDispatchRunnable.runAPTInFileBasedMode(BuildEnv) line: 317 |
| APTDispatchRunnable.build(BuildEnv) line: 655 |
| APTDispatchRunnable.access$1(APTDispatchRunnable, BuildEnv) line: 647 |
| APTDispatchRunnable$1.run(AbstractCompilationEnv) line: 265 |
| BuildEnv$CallbackRequestor.acceptBinding(String, IBinding) line: 611 |
| CompilationUnitResolver.resolve(ICompilationUnit[], String[], ASTRequestor, int, Map, WorkingCopyOwner, int) line: 766 |
| CompilationUnitResolver.resolve(ICompilationUnit[], String[], ASTRequestor, int, Map, IJavaProject, WorkingCopyOwner, int, IProgressMonitor) line: 478 |
| ASTParser.createASTs(ICompilationUnit[], String[], ASTRequestor, IProgressMonitor) line: 737 |
| BaseProcessorEnv.createASTs(IJavaProject, ICompilationUnit[], ASTRequestor) line: 856 |
| BuildEnv.createASTs(BuildContext[]) line: 356 |
| AbstractCompilationEnv.newBuildEnv(BuildContext[], BuildContext[], IJavaProject, AbstractCompilationEnv$EnvCallback) line: 111 |
| APTDispatchRunnable.build() line: 271 |
| APTDispatchRunnable.run(IProgressMonitor) line: 217 |
| Workspace.run(IWorkspaceRunnable, ISchedulingRule, int, IProgressMonitor) line: 1800 |
| APTDispatchRunnable.runAPTDuringBuild(BuildContext[], BuildContext[], Map<IFile,CategorizedProblem[]>, AptProject, Map<AnnotationProcessorFactory,Attributes>, Set<AnnotationProcessorFactory>, boolean) line: 142 |
| AptCompilationParticipant.processAnnotations(BuildContext[]) line: 193 |
| IncrementalImageBuilder(AbstractImageBuilder).processAnnotations(CompilationParticipantResult[]) line: 627 |
| IncrementalImageBuilder(AbstractImageBuilder).compile(SourceFile[]) line: 338 |
| IncrementalImageBuilder.build(SimpleLookupTable) line: 134 |
| JavaBuilder.buildDeltas(SimpleLookupTable) line: 265 |
| JavaBuilder.build(int, Map, IProgressMonitor) line: 193 |
| [higher calls omitted for brevity] |
| </pre> |
| <p> |
| Java 5 processors are invoked via the CompilationParticipant interface. The Java compiler calls |
| AptCompilationParticipant.processAnnotations(), passing in a list of all the files being built. |
| Note that the compiler may break large projects into multiple groups of files, resulting in |
| multiple calls to processAnnotations(). Files are marked with whether or not they contain annotations.</p> |
| <p> |
| Within processAnnotations(), annotation processor factories are discovered from the processor |
| factory path (and cached for subsequent invocations); factories and files are then passed in to |
| APTDispatchRunnable.runAPTDuringBuild(). This in turn stores the information in a new instance of |
| APTDispatchRunnable, and invokes its run() method within a workspace.run() operation.</p> |
| <p> |
| The run() method invokes BuildEnv.newBuildEnv(), passing in the AptDispatchRunnable as a callback. |
| The newBuildEnv() method creates a BuildEnv, gets compilation units for each of the files being |
| compiled, and then requests ASTs for each of the files. The parser is passed an ASTRequestor |
| callback that is implemented by an inner class of the BuildEnv; in this way, when the ASTs are |
| ready for consumption, APTDispatchRunnable.build(BuildEnv) is ultimately called, within the |
| ASTRequestor callback.</p> |
| <p> |
| Finally, within build(), processors are selected on the basis of the |
| annotations they support, and each processor's process() method is called on each appropriate file, |
| configuring the BuildEnv before each process() invocation so that it will provide the correct data. |
| The details of this vary depending on whether the processor is normal (file-based) or is marked |
| as requiring batch mode.</p> |
| <h3>Batch-mode Processors</h3> |
| <p>Processors can be marked in the Advanced Factory Path Options dialog as requiring batch mode. |
| If any of the processors on the project's factory path require batch mode, |
| AptDispatchRunnable.build(BuildEnv) calls runAPTInMixedMode(); if none of the processors |
| requires batch mode, runAPTInFileBasedMode() is called instead. The chief difference is that |
| batch-mode processors are only run during a full build, and that the batch-mode implementation of |
| AnnotationProcessorEnvironment.getTypeDeclarations() returns all the compilation units in the |
| build, whereas the normal implementation returns only a single compilation unit at a time.</p> |
| <p> |
| Additionally, a separate classloader is used to load batch-mode processors, and it is |
| discarded after each build; thus the classes are reloaded for each build, and static variables |
| are reinitialized. This is important because some processors written with command-line |
| (non-incremental) compilation in mind store dynamic state in static variables and thus cannot |
| be invoked multiple times. |
| </p> |
| <h3>File-mode Processors</h3> |
| <p>Most processors are designed (or at least should be designed) to support incremental compilation, |
| meaning that within the AnnotationProcessor.process() invocation, if the processor calls |
| AnnotationProcessorEnvironment.getTypeDeclarations(), it will only retrieve a single type at a |
| time; the process() method will be called repeatedly during the course of a build so that all |
| files are eventually processed. |
| </p> |
| <h2>Java 6 Build-time Processing</h2> |
| <p> |
| The invocation process for Java 6 processors is more straightforward. A typical call stack looks |
| like this:</p> |
| <pre> |
| ModelTesterProc.process(Set<TypeElement>, RoundEnvironment) line: 107 |
| RoundDispatcher.handleProcessor(ProcessorInfo) line: 139 |
| RoundDispatcher.round() line: 121 |
| IdeAnnotationProcessorManager(BaseAnnotationProcessorManager).processAnnotations(CompilationUnitDeclaration[], ReferenceBinding[], boolean) line: 159 |
| IdeAnnotationProcessorManager.processAnnotations(CompilationUnitDeclaration[], ReferenceBinding[], boolean) line: 134 |
| Compiler.processAnnotations() line: 810 |
| Compiler.compile(ICompilationUnit[]) line: 428 |
| IncrementalImageBuilder(AbstractImageBuilder).compile(SourceFile[], SourceFile[], boolean) line: 364 |
| IncrementalImageBuilder.compile(SourceFile[], SourceFile[], boolean) line: 321 |
| IncrementalImageBuilder(AbstractImageBuilder).compile(SourceFile[]) line: 301 |
| IncrementalImageBuilder.build(SimpleLookupTable) line: 134 |
| JavaBuilder.buildDeltas(SimpleLookupTable) line: 265 |
| JavaBuilder.build(int, Map, IProgressMonitor) line: 193 |
| [higher calls omitted for brevity] |
| </pre> |
| <p> |
| Note that the processing is performed by an IdeAnnotationProcessorManager; this is distinguished |
| from the BatchAnnotationProcessorManager, which would be used if processing was invoked from a |
| command line (ecj.jar) compilation. Command-line compilation with ecj.jar uses the same typesystem |
| implementation as IDE compilation, but somewhat different Messager and Filer implementations. |
| </p> |
| <h2>Reconcile-time Processing</h2> |
| <p> |
| Reconcile-time processing is only supported for Java 5 processors. This is chiefly because, after |
| developing the Java 5 processing implementation, it became clear that writing highly-performant |
| annotation processors requires a level of compiler experience that most programmers do not have, |
| and if less-performant processors are inserted into the reconcile step, reconcile can become |
| painfully and confusingly slow. Thus when the Java 6 processing implementation was added it was |
| decided to not support reconcile-time processing. Eclipse's effectiveness as an IDE depends on |
| reconcile being an extremely fast operation. In hindsight, edit-time processing should have been |
| limited to reporting errors, and should have been implemented as a background validation task |
| rather than within the reconcile step.</p> |
| <p> |
| TODO: describe practical differences between reconcile and build.</p> |
| <p> |
| The stack trace of a typical reconcile-time invocation looks like this:</p> |
| <pre> |
| SomeProcessor.process() line: xxx |
| APTDispatchRunnable.dispatchToFileBasedProcessor(AbstractCompilationEnv, boolean, boolean) line: 628 |
| APTDispatchRunnable.access$0(APTDispatchRunnable, AbstractCompilationEnv, boolean, boolean) line: 598 |
| APTDispatchRunnable$ReconcileEnvCallback.run(AbstractCompilationEnv) line: 77 |
| ReconcileEnv$CallbackRequestor.acceptBinding(String, IBinding) line: 135 |
| CompilationUnitResolver.resolve(ICompilationUnit[], String[], ASTRequestor, int, Map, WorkingCopyOwner, int) line: 766 |
| CompilationUnitResolver.resolve(ICompilationUnit[], String[], ASTRequestor, int, Map, IJavaProject, WorkingCopyOwner, int, IProgressMonitor) line: 478 |
| ASTParser.createASTs(ICompilationUnit[], String[], ASTRequestor, IProgressMonitor) line: 737 |
| BaseProcessorEnv.createASTs(IJavaProject, ICompilationUnit[], ASTRequestor) line: 856 |
| ReconcileEnv.openPipeline() line: 108 |
| AbstractCompilationEnv.newReconcileEnv(ReconcileContext, AbstractCompilationEnv$EnvCallback) line: 97 |
| APTDispatchRunnable.reconcile(ReconcileContext, IJavaProject) line: 211 |
| APTDispatchRunnable.runAPTDuringReconcile(ReconcileContext, AptProject, Map<AnnotationProcessorFactory,Attributes>) line: 159 |
| AptCompilationParticipant.reconcile(ReconcileContext) line: 223 |
| ReconcileWorkingCopyOperation$1.run() line: 257 |
| SafeRunner.run(ISafeRunnable) line: 42 |
| ReconcileWorkingCopyOperation.notifyParticipants(CompilationUnit) line: 244 |
| ReconcileWorkingCopyOperation.executeOperation() line: 94 |
| ReconcileWorkingCopyOperation(JavaModelOperation).run(IProgressMonitor) line: 728 |
| ReconcileWorkingCopyOperation(JavaModelOperation).runOperation(IProgressMonitor) line: 788 |
| CompilationUnit.reconcile(int, int, WorkingCopyOwner, IProgressMonitor) line: 1242 |
| JavaReconcilingStrategy.reconcile(ICompilationUnit, boolean) line: 126 |
| [higher calls omitted for brevity] |
| </pre> |
| |
| <h1>Processor Callbacks</h1> |
| <p>When annotation processors are invoked, they can in turn call back into the compiler through various |
| interfaces in order to generate new files and report errors. The interfaces are naturally not the same as |
| those used within the JDT compiler, so wrappers and proxies are employed to glue the different systems |
| together. The most significant difference in systems is that, in both the Java 5 and Java 6 processor |
| interfaces, the typesystem is separated into Type and Element hierarchies, in which a Type object |
| represents a particular reification of a type while an Element represents a declaration in code. The |
| distinction is easiest to understand in the context of generics: for example, the following code sample |
| contains declarations of classes Lister and Test, which contain declarations of method get() and fields |
| 'strings' and 'numbers' respectively. The type of the return of get() is List<T>, the type of strings is |
| Lister<String>, and the type of numbers is Lister<Number>. Note that the single declaration of Lister<T> |
| was able to generate multiple Types. |
| </p> |
| <pre> |
| import java.util.List; |
| class Lister<T> { |
| List<T> get() { ... } |
| } |
| class Test { |
| Lister<String> strings; |
| Lister<Number> numbers; |
| } |
| </pre> |
| |
| <h2>Java 5 Type System</h2> |
| <p>TODO</p> |
| |
| <h2>Java 5 Filer and Messager</h2> |
| <p>TODO</p> |
| |
| <h2>Java 6 Type System</h2> |
| <p>TODO</p> |
| |
| <h2>Java 6 Filer and Messager</h2> |
| <p>TODO</p> |
| |
| </body> |
| </html> |