| <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> |
| <html> |
| <head> |
| <title>Eclipse 3.0 builder issues</title> |
| <link rel="stylesheet" href="http://dev.eclipse.org/default_style.css" type="text/css"> |
| </head> |
| <body text="#000000" bgcolor="#FFFFFF"> |
| |
| <h1>Eclipse 3.0 builder issues</h1> |
| <font size="-1">Last modified: November 13, 2003</font> |
| <p> |
| This document outlines the current pressing builder |
| problems, maps out the solution space, and will eventually propose concrete changes |
| for Eclipse 3.0 and if necessary for a 2.1.x maintenance release. Since this is not |
| a committed plan item, no promises are being made at this point. |
| </p> |
| <h3>Problems</h3> |
| <h4>P1. Dynamic classpaths, static project references</h4> |
| <p> |
| There are many different PDE self-hosting styles. Some users build and run against |
| plugins outside the workspace, some use binary or source projects in the workspace, |
| and some use a combination of the two. Every time a user switches from building |
| against external libraries to building against internal projects, the classpaths of all |
| downstream plugins need to be updated. When the set of referenced projects on the |
| classpath change, so do the platform project references. These references are used |
| by the platform when computing the default project build order. |
| </p> |
| <p> |
| Before the Eclipse 2.1 release, JDT introduced the notion of classpath containers. |
| Classpath containers abstract away volatile sets of references so that the .classpath |
| file does not need to change every time these reference sets change. The two common |
| uses of containers are for JRE libraries, and PDE project references. This approach |
| solves the problem of .classpath files constantly changing as the user's self-hosting |
| style changes (or when the JRE changes). The remaining problem is that the project |
| references, manifested in the .project file, do not have a corresponding dynamic |
| container facility. Therefore the .project file continues to change whenever the |
| user switches between external and internal project references. |
| </p> |
| <h4>P2. Builders need more context</h4> |
| <p> |
| Individual project builders are not supplied with enough context information when |
| they are invoked. In particular, they do not know whether the current build is a single |
| project build or a workspace build. Also, there is no consistent hook that builder |
| implementors can use to be notified when the build process is starting and finishing. |
| If this information was available, builder implementors would have more flexibility |
| in optimizing multiple project builds. |
| </p> |
| <p> |
| Without going into too much detail, consider a simple case of projects A and B, |
| where B depends on A and so must be built after it. When A is built, it could leave |
| behind a "cache" that describes in more detail exactly what changes builder A made. |
| Builder B could then read the cached information and optimize its build accordingly. However, |
| with manual project builds, project A could be built several times before project B is |
| built. In this simple case, builder A could incrementally update this cache until |
| B is finally built. However, if there is also a project C that depends on A, it could be |
| built at a different time. In the general case, it is impossible for a builder to leave |
| behind a cache for dependent builders to read, because it can never know the appropriate |
| times to update or flush that cache. |
| </p> |
| <p> |
| To further complicate matters, other builders may have been installed on projects A or B |
| that run in between the two builders that are trying to optimize downstream dependency |
| changes. Builders cannot reliably determine if other builders have been run that affect |
| them, or whether changes they see in a delta have been caused by their own builder |
| or some other builder. |
| </p> |
| <h4>P3. Build scopes</h4> |
| <p> |
| Some Eclipse users work with massive workspaces of highly interconnected resources. |
| These workspaces often take a very long time to build when a resource with a large |
| number of downstream references changes. Builder implementors are able to optimize |
| these cases to some extent, but in cases where thousands of referenced resources are |
| genuinely affected by a change, these optimizations can only go so far. |
| </p> |
| <p> |
| One coping strategy that users have adopted is to only build small subsets of |
| the workspace frequently, and perform full workspace builds less often. This essentially |
| batches changes so that a single long build can replace frequent long builds, thus |
| freeing up more time for users. |
| </p> |
| <p> |
| The current Eclipse build facilities are not well suited to this approach. Autobuild |
| and the incremental build command (invoked with Ctrl+B), work only at a workspace |
| scope. To build smaller sets of projects, users have to perform cumbersome manual |
| steps involving multi-selecting the set of projects, and invoking "Build Project" |
| from the "Project" context menu. Autobuild has established a high |
| expectation about the transparency of building, making these manual steps unacceptable. |
| </p> |
| |
| <h3>Solutions</h3> |
| <h4>S1. Workspace builders</h4> |
| <p> |
| Currently a workspace build runs roughly as follows: |
| <pre> |
| for each project P(i) in workspace { |
| for each builder B(j) installed on P(i) { |
| run builder P(i)B(j) |
| } |
| } |
| </pre> |
| This process could be inverted by allowing builders to be installed on the entire workspace |
| that have complete control over how the projects they care about are built. The build |
| process would then run as follows: |
| <pre> |
| for each workspace builder B(i) { |
| run builder B(i) |
| } |
| </pre> |
| And a workspace builder B(i) would have a loop within its build like this: |
| <pre> |
| for each project P(j) in workspace { |
| if (P(j) has nature for builder B(i) { |
| build project builder B(i) on project P(j) |
| } |
| } |
| </pre> |
| </p> |
| <p> |
| The order in which builders run would be established by declarative rules in the |
| workspace builder extension point. These rules would be resolved by platform |
| as a topological graph ordering with cycles treated as unrecoverable errors. |
| </p> |
| <p> |
| <b>Pros:</b> With this approach, each builder has complete control over the build order it uses. |
| Project references are no longer needed, so problem P1 goes away. The builder |
| has all the context information it needs, so the optimizations discussed in problem P2 |
| are now possible. It is also no longer possible for other builders to run interleaved |
| with a particular builder as it iterates over the projects. Thus the builder has reasonably |
| complete knowledge of what changes have happened to each project, and what |
| the content of those changes are. |
| </p> |
| <p> |
| <b>Cons:</b> This is a breaking change to the workspace build order. If project |
| P1 and P2 have builders B1 and B2, the old order is: |
| <pre> |
| P1B1, P1B2, P2B1, P2B2 |
| </pre> |
| And the new order is: |
| <pre> |
| P1B1, P2B1, P1B2, P2B2 |
| </pre> |
| If there are projects relying on this order, they will be broken by this change. It is |
| true that since the platform has always exposed API to run an individual builder on |
| a single project, that all possible build orders have always been permitted and all |
| builders should be able to handle it. However, this is a breaking change to |
| autobuild and to the behaviour of the API method <tt>IWorkspace.build</tt>. |
| </p> |
| <p> |
| Second, there is a large class of builders that piggyback on other builders, and rely |
| on those builders to establish the build order for them. For example, a client may |
| install a builder that acts as a pre- or post-processor on the java builder, generating |
| extra source files or copying/mutating class files. These builders typically have no |
| knowledge of the correct build project order. In a world of workspace builders, the |
| workspace builder for these secondary builders would need a mechanism for computing |
| a reasonable project build order. The java builder and other "primary" |
| builders would need to expose such an API and all secondary builders would use this |
| (non generic) API to compute a reasonable build order. |
| </p> |
| <p> |
| In the general case, it would be impossible to provide a backwards compatibility mode |
| for builders implemented against Eclipse 2.1. For example, say a 2.1 workspace is |
| opened in 3.0, and it has projects with builder order {B1, B2}. Say B2 is now a |
| workspace builder that defines its own project build order. B2 was previously using |
| project references, but it immediately removes all project references when its plugin |
| is first loaded since these are no longer used (if it did not do this it would not be |
| solving problem P1 in our list). How does the platform determine an appropriate |
| project order for running builder B1? Since B2 has not run, it can not infer an |
| order from B2. B2 would have to expose some method that pre-declares the order |
| it intends to use. B1, in the absence of its own build order, would use this declared |
| order for its own projects. Now say there are two workspace builders (B2 and B3) on |
| each project, and each declares a different project order. It is now impossible to |
| infer an appropriate build order for B1. |
| </p> |
| <h4>S2. Pluggable build strategy</h4> |
| <a name="s2"/> |
| <p> |
| A less ambitious proposal is to allow pluggable build strategies to determine |
| build order, but then build each project in the same manner as Eclipse 2.1. A |
| build strategy extension point would allow plugins to provide a subclass of |
| BuildStrategy: |
| <pre> |
| public abstract class BuildStrategy { |
| public abstract void build(IProject[] projects, IProgressMonitor monitor); |
| public final void build(IProject project, IProgressMonitor monitor) {} |
| public final int getKind() {...} |
| ... other methods to provide additional context if needed ... |
| } |
| </pre> |
| </p> |
| <p> |
| A build strategy is a first class participant in the build progress. Each installed |
| strategy is invoked for every single build (workspace, project, incremental, |
| full, autobuild, etc). A strategy instance is given an array of projects to build, |
| and it must call BuildStrategy.build(IProject, IProgressMonitor) on the projects that it is interested |
| in building. This callback will then invoke all the builders on that project in the order |
| specified on the project build spec (same as previous Eclipse releases). The strategy |
| may build zero or more projects in the array, and it may build the same project |
| multiple times (in the case of cycles). |
| </p> |
| <p> |
| If multiple strategies are present, they are invoked in no |
| particular order. Strategies are only given the "left over" |
| projects that no previous strategy chose to build. Finally, a default platform |
| strategy runs to build the remaining unbuilt projects, using project references to determine |
| the order. |
| </p> |
| <p> |
| (Note: I am calling this solution "build strategy" to avoid confusing it with S1. If we |
| go with this route, it could easily be renamed to "workspace builder" |
| since everyone seems to like that term.) |
| </p> |
| <p> |
| <b>Pros:</b> A build strategy can be used as an alternative to project references |
| for setting the project build order. Since this mechanism is dynamic, it fits with the |
| JDT dynamic build path story and avoids modifying the project description file every |
| time the build path changes (solving problem P1). This hook also gives a build strategy |
| implementor all the context information they need, including the complete set of |
| projects being built, and an opportunity to add pre- and post-processing on the |
| entire build cycle (solving problem P2). |
| </p> |
| <p> |
| This approach is also backwards compatible. If no builders made any changes, then |
| the default build strategy would continue to determine build order from project references |
| as it did in previous releases. Each builder that was influencing the build order by |
| changing project references can now supply a build strategy extension if they want |
| to compute build order dynamically. |
| </p> |
| <p> |
| <b>Cons:</b> If there are multiple strategies installed, they may only operate |
| on disjoint sets of projects. This is the same limitation that existed with project |
| references, since multiple builders setting the project references prior to Eclipse 3.0 |
| would have been overwriting each other's references. |
| </p> |
| <p> |
| This does not solve the final problem discussed in P2, the interleaving of other builders |
| between two invocations of a given builder extension. This problem cannot be |
| solved without breaking changes to the workspace build order. |
| </p> |
| <p> |
| Another drawback is that this puts responsibility for complex aspects of the build |
| infrastructure into the hands of strategy implementors. For example, handling |
| cycles in the build order, and handling a user specified build order. This increase |
| in delegated responsibility comes with the usual cost of decreased flexibility |
| to make future changes to the build infrastructure. |
| </p> |
| |
| <h4>S3. Working set build</h4> |
| <p> |
| The two build actions that currently surface in the UI are project build and workspace |
| build. A third option, working set build, could be introduced for building a subset |
| of the workspace. This option would only be visible after entering some advanced |
| mode (for example from a preference page) to avoid confusion for new users. |
| This option would only run builders on the specified set of projects, avoiding builds |
| of either prerequisite or downstream dependent projects. This option could be |
| made available in both autobuild and manual build modes. While a manual working |
| set build can be done using current API, it would be beneficial to introduce a new |
| core API method for working set builds. |
| This would ensure correct computation of build order, and allow for participation |
| by other proposed mechanisms such as workspace builders or pluggable |
| build strategies. |
| </p> |
| <p> |
| This solution aims to address only problem P3 listed above. It may work in |
| conjunction with other solutions listed in this document. |
| </p> |
| <p> |
| <b>Pros:</b> Allows user with large, heavily interconnected workspaces to |
| build smaller sets of projects more often, and defer building the entire workspace |
| until a larger number of changes have been made. |
| </p> |
| <p> |
| <b>Cons:</b> Working set autobuild in particular would introduce significant |
| complexity. To avoid compatibility problems it would have to be treated as |
| a disjoint mode from workspace autobuild. I.e., <code>IWorkspace.isAutoBuilding</code> |
| would return false when in working set autobuild mode, and some new flag could be |
| used to find out if the workspace is in working set autobuild mode. This mode would |
| be difficult to explain to both API clients and users. Workspace autobuild is simple |
| to explain: when in this mode, all projects are always in a fully built state. |
| Working set autobuild is more complex: when in this mode, and when all prerequisites |
| of projects in the working set are in a built state, then all projects in the working set |
| will remain in a built state, but all projects outside of the working set will not |
| be built. This story would be easier to explain if we required that the working set |
| be internally consistent (not referencing projects outside the working set), but this |
| assertion is difficult to verify if we introduce a dynamic build order such as |
| workspace builders or pluggable build strategies. Similar difficulties exist with |
| manual working set build, but the notion of manual build at a different scope |
| already exists. |
| </p> |
| <h4>S4. Disabling builders on a per-project basis</h4> |
| <p> |
| An alternative to working set build is to add new API for disabling build on a |
| per-project (or even per-builder) basis. The setting would apply to all forms of builds |
| (incremental ,full, auto, project, workspace). The value would be persisted by the |
| platform. Builds scoped at the workspace level (including autobulid) would simply omit |
| projects for which building is disabled. |
| </p> |
| <p> |
| This solution is essentially equivalent in functionality to working set build. It answers |
| the question "what not to build?" rather than the question "what |
| to build?". This is perhaps an easier story to explain to users (similar concept |
| in some ways to project close/open). With working sets there is confusion about |
| what types of builds it will apply to, in addition to overloading the working set |
| concept currently used in the UI for filtering views. |
| </p> |
| <p> |
| If this solution was carefully implemented, it could also solve the long standing |
| <a href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=21311"> |
| enhancement request</a> to be able to turn off the java builder (in order to replace it with |
| javac or other compiler). |
| </p> |
| |
| <h3>Proposed action</h3> |
| <p> |
| TBD |
| </p> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| </body> |
| </html> |