| <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN2"> | |
| <html> | |
| <head> | |
| <title>Update Site Proposal</title> | |
| <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> | |
| <meta http-equiv="Content-Style-Type" content="text/css"> | |
| <link href="http://dev.eclipse.org/default_style.css" rel="stylesheet" type="text/css"> | |
| </head> | |
| <body bgcolor="#ffffff"> | |
| <table width="100%"> | |
| <tr><td style="background:#0080C0"><b><span style="color:white">Proposal</span></b></td></tr> | |
| </table> | |
| <h1>Eclipse Update Site Proposal</h1> | |
| <blockquote> | |
| <p><b>Summary</b> <br> | |
| The current packaging of the Eclipse SDK as a set of large zips (one per platform) | |
| is a convenient one stop shop for getting up and running with the SDK. Unfortunately, | |
| this approach does not scale well. As more and more Eclipse projects (e.g., | |
| EMF, GEF, ...) come into common use, there is increasing interest in including | |
| these in pre-packaged zips. Similarly, as Eclipse is used in more varied scenarios, | |
| so increases the demand for different packagings (smaller and larger). It | |
| is not feasible for eclipse.org to create, manage and distribute pre-packaged | |
| configurations to meet all the demands. This proposal details a structure | |
| which enables Eclipse component providers to contribute their features and | |
| plug-ins and for Eclipse consumers to efficiently select, download and install | |
| their components of interest.</p> | |
| <p> Last Modified: November 22, 2004</p> | |
| </blockquote> | |
| <h2>Problem Definition</h2> | |
| <p>The base Eclipse function is currently distributed as roughly the following | |
| zip files (as of 3.1M3)</p> | |
| <ul> | |
| <li>RCP Binary (one per target OS/WS) (8 x ~5MB)</li> | |
| <li>RCP SDK (one per target OS/WS) (8 x ~18MB)</li> | |
| <li>Platform Binary (one per target OS/WS) (8 x ~25MB)</li> | |
| <li>Platform SDK (one per target OS/WS) (8 x ~56MB)</li> | |
| <li>JDT Binary (OS/WS independent) (~15MB)</li> | |
| <li>JDT SDK (OS/WS independent) (~27MB)</li> | |
| <li>PDE Binary (OS/WS independent) (~4MB)</li> | |
| <li>PDE SDK (OS/WS independent) (~5MB)</li> | |
| <li>Full Eclipse SDK zips including source and all doc (one per target OS/WS) | |
| (8 x ~88MB)</li> | |
| <li>assorted other zips (e.g., examples, webdav, ...)</li> | |
| </ul> | |
| <p>Eclipse tools and technology project downloads are available separately and | |
| come in a variety of flavours, sizes and organizations. People wanting the base | |
| SDK are quite comfortable with the current situation. They need only download | |
| the single zip and go. If they need to run on two different platforms they download | |
| the zip for that platform and ignore the fact that 95% of the second download | |
| is the same as the first.</p> | |
| <p>Users looking to combine the base Eclipse with additional plug-ins/features | |
| are somewhat less happy. They have to search over several web sites, identifying | |
| downloads (zips) of interest and correlating versions manually. In general, | |
| the suppliers of these additional zips do not make it easy for users to understand | |
| their offerings.</p> | |
| <p>The main issues with the current setup are:</p> | |
| <ul> | |
| <li>Does not promote the easy provision, acquisition and use of plug-ins from | |
| a variety of sources. The basic premise of Eclipse is that plug-in consumers | |
| can gather a set of interesting plug-ins, put them together in an Eclipse | |
| platform configuration and enjoy an integrated seemless environment. Currently | |
| plug-in | |
| producers provide their plug-ins in various forms and the lack of any managed | |
| acquisition mechanism forces potential users to grovel around for plug-ins | |
| and then hope that they got the right thing.</li> | |
| <li>Does not scale as the scope of Eclipse increases. The current approach addresses | |
| the provision/acquisition problem by packaging collections of interesting | |
| plug-ins together in a single zip. This works for Eclipse since it is at the | |
| bottom of the plug-in stack. However, others providers are not so lucky and | |
| are challenged with questions like "how many of my prerequisites should | |
| I include in my download?". Further, as more and more plug-ins are added | |
| to the Eclipse world, the size of the complete zip increases. A zip of a reasonable | |
| set of SDK drops from projects on eclipse.org could easily reach 150MB (i.e., | |
| double the Eclipse SDK size). Following the current model, there would be | |
| one of these uber-zips per target OS/WS.</li> | |
| <li>Does not scale as the Eclipse use scenarios increase. As the range of people | |
| using Eclipse broadens, there will be increasing demand for alternative | |
| collections | |
| of plug-ins. These will contain either more or less than the Eclipse SDK. | |
| While we can take a stab and produce some common packagings, we are are | |
| not likely to satisfy all/most people.</li> | |
| <li>Makes poor use of existing disk and bandwidth resources. As mentioned earlier, | |
| something like 95% of each OS/WS specific SDK zip is identical content. Further, | |
| from build to build, not all plug-ins change. The user doc for example tends | |
| to change slowly. Near the end of a milestone or release cycle we frequently | |
| rebuild to fix isolated problems. Rather than rebuilding just the plug-ins | |
| in question and providing these for download, we rebuild the whole stack and | |
| repackage an entire set of full zips. Subsequently, interested developers | |
| are forced to redownload the bulk of what they already have. This wastes diskspace | |
| and bandwidth for both eclipse.org and our users.</li> | |
| <li>Finally, it is somewhat ironic that the update/install mechanism has been | |
| designed in part to address exactly these kinds of concerns yet it is one | |
| of the few components that does not see regular use by the Eclipse team itself. | |
| As a result, the update/install team misses out on the regular feedback | |
| enjoyed | |
| by the other Eclipse teams. It is seemingly hard to justify putting update/install | |
| forward as a useful technology while not using it ourselves. </li> | |
| </ul> | |
| <h2>Scenarios</h2> | |
| <h3>Scenario 1: minimal install</h3> | |
| <p>User starts with a minimal Eclipse base install. This contains just the runtime, | |
| some UI elements and an Update manager UI. Based on this they want to build | |
| an install containing additional elements from various sources.</p> | |
| <ol> | |
| <li>download and unzip the minimal install form eclipse.org or a mirror</li> | |
| <li>start the new install </li> | |
| <li>the user is presented with an update manager UI </li> | |
| <li>user discovers or is presented with update sites </li> | |
| <li>user selects features to install (the candidates presented are appropriate | |
| to the current environment e.g., OS/WS/etc as well as already installed | |
| components)</li> | |
| <li>the selected features are downloaded</li> | |
| <li>the features are installed into the current configuration</li> | |
| <li>the appropriate product/application (one that was just downloaded) is started | |
| and the update manager GUI goes away</li> | |
| </ol> | |
| <h4>Variations</h4> | |
| <p>After following the above scenario, the user can manage their configuration | |
| either using the steps outlined in Scenario 2 or by restarting their configuration | |
| with an indication that the update manager UI should be started. This puts them | |
| back at step 3 but with all their previously installed plug-ins still installed.</p> | |
| <h3>Scenario 2: building on a product install</h3> | |
| <p>In this scenario, the user has an Eclipse product installed (possibly just | |
| the Eclipse SDK from a downloaded zip file or the configuration from the steps | |
| in scenario 1) and wants to add more function. This pretty much follows the | |
| normal update/install scenarios.</p> | |
| <ol> | |
| <li>user installs an eclipse based product (e.g., Eclipse SDK)</li> | |
| <li>at some point, while running the product, the user invokes the update manager | |
| UI </li> | |
| <li>user discovers or is presented with update sites </li> | |
| <li>user selects features to install (the candidates presented are appropriate | |
| to the current environment e.g., OS/WS/etc as well as already installed | |
| components)</li> | |
| <li>the selected features are downloaded</li> | |
| <li>the features are installed into the current configuration</li> | |
| <li>the installed function is now available to the user</li> | |
| </ol> | |
| <h3>Scenario 3: Web install</h3> | |
| <p>Note: This is an advanced scenario that may not be supported in the near term.</p> | |
| <p>This scenario is very much like scenario 1. The main difference is in how the | |
| initial install is obtained. Rather than manually downloading and unzipping | |
| the minimal install, JNLP (Webstart) is used to fetch and run the minimal install. | |
| </p> | |
| <ol> | |
| <li>user browses some website and clicks on a JNLP link</li> | |
| <li>the appropriate jars are downloaded and "installed" using the | |
| JNLP mechanism</li> | |
| <li>the minimal Eclipse install is started </li> | |
| <li>go to step 3 of scenario 1</li> | |
| </ol> | |
| <h2>Problem Solution</h2> | |
| <p>We propose to use the Eclipse update technology to address the described problems | |
| and implement the listed scenarios. Note that this new structure is a complement | |
| to the existing large zip packagings of Eclipse. We are not proposing to stop | |
| creating the familiar SDK etc zips. (we leave it to someone more brave to propose | |
| that!)</p> | |
| <h3>Update site</h3> | |
| <p>eclipse.org (and others) will provide update sites sporting all the relevant | |
| features and plug-ins. Collections of projects can cooperate and share an update | |
| site or each supply/support their own update site. Managing your own eliminates | |
| the need for some sort of shared site update mechanism. However, the user must | |
| be able to navigate these individual sites seamlessly (see the section on Update | |
| manager). </p> | |
| <p>There will be one update site per <em>release family</em>. A release family | |
| is a set of releases which form a substantial and separable effort. For example, | |
| Eclipse 2.0, 2.1, 3.0 and 3.1 are all different release families. The life | |
| of release family continues after its initial release (e.g., 3.0) to include | |
| maintenance releases (e.g., 3.0.1, 3.0.2, ...). Isolating release families | |
| to their own update sites helps maintain a separation between development going | |
| on for the next release family and maintenance work on the previous family.</p> | |
| <h3>Versioning</h3> | |
| <p>Update sites are not possible unless we use the Eclipse version numbering | |
| scheme effectively. The main challenge is for plug-ins to have version numbers | |
| that | |
| accurately relate to their content. The essential element here is to ensure | |
| that if two plug-ins have the same id and version, their content is identical. | |
| The | |
| easiest | |
| way to do this is to use the version qualifier (i.e., the forth version element). | |
| For normal (not nightly) builds, the qualifier should be the version tag from | |
| the map file used for the build | |
| which generated them (e.g., 3.1.0.v20041122). The map file defines the content | |
| going into the build. If that does not change, then we can assume that the | |
| content | |
| of the output plug-in will not change. In this way, identical plug-ins will | |
| not be duplicated on the update site and consumers will not have to download | |
| them | |
| if they already have them. For more detail on plugin versioning, see the <a href="http://dev.eclipse.org/viewcvs/index.cgi/%7Echeckout%7E/platform-core-home/documents/plugin-versioning.html">Plug-in | |
| Versioning Proposal</a>.</p> | |
| <p>Features on the other hand are collections of plug-ins. They are also relatively | |
| small. Since the content of a feature is indirectly defined by the content | |
| of the plug-ins included in the feature, it is hard to detect change. The | |
| safe bet | |
| here is to use the build stamp as the qualifier for feature version (e.g., | |
| 3.0.0.200402131200). This has the benificial effect of clearly identifying | |
| the features for a particular | |
| build. Further, since the entire feature set is regenerated, the collection | |
| of features with the same version number completely defines the entire build | |
| output.</p> | |
| <p>Nightly builds do not go into an update site. There is no easy way to correlate | |
| the content in a nightly build (out of HEAD) with that of an integration build | |
| (using versioned resources). As a result, all plug-ins would be new and the | |
| economies of the proposed solution would not be realized. Further, nightly | |
| builds are | |
| primarily there for the individual teams to ensure they have not adversely | |
| affected other components etc. That is, they do not figure prominently in | |
| the scenarios | |
| driving this proposal.</p> | |
| <p>Plug-in directories currently reflect the id and version of the plug-in. With | |
| the addition of the map stamp, the directory names will grow by typically ~10 | |
| chars). It is unlikely but this may push some path length limits. The use of | |
| the map stamp in a file/dir name may also place additional constraints the set | |
| of chars that can be used in a map stamp (i.e, CVS tag).</p> | |
| <p>Sidenote on version qualifiers: The qualifier (fourth segment) is | |
| not interpreted. Qualifiers are compared using lexographical sorting. As such, | |
| care should be | |
| taken to ensure that the qualifier for subsequent builds monotonically increase. | |
| For example, if a build type identifier is to be included in the qualifier, | |
| it should be appended and | |
| the build date used | |
| as the main | |
| body (e.g., 20040217-I and 20040213-M7). See the complementary proposal on | |
| version numbering for more information.</p> | |
| <h3>Update manager</h3> | |
| <p>The main challenges for the update manager are in making all of this scale | |
| and usable. </p> | |
| <ul> | |
| <li>the UI must support a structured (hierarchical?) view of the available features. | |
| This view (and in fact the underlying feature structure) must be able to span | |
| update sites/servers. </li> | |
| <li>users must be able to select some set of root features and versions and | |
| then click "all required features". Again, this should span update | |
| sites/servers</li> | |
| <li>using these capabilities, user can then create their own uber-features/sites | |
| which simply reference features/sites. This becomes in effect a favourites | |
| list. They select their favourite root, click "all required" and | |
| get everything they need.</li> | |
| <li>the update manager mechanism may cache downloaded elements | |
| (features and plug-ins) for use in future install operations. That is, if | |
| you already have the plug-in/feature, you don't need to download it again. | |
| The | |
| cache location should be (optionally) set by the user.</li> | |
| <li>ideally the update/install scenarios described will be possible when running | |
| offline providing that all required elements have been previously cached. | |
| That is, | |
| if the user has already installed the features for a build, she should be | |
| able to create a new install without touching a server.</li> | |
| <li>as an advanced (i.e., longer term) item, update manager needs a way of | |
| managing its cache/store -- users need a way of purging old elements. Given | |
| a set | |
| of 1000 | |
| plugin jars | |
| of various | |
| versions, | |
| determining which to keep and which to delete is not possible without working | |
| from higher level feature groupings. In the near term, the user can delete | |
| the whole cache.</li> | |
| </ul> | |
| <p>Interesting side note: using the runtime's new ability to run directly out | |
| of jars, it is possible to have many different eclipse configurations sharing | |
| the same actual plug-in files from the update manager's cache/store. This will | |
| break some assumptions about plug-ins being together in a plug-ins directory etc | |
| (the update reconciler may get upset) and it introduces some challenges for | |
| cache clean up but it is an interesting possibility.<br> | |
| </p> | |
| <h3>PDE Build</h3> | |
| <p>PDE Build already updates feature.xml files with the version numbers of the | |
| plug-ins involved in the build. To support the version numbering scheme outlined | |
| above, PDE Build will be updated to </p> | |
| <ul> | |
| <li>modify the plug-in version number in each plugin.xml (i.e., add the map stamp). | |
| The version number of a plugin.xml is updated only if it contains <_map | |
| stamp_> as the version qualifier. This allows plug-ins to opt out of this | |
| update mechanism.</li> | |
| <li>modify the version number in the feature.xmls (i.e., add the build stamp). | |
| The version number of a feature.xml is updated only if it contains <_map | |
| stamp_> as the version qualifier. This allows features to opt out of this | |
| update mechanism. </li> | |
| <li>(advanced) modify the version number on <import> statements. Some | |
| plug-ins and features may specify a perfect match rule in when listing their | |
| dependencies. While this is not recommended, it is certainly possible. Unfortunately, | |
| there is a conflict if the intention was that plug-in A depend perfectly on | |
| the version of plug-in B with which it was built. During the build, B's version | |
| number will be updated according to its map stamp. Plug-in A's <import> | |
| statement for B needs to be updated to reflect this new version. While this | |
| modification is technically possible, it requires more complex analysis of | |
| the plugin.xml file. Updating the version of the plug-in itself can be done | |
| using simple string replacement. In this case the builder has to | |
| <ul> | |
| <li>look at each <import> using perfect match and wanting build-time | |
| map stamp substitutoin, </li> | |
| <li>identify the pluign being imported, </li> | |
| <li>discover the new version number</li> | |
| <li>replace the "substitutution desired" flag in the version number | |
| with the correct version number without loosing any other formatting, | |
| comments etc.</li> | |
| </ul> | |
| </li> | |
| </ul> | |
| <p>Note: the plugin.xml updating outlined above must be carried out on any manifest.mf | |
| files present in the build.</p> | |
| <p>PDE Build will also be updated to output directly to update site format. This | |
| is already largely supported but there maybe some modest work required.</p> | |
| <p>A possible longer term advance is to change PDE build to only build the plug-ins | |
| which actually changed. Since we know the map stamp of the plug-ins we want in | |
| the output, we can first look for those in the update site (or some cache). | |
| If present, there is no need to rebuild them. This will have a huge (positive) | |
| impact on rebuilds during release cycles.</p> | |
| <h2>Summary</h2> | |
| <p>The problem outlined above is making it hard to scale Eclipse. It is hard for | |
| people to use large numbers of Eclipse components (e.g., EMF, GEF, ...) and | |
| we are failing to put forward a good model of how others would build/stock component | |
| systems. The ability to do this easily is key to the overall Eclipse component | |
| model. The steps proposed here are a good first try at solving the problem. | |
| The result will not be perfect but will allow us to understand what is wrong | |
| and fix it. Fortunately, the bulk of the technology required exists in Eclipse | |
| today. Also, since we are proposing a complementary approach to the current | |
| download strategy, we can stage the implementation of this new mechanism both | |
| as we have time and as we learn more.</p> | |
| <p>The key steps that should be taken for 3.0 are </p> | |
| <ul> | |
| <li>create/populate an update site</li> | |
| <li>update PDE build to automate the stamping of plug-ins and features</li> | |
| <li>a modest amount of work on update manager to enable reasonable workflows | |
| (i.e., set a stake in the ground)</li> | |
| </ul> | |
| </body> | |
| </html> |