blob: 6729bcf800fc6e9d4fc67dd8a23769f6393d9ab4 [file] [log] [blame]
<!-- saved from url=(0022)http://internet.e-mail -->
<!doctype html public "-//w3c//dtd html 4.0 transitional//en">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Author" content="Eclipse Project">
<title>Towards More Flexible Project Structure</title>
<link rel="stylesheet" href="../../default_style.css" type="text/css">
</head>
<body bgcolor="#FFFFFF" >
<div align="left">
<h1> Towards More Flexible Project Structure</h1>
</div>
<blockquote>
<p><b>Summary</b> <br>
Eclipse requires that the contents of each project be stored in a single directory on disk.
Every file and folder in that directory tree on disk must belong to the project in the workspace.
This restriction makes it difficult to use Eclipse in conjunction with tools that have specific layout
requirements for their files, or with users who have legacy file base layouts that they need
to maintain. We would like to improve the situation for the 2.1 release of
Eclipse. There is a separate, first <a href="http://www.eclipse.org/eclipse/development/inflexible-projects-problem.html">document
describing the nature of the problem</a>. This document explores the space of
potential solutions, without drilling down into the details of any of them.<br><br>
Last Modified: 14:00 October 4, 2002</p>
</blockquote>
<p>In this document we explore the space of potential solutions, to better
understand where the various aspects of the problem can and cannot be addressed.
Although much of the discussion will center around Java, the problems are likely not
confined to
Java and will affect tools for other languages. In most cases, we will have the option of addressing the problem at the JDT
level without changing the Eclipse Platform. This has its pros and cons. Pro:
the solution can be specialized for Java. Con: The solution would not apply to C
tools, for example. On the other hand, addressing the problem at the level of
the Eclipse Platform also has its pros and cons. Pro: The solution would apply
to C tools, etc. Con: The change may impact all tools and users. All this must be
factored in to the decision-making process.</p>
<p>In the final decision, we may only be able to address a subset of the
problems. But here we proceed with an unlimited budget for exploration.</p>
<h2>Pros of the current Eclipse</h2>
<p>Although the restrictions on the structure of Eclipse projects may come
across as inflexible to some, the current scheme has several important virtues:</p>
<ul>
<li>Single-rooted Eclipse project is simple to explain to users and plug-in
writers alike: All of a project's files are in a single file system
subdirectory. End of story.</li>
<li>All-inclusive Eclipse project means that the project resource tree is an
accurate in-memory model of the corresponding tree of files in the file
system. Plug-ins can quickly traverse the in-memory tree without having to
make expensive OS file IO calls. Tools that operate directly in the file
system can be readily integrated into Eclipse with the after-the-fact
application of a simple refresh mechanism to resynchronizes the resource
tree with the file system.</li>
<li>Non-overlapping Eclipse projects is simple. The mapping from files in the
file system to files in the workspace resource tree is 1-1, allowing
unambiguous lookup of a resource given a file system path. If projects
overlap, that mapping becomes 1-many (multiple distinct resource handles map
to the same file).</li>
</ul>
<h2>What do other IDEs do?</h2>
<p>
<b>NetBeans 3.4 (</b>and Sun™ ONE Studio, formerly Forte™ for Java™)</p>
<ul>
<li>Projects are multi-rooted at the top-level.</li>
<li>To use sources and libraries in IDE they must be mounted as
filesystems.</li>
<li>Filesystem can be a local folder, an archive, or something from VCM.</li>
<li>Filesystems are all-inclusive by default.
<ul>
<li>Regular expression exclusion rules stated at filesystem root.</li>
<li>(Refresh rate also set at filesystem root; defaults to 15
s.)&nbsp;</li>
</ul>
</li>
<li>Projects provide context for source files, compilation, and execution.</li>
<li>All work done in an open project.</li>
<li>Default project if none specified.</li>
<li>Can define new projects.</li>
<li>Only one project open at a time.</li>
<li>Filesystems can be mounted into projects.</li>
<li>Filesystems can overlap between projects.</li>
<li>Filesystems can overlap even within projects.</li>
<li>Project level control of location of generated class files.
<ul>
<li>Next to source file (default).</li>
<li>Output to specific filesystem.</li>
</ul>
</li>
</ul>
<b>IntelliJ IDEA 2.6</b>
<ul>
<li>Projects are multi-rooted at the top-level.</li>
<li>Roots are directories or JARs in file system.
<ul>
<li>Automagically computes correct source folder based on where it
finds Java source files</li>
</ul>
</li>
<li>All files under root directory are included in project.</li>
<li><i>Remove</i> a root from project; <i>delete</i> a file or folder from
a root.</li>
<li>Info about project is kept in separate project configuration file
named &lt;ProjectName&gt;.ipr</li>
<li>Project has compiler option for ignoring specified files or
subdirectories.</li>
<li>Workspace consists of a single open project.</li>
<li>Default project if none specified.</li>
<li>Can define new projects.</li>
<li>Only one project open at a time.
<ul>
<li>But you can run multiple instances of IDE at same time.</li>
</ul>
</li>
<li>Separate projects are free to overlap (no special support).</li>
<li>Within a project, separate roots may overlap.</li>
<li>Source files appearing under more than one root generate compile
errors.
<ul>
<li>Cannot be remedied with compiler option.</li>
</ul>
</li>
<li>One output folder per project.
<ul>
<li>Location of generated class files is a project property.</li>
<li>(But...IDEA 3.0 is more flexible: different output folder for each
source folder)</li>
</ul>
</li>
</ul>
<b>Together® ControlCenter™ 6.0.1</b>
<ul>
<li>Projects are multi-rooted at the top-level.</li>
<li>Roots are directories or JARs in file system.</li>
<li>Roots are added and removed only in project properties dialog.</li>
<li>All files under root directory are included in project unless excluded
by a &quot;skip path&quot; entry.</li>
<li>Info about project is kept in separate project file named
&lt;ProjectName&gt;.tpr.</li>
<li>Project file is kept in primary root directory along with other
project-level property files.</li>
<li>Workspace consists of a single open project.</li>
<li>Can define new projects.</li>
<li>Only one project open at a time.
<ul>
<li>But you can run multiple instances of IDE at same time.</li>
</ul>
</li>
<li>Separate projects are free to overlap (no special support).</li>
<li>Within a project, separate roots may overlap.</li>
<li>Source files appearing under more than one root generate compile
errors.
<ul>
<li>Can be remedied with &quot;skip path&quot; feature.</li>
</ul>
</li>
<li>Location of generated class files is internal and unspecified.</li>
</ul>
<b>Microsoft Visual Studio .NET</b>
<ul>
<li>Projects are single-rooted.</li>
<li>File inclusion is inherently selective.
<ul>
<li>There are Include in Project and Exclude from Project menu items.</li>
</ul>
</li>
<li>Workspace consists of several projects (collected into solutions).</li>
<li>[TBD - whether projects are allowed to coincide or overlap]</li>
</ul>
<p>
<b>Borland® JBuilder™</b> &nbsp;</p>
<p>[This section is incomplete.]</p>
<ul>
<li>Projects are multi-rooted at the top-level.</li>
<li>For each specified root, all-inclusive is the default position, with some
filtering mechanisms.</li>
<li>Workspace consists of a single project.</li>
<li>Separate projects are free to overlap (no special support).</li>
<li>Within a project, separate roots may overlap; duplicate files are
allowed.</li>
<li>Single output folder per project.</li>
</ul>
<h2>Improving the Output Folder</h2>
<p>How could we change Eclipse to make the Java output folder more flexible?</p>
<p>Here are some options (in no particular order):</p>
<p><b>Option P-1</b>: Change JDT to allow the output folder to be external to
the workspace.</p>
<p>JDT allows library JARs to be either local to a Java project (specified by
workspace path) or external to the workspace (specified by an absolute file
system path). So one option is to be similarly flexible with the location of a
Java project's output folder.</p>
<p>This would entail changing JDT Core API as well as JDT UI, but could likely
be done in a non-breaking fashion.</p>
<p>The chief drawback of this approach is that there would be no resource deltas
for tracking changes to files in a project's output folder in the case where it
was external to the workspace. This would have serious repercussions in at least
three areas: (a) the JDT debugger's HotSwap facility relies on resource deltas to
class files in the output folder; (b) the Java incremental project builder uses
builder-supplied resource deltas to track changes to class files in dependent
projects, and (c) other incremental project builders on the same project would
be unable to track changes to anything in the output folder as well. Without resource deltas, it is hard to see how either could be
supported.</p>
<p>A smaller drawback is that any code that creates and reads class files in the
output folder would have to be written in such a way that it could deal with
resources in the workspace (using resource handles provided by the workspace
API) and outside the workspace (using java.io.File API directly).</p>
<p><b>Option P-2</b>: Change Eclipse Platform workspace to allow arbitrary
directory in the file system to be &quot;mounted&quot; as a folder inside a
project.</p>
<p>A user would then be able to mount an external file system subdirectory into
their Java project, and designate their output folder at (or below) that mount
point. The net effect is that the generated class files would be written
directly to the desired location. As long as mounted subdirectories are
considered to be a regular part of the project's file, the regular resource API
and resource deltas apply. JDT would not even need to be aware that the output
folder was separated from the rest of the project's files.</p>
<p>The pros and cons of this approach are chiefly those of mount points
generally (covered below).</p>
<p>PDE launches a runtime
workbench with a special command line option specifying one or more
plug-in-relative paths (e.g., -dev bin) that are to be included on each plug-in
class loader's class path. PDE includes the project-relative paths of all output
folders; this is just the path &quot;bin&quot; when all plug-in projects locate
their output folder in the default location. If an external directory is mounted
as a Java project's output folder, the existing Eclipse Platform Core Runtime
mechanisms would need to be revamped to allow PDE to supply an absolute file
system path for each plug-in.</p>
<p>Some other observations:</p>
<ul>
<li>JDT marks the output folder as containing &quot;derived&quot; resources,
and repository providers tend to ignore resources marked as
&quot;derived&quot;. This means that it probably does not matter whether a
mounted folder is considered part of the project for VCM purposes.</li>
<li>If the JDT UI exposes the support for mounting the output folder, there
would be no need to provide generic mount support in the Workbench UI. This
would allow the complicating nature of mount points to be confined to the
types of projects that are prepared for them.</li>
</ul>
<p><b>Option P-3</b>: Advocate use of OS-level symlinks. </p>
<p>Unix file system have symlinks, and Windows 2000 NTFS has an unsupported
feature called <a href="http://www.sysinternals.com/ntw2k/source/misc.shtml#junction">junctions</a>. These
file system mechanisms allow a directory in one part of the file system to be
linked in to somewhere else. For example, if the project root directory of
Project1 is C:/myws/project1 and the Java output folder is in its standard
location, a symlink from C:/myws/project1/bin to C:/tomcat/webapps/mytest/WEB-INF/classes
ensures that the files in the output folder are created directly in the web
server's subdirectory. </p>
<p>The advantage of this option is that the user can create this arrangement
using existing OS-specific tools, and does not require explicit support from the
Eclipse. So this option is already supported in 2.0. </p>
<p>Operating systems&nbsp; may have restrictions on symlinks that make this option less
workable for some purposes. For instance, NTFS junctions work on Windows
2000 or better, and only work within
and between local drives. Furthermore, junctions are an unsupported feature. </p>
<p><b>Option P-4</b>: Advocate use of external tool builders for copying the output folder to an
external location. </p>
<p>Eclipse allows the user to add Ant-scripted steps as an external tool
builder. Coping new and changed files from the output folder to another location
can be done with one Ant command. </p>
<p>This option is low-tech and works in Eclipse 2.0; however, it only ensures
that the files in the output folder end up in the desired location; it does not
reduce the network traffic when the project's files are stored on a network
drive. </p>
<p><b>Option Q-1 (orthogonal)</b>: Allow one output folder per source folder. </p>
<p>Orthogonal to the other options, flexibility could be improved by allowing
different output folders to be specified for each source folder. In other words,
allowing the output folder could be partitioned along source folder lines. The
changes allows, for example, a single Java project to contain both product
source code and test suites partitioned into separate source and output
folders.&nbsp; </p>
<p>The additional output folders would need to find their way onto the runtime
class path, and the debugger would need to know how to map from generated class
file to source file. Neither should be a problem. </p>
<p>This change would break existing clients that assume that all generated class
files for a Java project are in a single output folder. </p>
<h3>Assessment </h3>
<ul>
<li>P-1 is a non-starter.</li>
<li>P-2 is promising, but requires changes to the Eclipse Platform.&nbsp;</li>
<li>P-3 requires no work but is only a partial solution.</li>
<li>P-4 requires no work but is only a partial solution.</li>
</ul>
<h2>Allowing Multi-rooted Projects</h2>
<p>How could we change Eclipse to make Java projects (source folders in
particular) be multi-rooted rather than single-rooted?</p>
<p>Here are some options (in no particular order):</p>
<p><b>Option M-1</b>: Change JDT to allow a Java project source folder to lie
inside another project, or to lie outside the workspace.</p>
<p>It would be relatively straightforward to allow a source folder for a Java
project to lie inside a different Java project. Because these files are still
inside the workspace, resource deltas can still be used for updating the Java
model and triggering recompilation.</p>
<p>Supporting a source folder outside the workspace entirely would have many of
the same problems as with external output folders. These files would need to be
accessed with java.io.File API (not resource handles), and there would be no
resource deltas for updating the Java model or triggering recompilation.</p>
<p><b>Option M-2</b>: Change Eclipse Platform to &quot;mount&quot; a file system
directory as a project folder.</p>
<p>We would add a new workspace concept of mount points, and allow a directory
in the file system to be grafted into a project's resource tree. For example,
mounting C:/JavaSources at /Project1/src would mean that the files and folders
under /Project1/src are stored under the subdirectory C:/JavaSources; e.g., the
file C:/JavaSources/com/example/Egg.java would appear in the workspace at the
resource path /Project1/src/com/example/Egg.java. Any project files not below a
mount point would be held within the project's root directory, as they are
currently. Tools that navigate the resource tree using the workspace API need
not be aware of the file's location.&nbsp;</p>
<p>It would also be possible to allow a portion of one project's resource tree
to be directly grafted into another (instead of doing this indirectly through the file system).
This implies some form of overlap be allowed.</p>
<p>
With the ability to mount a file system directory into the root of a project, a
Java project can have its source folders in different locations on disk.
</p>
<p>
This would be a breaking change for clients of the workspace because it alters the current
implicit understanding that all project contents are stored in one file system
subtree. However, it should be possible to minimize the effects of this breakage
and provide good compatibility with existing 2.0 tools and team (VCM) providers.&nbsp;
</p>
<p><b>Option M-3</b>: Advocate use of OS-level symlinks. </p>
<p>The use of symlinks is discussed above (Output Folder, Option P-3). These are
the file system equivalents of the mount points in Option M-2. </p>
<p>The advantage of this option is that the user can create this arrangement
using existing OS-specific tools, and does not require explicit support from the
Eclipse. So this option is already supported in 2.0. </p>
<p>Operating systems&nbsp; may have restrictions on symlinks that make this option less
workable for some purposes. For instance, NTFS junctions work on Windows
2000 or better, and only work within
and between local drives. Furthermore, junctions are an unsupported feature. </p>
<h3>Assessment </h3>
<ul>
<li>M-1 is a non-starter.</li>
<li>M-2 is promising, but requires changes to the Eclipse Platform (same
change as P-2).&nbsp;</li>
<li>M-3 requires no work but is only a partial solution.</li>
</ul>
<h2>Allowing Selectivity</h2>
<p>How could we change Eclipse to make the Java source folders selective rather
than all-inclusive?</p>
<p>Here are some options (in no particular order):</p>
<p><b>Option S-1</b>: Change Eclipse Platform to add support for resource working sets with the capability to browse and manipulate a subset of the workspace where certain files and folders are excluded.</p>
<p>This option does not involve changing the nature of the workspace resource
tree and its relation to the file system. It simply adds ways to provide
filtered views of that resource tree.</p>
<p>A Java project source folder is currently specified with a project-relative
folder path. All *.java source files found in the project subtree are
automatically included in the Java model and on the Java compiler's class
path. The source folder specification could be enhanced to permit the user to
specify which files and folders to exclude (think <a href="http://jakarta.apache.org/ant/manual/CoreTypes/fileset.html">Ant
filesets</a>).</p>
<p>The source folders of a Java project are not currently allowed to coincide or
otherwise overlap. Some relaxation of this rule is required to handle the case where a source folder
containing tests is nested inside the main source folder (<a href="http://www.eclipse.org/eclipse/development/inflexible-projects-problem.html">Example
3</a>). One relaxation is to allow source folders to coincide or overlap but use the mechanism for excluding
files to prevent the same file from being included in both source folders.</p>
<p>This option would entail changing Eclipse Platform Core to provide infrastructure, possibly in the form of a rule-based resource
visitor. We would explore a possible connection to the existing UI notion of working sets. JDT Core API
and JDT UI would need to change as well, but could
likely be done in a non-breaking manner by adding working sets to
classpath entries (<a href="http://dev.eclipse.org:8080/help/content/help:/org.eclipse.jdt.doc.isv/reference/api/org/eclipse/jdt/core/IClasspathEntry.html">IClasspathEntry</a>)
and revamping the rules for legal source folder overlap.</p>
<p><b>Option S-2 (orthogonal)</b>: Change JDT to allow overlapping source folders but
automatically exclude the overlapped portion from the parent.</p>
<p>A different approach is to provide the kind of selectivity required to handle
cases like where a source folder
containing tests is nested inside the main source folder (<a href="http://www.eclipse.org/eclipse/development/inflexible-projects-problem.html">Example
3</a>). The solution is to allow the overlap, but automatically exclude the
files accounted for by the more specific folder from the less specific one. For
example, if /src/ and /src/tests/ as both specified as source folders, files
under /src/tests/ would only appear under the latter source folder, and the
former source folder would act as if it had no child named tests.</p>
<p>This option would involve a non-breaking change to JDT to remove the current
restrictions that prohibit overlap and define how overlapping source folders are
interpreted. This option is orthogonal to the other selectivity options, and
could be done in conjunction with any of them.</p>
<p><b>Option S-3</b>: Radically change the Eclipse Platform policy for adding and removing
files from the workspace.</p>
<p>The workspace folder refresh operation (<a href="http://dev.eclipse.org:8080/help/content/help:/org.eclipse.platform.doc.isv/reference/api/org/eclipse/core/resources/IResource.html#refreshLocal">IResource.refreshLocal</a>) automatically creates entries in the workspace resource tree
for newly discovered files and folders out in the file system, and discards
entries for any files and folders that are no longer present in the file system.
(This refresh action is also implicit in a number of workspace operations when
the force flag is used). This characteristic makes a project's workspace
resource tree a more-or-less faithful mirror of the project's subtree in the
file system. This gives workspace projects their all-inclusive feel.</p>
<p>There are other IDEs (including Visual Studio .net) that allow the user to
control which files and folders belong in a project; other files and folders may
exist in the local file system, but are simply ignored by the IDE if not flagged
as belonging to the project. This characteristic means that the IDE's notion of
project is more like a &quot;view&quot; onto files in the file system. This
alternate conception of workspace could be achieved by changing Eclipse policy
so that file and folder entries are never added to or removed from the resource
tree due to a refresh operation; new API would be added so that entries could be
added or removed explicitly. This would give workspace projects a selective
feel.</p>
<p>On the plus side, the resource tree would be smaller if some files and
folders were to be excluded, and this would improve performance. It would also
reduce workspace clutter and make Eclipse behave more like certain other popular
IDEs (Visual Studio .NET).</p>
<p>The downside of this option is that it is impossible to do while maintaining
compatibility. Consider an existing tool that is calling IResource.refreshLocal
to cause files created directly in the file system to appear in the workspace (a
common problem when bridging external programs to Eclipse). Changing the
semantics of IResource.refreshLocal so that it does not discover new files would
clearly break this tool. On the other hand, if the semantics of
IResource.refreshLocal is not changed, then running this tool would
automatically admit files to the workspace, even against the user's wishes.
Either path leads to a different kind of breakage.</p>
<p><b>Option S-4</b>: Add Eclipse Platform mechanism to prevent refreshLocal from
adding certain files to the workspace resource tree.</p>
<p>Another option is to change the Eclipse Platform refresh mechanism is a more
modest way to provide a way to selectively exclude files. If the refresh
operation was governed by exclusion rules, it would be possible for the user to
explicitly prevent certain files and folders in the file system from being added
automatically to the workspace. By default, there would be no exclusion rules,
and the workspace would behave exactly as in 2.0. If the user indicates that a
particular file or folder is to be excluded, the refresh operation would never
add it to the resource tree, even if the file or folder exists in the file
system. This would give workspace projects an initially all-inclusive feel which
could be made more selective as required.</p>
<p>New API would be required to change the exclusion rule set, to explicitly add
a file or folder to the resource tree even if on the excluded list, and to
remove a file or folder from the resource tree without deleting the file in the
file system. The semantics of IResource.refreshLocal would not have to change in
a breaking way. A project's workspace
resource tree is a more-or-less faithful mirror of the project's subtree in the
file system except for the files and folders that the user has explicitly taken
off the table.</p>
<p>On the plus side, the resource tree would be smaller if some files and
folders were to be excluded, and this would improve performance. It would also
reduce workspace clutter and make Eclipse behave more like certain other popular
IDEs. On the minus side, it does mean that in-memory workspace resource tree is
not quite as faithful a&nbsp;mirror of the file system.</p>
<p><b>Option S-5</b>: Add Eclipse Platform mechanism to hide certain files in the workspace resource
tree.</p>
<p> Eclipse Platform 2.0 has a &quot;team private member&quot; mechanism for
hiding certain resources. These resources are represented by entries in the
workspace resource tree, but are explicitly marked as hidden so that clients
will not see these resources unless they ask to see them. Imagine that this
mechanism was generalized and extended so that other resources could be
similarly hidden from clients that do not ask for them. By default, there would be no
hidden files,
and the workspace would behave exactly as in 2.0. The refresh operation would
automatically add and remove files as in 2.0. The user could explicitly control
whether they were marked as hidden, and whether hidden files were shown in the
UI. This would give workspace projects an all-inclusive feel
which the user
could make more selective as required.</p>
<p>New API would be required to change the hidden flag and to include or exclude
hidden resources when navigating the workspace. The semantics of IResource.refreshLocal would not have to change. A project's workspace
resource tree is a more-or-less faithful mirror of the project's subtree in the
file system except for the files and folders that the user has explicitly taken
off the table.</p>
<h3>Assessment </h3>
<ul>
<li>S-1 is promising.</li>
<li>S-2 is also promising; it provides less flexibility that S-1, but is also
simpler all around.</li>
<li>S-3 is a non-starter.</li>
<li>S-4 is promising.</li>
<li>S-5 is promising.</li>
</ul>
<h2>Allowing Overlap</h2>
<p>How could we change Eclipse to allow Java source folders to overlap?</p>
<p>Here are some options (in no particular order):</p>
<p><b>Option V-1</b>: Change JDT to allow overlap within a project's source
folders.</p>
<p>This option is discussed in S-1 and S-2.</p>
<p><b>Option V-2</b>: Allow &quot;mounted&quot; file system directories to
overlap.</p>
<p>One of the design details of option M-2 is whether to allow mounted
directories to overlap, either within a single project or across all projects in
the workspace.&nbsp;</p>
<p> The simple interpretation of overlapping mount points would mean that the same file on
disk might have multiple distinct resource handles. This would create a semantic
problem for the IWorkspaceRoot.getContainer/FileForLocation API methods which
presuppose that there is at most one resource handle for any candidate file system path.</p>
<p>Other issues if two resources /P1/Foo.txt and /P2/Foo.txt correspond to the
same file C:/Sources/Foo.txt:</p>
<ul>
<li>Refresh: Does refreshing one resource also refresh the other?</li>
<li>VCM: How to decide which project is responsible for VCM?</li>
</ul>
<h3>Assessment </h3>
<ul>
<li>V-1 is promising.</li>
<li>V-2 is promising, but has some API breakage, and some semantic questions
that need to be worked through.</li>
</ul>
<p>Options V-1 and V-2 are not necessarily mutually exclusive. There may be reason to allow overlapping source folders regardless of mount points, and vice-versa.</p>
</body>
</html>