blob: bce264ace610a3525b5180c581d5dc95504c1643 [file] [log] [blame]
<?php
/*******************************************************************************
* Copyright (c) 2015, 2016 Eclipse Foundation and others.
* All rights reserved. This program and the accompanying materials
* are made available under the terms of the Eclipse Public License v1.0
* which accompanies this distribution, and is available at
* http://eclipse.org/legal/epl-v10.html
*
* Contributors:
* Eric Poirier (Eclipse Foundation) - Initial implementation
* Christopher Guindon (Eclipse Foundation)
*******************************************************************************/
// This file must be included
if(basename(__FILE__) == basename($_SERVER['PHP_SELF'])){exit();}
?>
<h1 class="article-title"><?php echo $pageTitle; ?></h1>
<p><i>Xpect is a framework that helps you to specify, test and discuss Xtext languages.</i></p>
<p>
<a target="_blank" href="http://www.xpect-tests.org">Xpect</a> has been around for a while <a target="_blank" href="https://github.com/meysholdt/Xpect">on GitHub</a>. The <a target="_blank" href="http://blog.moritz.eysholdt.de/2013/09/introduction-to-xpect.html">first announcement</a> is actually from 2013. Since then, even though there was never much marketing behind Xpect, it has seen a spectacular adoption among Xtext-projects. A large one, <a href="https://www.eclipse.org/n4js/">N4JS</a>, has recently moved to the Eclipse Foundation.
</p>
<p>Now we are happy to announce the next step: Xpect is <a target="_blank" href="https://twitter.com/i/web/status/893096871761195009">becoming an Eclipse project</a>. As of writing this, the <a href="https://projects.eclipse.org/reviews/xpect-creation-review">creation review is imminent</a>.
</p>
<p>Before I start explaining what Xpect is exactly, I would like to describe the context in which it is useful: Create Xtext
languages. Eclipse Xtext makes it very easy to implement smart editors, code generators, or compilers for your existing or self-designed
languages. The smart editors offers most of the nice features you probably love from Eclipse Java Tooling: Semantic Highlighting,
Content Assist, Live-Validation, Outline, Refactoring, etc. Also, there is Incremental Building, a global Index, and much more. Now when
you and your team implement an Xtext languages, you are in fact working on the grammar file, the ScopeProvider to define how cross
references are resolved, the validator, the generator, etc. During this work you typically:</p>
<ul>
<li>want to have automated tests. Tests give you this blissful peace of mind when refactoring your code and when rolling out the
next release.</li>
<li>review language syntax, semantics, and tooling behavior. Often, the review should be done by a domain expect who may not be a
Java programmer.</li>
<li>set up test stubs to make it easy for fellow developers to implement tests.</li>
</ul>
<p>Xpect supports these use cases by embedding tests and test-expectations inside comments inside documents of your language.
Basically, you create a an example document written in your Xtext-language and at any point you can put in a comment, saying, for
example, "here I expect a a validation error", "I expect the next cross reference to link to xyz". I will explain the actual syntax
later in this article. With this approach, Xpect is based on the following principles:</p>
<ul>
<li><b>Separate test implementation and test data+expectation</b>. This is great to have domain experts review your tests without the
needing to understand Java or Xtext's API.</li>
<li><b>Fast test creation</b>. Since a test is in fact a document written in your Xtext language, you can simply use the Xtext editor of
your language to create tests. Additionally, there is special editor that combines support for your language with
Xpect-syntax-support.</li>
<li><b>Tests should be self-explanatory on failures</b>. You, as a developer, break tests regularly when you refactor the underlying code.
Good tests really make a difference when a failing tests immediately lets you understand the cause of the problem. Tests that require
you to launch a debugger to unveil the problem are a time-sink.</li>
<li><b>Tests should run fast</b>. Almost all tests can run without OSGi/SWT and thus launch and finish in milliseconds. Furthermore, they
may use a shared setup. This allows tests to run tremendously fast and developers are actually running them locally, because that's
faster than triggering a Jenkins job.</li>
<li><b>It's OK when test expectations are a bit verbose</b>. This helps understanding them and makes good documentation. If you're worried about having slightly redundant tests that break more often than necessary: That's not a problem since the tests are easy to fix with
the "Synchronize Expectation and Actual Test Result" mechanism. Of course, this not an absolute statement and a good sense for "the
right amount" is needed.</li>
<li><b>Self-contained tests</b>. Self-contained tests are easier to understand tests: Both on failure and for domain experts, because no
exploration in other files or methods is needed to gather all relevant information.</li>
<li><b>No premature test failures</b>. Asserting a test result should happen with a single Assert-Statement, if possible. Unit tests in
which a failing assertion prevents subsequent assertions from being executed can be unnecessarily hard to analyze.</li>
</ul>
<h2>One File, Two Languages</h2>
<p align="center"><a target="_blank" href="images/xpect_editor_junit_view.gif"><img class="img-responsive" src="images/xpect_editor_junit_view.gif" /></a></p>
<p>What you can see in the screen-cast above is a JUnit test that has been executed and which passed. The test class is <i>org.domainmodel.tests.validation.DMValidationTest</i>
and it has executed a file named <i>test1.dmodel.xt</i>. The file defines two test cases, one called <i>warnings</i> and one called <i>errors</i>.
On the right hand side you can see the file's contents in an editor. The editor generically combines support for two languages: First,
the language that is being tested. In the screenshot this is the Domainmodel language (*.dmodel) which ships with Xtext as an example
language. Secondly, the editor supports the Xpect language (*.xt). The two languages do not interfere with each other since the
Domainmodel language ignores text insides comments and the Xpect language ignores text that doesn't start with an "XPECT" keyword. The
editor applies a greenish background color to Xpect syntax.
</p>
<p>
Let's take a closer look at the Xpect syntax. At the beginning of <i>test1.dmodel.xt</i>, there is a region called <span
style="font: 11.0px Monaco; background-color: #c5f9e0; color: #931a68">XPECT_SETUP</span>. It holds a reference to the JUnit test that
can run this file. Further down we find test cases such as:
</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Monaco; color: #4e9072">// capitalized property names are discouraged</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Monaco">
<span style="color: #4e9072;">// </span><span style="background-color: #c5f9e0; color: #931a68">XPECT</span> warnings --&gt; <span
style="background-color: #c5f9e0;">"Name should start with a lowercase" at "Property1"</span>
</p>
<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Monaco">Property1 : String</p>
<p>
The first line is an optional title for the test. The <i>warnings</i> references is a JUnit test method (implemented in Java) and the
part following the <span style="white-space: nowrap;"><i>--&gt;</i></span> is the test expectation, which is passed as a parameter into
the JUnit test method. The test method can then compare the test expectation with what it calculated to be the actual test value and
pass or fail accordingly. For this example, the test expectation is composed of the error/warning message (<span
style="font: 11.0px Monaco; background-color: #c5f9e0;">Name should start with a lowercase</span>) and the text that would be
underlined by the red curly line in the editor (<span style="font: 11.0px Monaco; background-color: #c5f9e0;">Property1</span>). For
this validation test, the <span style="font: 11.0px Monaco; background-color: #c5f9e0; color: #931a68">XPECT</span> statement collects
all errors or warning occurring in the next line. A feature you might find very useful, is, that the <span
style="font: 11.0px Monaco; background-color: #c5f9e0; color: #931a68">XPECT</span> statement <i>consumes</i> error or warning
markers: An expected error or warning will <i>not</i> be shown as an error marker in the Eclipse editor.
</p>
<h2>Synchronize Expectations and Implementation</h2>
<p>The fact that Xpect uses textual expectations and embeds them into DSL documents opens the door for another awesome (IMHO) feature:
Using the Eclipse comparison editor to inspect failed tests and to fix out-dated test expectations:</p>
<p align="center"><a target="_blank" href="images/xpect_test_failure.gif"><img class="img-responsive" src="images/xpect_test_failure.gif" /></a></p>
<p>When one or more tests fail and you want to fix them, it is crucial to quickly get an overview over all failed tests. With Xpect
you can not only see all failed test cases from one file in a single comparison editor, but there are also no assert statements which
sometimes prevent execution of follow-up assert statements and thereby hide valuable hints on why the test failed. The comparison
editor, as the name suggests, also lets you edit the test file.</p>
<p>
With this approach you can still start test-first in the spirit of test-driven-development. However, later, when your implementation is
actually running, a second phase begins: Your implementation offers suggestion on how the test expectation should look like. It is your
job <i>review</i> this suggestion very carefully, and then, maybe, accept it as the better expectation.
</p>
<h2>Reusable Test Library</h2>
<p>
In Xtext projects, there are several scenarios where it is reasonable to have test coverage. The validation test I explained earlier in
this article is just one of these scenarios. Xpect ships the Java-part for such tests <a target="_blank"
href="https://github.com/meysholdt/Xpect/tree/master/org.xpect.xtext.lib/src/org/xpect/xtext/lib/tests">as a library</a>. There is
also an <a target="_blank"
href="https://github.com/meysholdt/Xpect/tree/master/org.eclipse.xtext.example.domainmodel.xpect.tests/src/org/eclipse/xtext/example/domainmodel/xpect/tests">example
project</a> that demonstrates their usage.
</p>
<p>There are test for:</p>
<ul>
<li>The parser and Abstract Syntax Tree (AST) structure (demo only, no library).</li>
<li>Code generators implemented via Xtext's IGenerator interface.</li>
<li>Validation: Test for absence, presence, message and location of errors and warnings.</li>
<li>Linking: Verify a cross reference resolved to the intended model element</li>
<li>Scoping: Verify the expected names are included or excluded from a cross references's scope.</li>
<li>ResourceDescriptions: Verify a document exports the intended model elements with proper names.</li>
<li>JvmModelInferrer: For languages using Xbase, test the inferred JVM model</li>
</ul>
<p>There will be more tests in future versions of Xpect.</p>
<h2>Support for Standalone and Workspace Tests</h2>
<p>UI-independent parts of Xtext, such as the parser, can operate standalone (i.e. without OSGi and Eclipse Workspace). The same is
true for Xpect. For capabilities where Xtext does not require OSGi or an Eclipse Workspace, Xpect does not do so either. Consequently,
Xpect tests can be executed as plain JUnit test or Plug-In JUnit tests.</p>
<p>
Since both scenarios require different kinds of setups, both can be configured separately in the <span
style="font: 11.0px Monaco; background-color: #c5f9e0; color: #931a68">XPECT_SETUP</span> section. When executed as plain JUnit test,
the ResourceSet-configuration is used and for Plug-In JUnit tests, the Workspace-Configuration is used.
</p>
<p align="center"><a target="_blank" href="images/xpect_setup.png"><img class="img-responsive" src="images/xpect_setup_sm.png" /></a></p>
<p>The screenshot also illustrates how an additional file can be loaded so that it is included in the current ResourceSet or Workspace
during test execution.</p>
<h2>Suites: Combine Tests in the Same File</h2>
<p>When explaining a specific concept of a language, it is helpful to look at the language concept from all sides: "This is the valid
syntax", "these scenarios are disallowed", "this is how cross references resolve", "this is how it is executed", etc. So far, testers
were required to create a new test file for every technical aspect. With Xpect test suites, however, it is possible to combine the
Java-parts of various tests into a single test suite.</p>
<p align="center"><a href="images/xpect_suite.png"><img class="img-responsive" src="images/xpect_suite_sm.png" /></a></p>
<p>The test suite <i>XtextTests</i> combines several tests so that in <i>CombiningMultipleTests.dmodel.xt</i> test methods from all these
tests can be used. This allows to group tests by language concept.
</p>
<h2>Works with CI Builds</h2>
<p>Xpect tests have proven to run fine with Gradle, Maven, Tycho, Surefire, and Jenkins. No special integration is necessary since
Xpect tests run as JUnit tests.</p>
<h2>Source & Download</h2>
<p> You can find a link to the update site at <a target="_blank" href="http://www.xpect-tests.org/">www.xpect-tests.org</a>. The source code and an issue tracker are on the <a target="_blank" href="https://github.com/meysholdt/Xpect">GitHub project page</a>.
</p>
<div class="bottomitem">
<h3>About the Author</h3>
<div class="row">
<div class="col-sm-12">
<div class="row">
<div class="col-sm-8">
<img class="img-responsive"
src="/community/eclipse_newsletter/2017/august/images/moritz.jpeg"
alt="Moritz Eysholdt" />
</div>
<div class="col-sm-16">
<p class="author-name">
Moritz Eysholdt<br />
<a target="_blank" href="http://typefox.io/">TypeFox</a>
</p>
<ul class="author-link list-inline">
<li><a class="btn btn-small btn-warning" target="_blank" href="https://twitter.com/docfx">Twitter</a></li>
<li><a class="btn btn-small btn-warning" target="_blank" href="https://github.com/meysholdt">GitHub</a></li>
<?php // echo $og; ?>
</ul>
</div>
</div>
</div>
</div>
</div>