eclipse_newsletter/2014/august/content/en_article3.php - gerrit/www.eclipse.org/community - Git at Google

 <?php
 /*******************************************************************************
  * Copyright (c) 2015 Eclipse Foundation and others.
  * All rights reserved. This program and the accompanying materials
  * are made available under the terms of the Eclipse Public License v1.0
  * which accompanies this distribution, and is available at
  * http://eclipse.org/legal/epl-v10.html
  *
  * Contributors:
  *    Eric Poirier (Eclipse Foundation) - Initial implementation
  *******************************************************************************/
 ?>

 <h1 class="article-title"><?php echo $pageTitle; ?></h1>

     <p>
       Xtext languages should be built, tested and deployed continuously,
       just like any other piece of software. Users expect generators to
       run inside popular build tools and not just inside the IDE. In
       this post you will learn how to create a <a target="_blank"
         href="http://maven.apache.org/">Maven</a> build for your Xtext
       language and how to consume the resulting Maven artifacts from
       other projects, both Maven and <a target="_blank"
         href="http://www.gradle.org/">Gradle</a>.
     </p>

     <p>
       The example language that we will be building is the <a
         target="_blank"
         href="https://www.eclipse.org/Xtext/7languagesDoc.html#builddsl">BuildDSL</a>
       from the Seven Languages project. The language itself (i.e. the
       grammar and tooling) are not important for this exercise. However,
       it has several interesting properties that allow me to show you
       most of the challenges you will encounter when building a
       language:
     </p>
     <ul>
       <li>It uses Xbase, linking and generating Java code</li>
       <li>It has a standard library that needs to be shipped with the
         language</li>
       <li>It has UI tests which need some additional setup</li>
     </ul>
     <p>
       The code for the examples is available on <a target="_blank"
         href="https://github.com/oehme/xtext-maven-example">Github</a>.
     </p>

     <h2>Choose your path</h2>

     <p>There are two ways of "mavenizing" your language build. The first
       is the minimalistic approach, which relies on checking in
       generated code. This means that this Maven build is not able to
       generate the language infrastructure or clean the output folders
       by itself. It is just there to create Maven artifacts for others
       to consume. This approach is very non-intrusive for developers who
       are used to an Eclipse build. There are two main disadvantages to
       this. Checking in generated sources will make your diffs pretty
       hard to read. And if one of your developers has a slightly
       different setup than everyone else (line endings, encoding, etc.),
       the build might run on his machine, but fail when he checks in.</p>

     <p>The alternative is a Maven build that can generate everything
       from just the source code. This means you can stop checking in
       generated code. And since all important configuration is part of
       the build script, you will get the same result on all platforms.
       But it also means that you will have to run the Maven build once
       after checkout in order for your projects to compile. In the end
       this is a matter of preference and you and your team should decide
       which suits you better.</p>

     <h2>The starting point</h2>
     <ul>
       <li>Let's fist have a look at the unaltered language projects. You
         will find them in the <i>language/0-initial</i> folder. I will
         give you just a quick rundown what each of them does. If you
         have worked with Xtext before, you should feel right at home.
       </li>
       <li><i>builddsl</i> contains the core language infrastructure and
         code generator</li>
       <li><i>builddsl.lib</i> is the standard library that is shipped
         with the language</li>
       <li><i>builddsl.ui</i> contains the Eclipse UI parts (like content
         assist)</li>
       <li><i>builddsl.tests</i> contains all tests for the language,
         including UI tests</li>
       <li><i>builddsl.target</i> is a target platform definition for the
         language. If you don't use one, then your language will be built
         against whatever is installed in your host workbench. This makes
         builds highly platform dependent. So every project should have a
         target platform.</li>
       <li><i>builddls.repository</i> assembles the plugins into a p2
         repository, from which the plugins can be installed</li>
     </ul>

     <h2>The minimalist approach</h2>

     <p>
       Next up we will create the minimal required POMs to build the
       language with Maven. You can find these projects under <i>language/1-simple</i>.
       The most important addition is the <i>builddsl.parent</i> project.
       This is the main entrypoint for Maven. It lists all other projects
       which should be built and contains common configuration for them.
       The crucial part is enabling the Tycho Maven plugin.
     </p>

     <div>
       <script
         src="https://gist.github.com/oehme/9bc23b6f192aeac5f6f1.js"></script>
       <span style="font-family: Arial;"><span
         style="font-size: 15px; line-height: 17.25px; white-space: pre-wrap;"><br /></span></span>
     </div>

     <p>Tycho is a Maven extension that can read dependency information
       from Manifest files and fetch these dependencies from p2
       repositories. It also teaches Maven which build steps are required
       for Eclipse plugins. The second important bit of configuration is
       the target platform definition. We just reference our target
       project and list the environments for which we want to build.
       Without specifying these, Tycho would just use the currently
       running system as the default, which would make the build platform
       dependent.</p>

     <div>
       <span
         style="font-family: Arial; font-size: 15px; line-height: 17.25px; white-space: pre-wrap;"><br /></span>
       <script
         src="https://gist.github.com/oehme/d1ee67076c3613c29a43.js"></script>
     </div>
     <br />
     <p>The POMs for the individual projects are all very
       straightforward. Each inherits from the parent and sepcifies a
       packaging type. Tycho adds packaging types for Eclipse plugins,
       target platforms and repositories. One of the more interesting
       POMs is that of the core language plugin. It needs a dependency to
       the library plugin, even though that dependency is already present
       in the Manifest.</p>

     <div>
       <span style="font-family: Arial;"><span
         style="font-size: 15px; line-height: 17.25px; white-space: pre-wrap;"><br /></span></span>
       <script
         src="https://gist.github.com/oehme/dbdddcecd7e28e3b0cee.js"></script>
     </div>


     <p>This is the downside of using Tycho. If you want Maven clients to
       understand your dependencies, they always need to be in the POM
       But Tycho does not "enhance" the POM with the dependency
       information from the Manifest. Luckily, this is the only bit of
       duplication we need. All Xtext runtime dependencies are already on
       Maven Central and part of the Xtext plugins for Maven and Gradle,
       which we will use later.</p>

     <p>The other more elaborate POM is that of the tests plugin. You
       will see that we enable the UI harness so that we can run the
       content assist tests contained in the project. Sadly, we need to
       add some platform specific handling for OSX, as otherwise SWT will
       not run. This is the price you pay if you want your build to run
       well on all platforms.</p>

     <p>
       Try running <i>mvn install</i> from the parent folder to install
       the language into your local Maven repository. Deploying the
       artifacts to some other repository like Maven Central is out of
       scope for this post, but well documented and no different from any
       other Maven project. Also, you will find a p2 repository in the <i>/target</i>
       folder of the <i>builddsl.repository</i> project. You could
       transfer this to some public server so that other people can add
       your language to their Eclipse installation.
     </p>


     <h2>Using your language</h2>

     <p>You are now ready to use your language from the client projects.
       There are two example clients. Each of them contains the same
       code: A builddsl file which references an Xtend file. This is just
       to show that referencing existing Java code works as desired. As a
       bonus, both projects also showcase how to compile Xtend files.
       Note that both clients lack any Eclipse configuration, because all
       the information is present in the build scripts. If you want to
       look at them inside Eclipse, you need to use their respective
       Eclipse integration.</p>

     <p>
       For Maven users, there is the xtext-maven-plugin. You just add
       your core language artifact as a dependency to the plugin and add
       the language configuration. This configuration contains the name
       of the StandaloneSetup class and the output folders. If your
       project has a large classpath, you can also filter which jars to
       search for model files, greatly speeding up your build. As of now,
       you manually have to add the output directory to the list of Java
       source folders in order for the generated code to be picked up by
       the Java compiler. This is done using the build-helper plugin. Try
       running the build with <i>mvn clean verify</i>. If you want to see
       the project in Eclipse. use the "Import existing Maven Project"
       wizard.
     </p>

     <div>
       <span style="font-family: Arial;"><span
         style="font-size: 15px; line-height: 17.25px; white-space: pre-wrap;"><br /></span></span>
       <script
         src="https://gist.github.com/oehme/36377efb3d43d0556fa9.js"></script>
       <span style="font-family: Arial;"><span
         style="font-size: 15px; line-height: 17.25px; white-space: pre-wrap;"><br /></span></span>
     </div>

     <p>
       For Gradle users, there is the even more convenient
       xtext-gradle-plugin. Similar to Maven, you add your core language
       artifact to the xtextTooling dependency configuration. Then you
       configure the StandaloneSetup and output folders. You can directly
       specify that an output folder produces Java code, which will
       automatically make it a source folder. Try running the build with
       <i>gradlew clean build</i>. If you want to see the project in
       Eclipse. run <i>gradle eclipse</i> on the command line and then
       use the normal "Import existing Project" wizard.
     </p>


     <div>
       <span style="font-family: Arial;"><span
         style="font-size: 15px; line-height: 17.25px; white-space: pre-wrap;"><br /></span></span>
       <script
         src="https://gist.github.com/oehme/1156982157729406a9ce.js"></script>
     </div>

     <h2>The fully mavenized way</h2>

     <p>
       Now for the final part of our journey: Building the language
       infrastructure from scratch using Maven. You can find the projects
       under <i>language/2-full</i>. In the parent POM you can see that
       we have added common configuration for Xtend. This is because part
       of the language infrastructure and tests are implemented using
       Xtend. Subprojects now only need to add a two-line configuration
       to use Xtend and inherit everything else from the parent. The
       other big change happened in the core language plugin. In order to
       run the mwe2 workflow to generate the language infrastructure, we
       use the exec-maven plugin.
     </p>


     <span
       style="font-family: Arial; font-size: 15px; line-height: 17.25px; white-space: pre-wrap;"><br /></span>
     <script src="https://gist.github.com/oehme/019186d7b6f2ac1e720c.js"></script>

     <p>
       We add <i>org.eclipse.xtext.xtext</i> as a dependency, which
       contains the grammar language and generator including all
       dependencies like mwe2. We also depend on Xbase because our
       language inherits from it. Then we just point the Mwe2Launcher at
       our workflow file. For the proper cleanup, we configure the
       clean-plugin to throw away everything that is generated by the
       workflow. By default the clean-plugin only cleans the /target
       folder. Last but not least there is a one-line change to the
       workflow itself. The grammarUri cannot be a classpath Uri, since
       at the time the workflow is run in Maven, the grammar file is not
       on the classpath. Instead we switch to a platform resource Uri.
     </p>

     <span
       style="font-family: Arial; font-size: 15px; line-height: 17.25px; white-space: pre-wrap;"><br /></span>
     <script src="https://gist.github.com/oehme/67b6d1773d3c0896f2df.js"></script>

     <p>Now try running mvn clean to see how everything but the
       handwritten source gets deleted and then mvn install to build the
       language from scratch. You could now .gitignore the generated
       code.</p>

     <h2>Conclusion</h2>

     <p>
       I hope this guide has helped you set up your prefered build style.
       If you have any further questions, just ask them in the <a
         href="https://www.eclipse.org/forums/index.php?t=thread&frm_id=27">Xtext
         forum</a>. There are lots of helpful people who will probably
       have an answer =)
     </p>

 <div class="bottomitem">
   <h3>About the Authors</h3>

   <div class="row">
     <div class="col-sm-12">
       <div class="row">
         <div class="col-sm-8">
           <img class="author-picture"
         src="/community/eclipse_newsletter/2014/august/images/stefan1.jpg"
         width="75" alt="Stefan Oehme" />
         </div>
         <div class="col-sm-16">
           <p class="author-name">
         Stefan Oehme<br />
         <a target="_blank" href="http://www.itemis.com/">itemis</a>
       </p>
       <ul class="author-link">
         <li><a target="_blank" href="http://mnmlst-dvlpr.blogspot.de/">Blog</a></li>
         <!-- <li><a target="_blank" href="http://twitter.com/zxiiro">Twitter</a></li> -->
         <li><a target="_blank"
           href="https://plus.google.com/+StefanOehme/posts">Google +</a></li>
         <?php echo $og; ?>
       </ul>
         </div>
       </div>
     </div>
   </div>
 </div>
	<?php
	/*******************************************************************************
	* Copyright (c) 2015 Eclipse Foundation and others.
	* All rights reserved. This program and the accompanying materials
	* are made available under the terms of the Eclipse Public License v1.0
	* which accompanies this distribution, and is available at
	* http://eclipse.org/legal/epl-v10.html
	*
	* Contributors:
	* Eric Poirier (Eclipse Foundation) - Initial implementation
	*******************************************************************************/
	?>

	<h1 class="article-title"><?php echo $pageTitle; ?></h1>

	<p>
	Xtext languages should be built, tested and deployed continuously,
	just like any other piece of software. Users expect generators to
	run inside popular build tools and not just inside the IDE. In
	this post you will learn how to create a <a target="_blank"
	href="http://maven.apache.org/">Maven</a> build for your Xtext
	language and how to consume the resulting Maven artifacts from
	other projects, both Maven and <a target="_blank"
	href="http://www.gradle.org/">Gradle</a>.
	</p>

	<p>
	The example language that we will be building is the <a
	target="_blank"
	href="https://www.eclipse.org/Xtext/7languagesDoc.html#builddsl">BuildDSL</a>
	from the Seven Languages project. The language itself (i.e. the
	grammar and tooling) are not important for this exercise. However,
	it has several interesting properties that allow me to show you
	most of the challenges you will encounter when building a
	language:
	</p>
	<ul>
	<li>It uses Xbase, linking and generating Java code</li>
	<li>It has a standard library that needs to be shipped with the
	language</li>
	<li>It has UI tests which need some additional setup</li>
	</ul>
	<p>
	The code for the examples is available on <a target="_blank"
	href="https://github.com/oehme/xtext-maven-example">Github</a>.
	</p>

	<h2>Choose your path</h2>

	<p>There are two ways of "mavenizing" your language build. The first
	is the minimalistic approach, which relies on checking in
	generated code. This means that this Maven build is not able to
	generate the language infrastructure or clean the output folders
	by itself. It is just there to create Maven artifacts for others
	to consume. This approach is very non-intrusive for developers who
	are used to an Eclipse build. There are two main disadvantages to
	this. Checking in generated sources will make your diffs pretty
	hard to read. And if one of your developers has a slightly
	different setup than everyone else (line endings, encoding, etc.),
	the build might run on his machine, but fail when he checks in.</p>

	<p>The alternative is a Maven build that can generate everything
	from just the source code. This means you can stop checking in
	generated code. And since all important configuration is part of
	the build script, you will get the same result on all platforms.
	But it also means that you will have to run the Maven build once
	after checkout in order for your projects to compile. In the end
	this is a matter of preference and you and your team should decide
	which suits you better.</p>

	<h2>The starting point</h2>
	<ul>
	<li>Let's fist have a look at the unaltered language projects. You
	will find them in the <i>language/0-initial</i> folder. I will
	give you just a quick rundown what each of them does. If you
	have worked with Xtext before, you should feel right at home.
	</li>
	<li><i>builddsl</i> contains the core language infrastructure and
	code generator</li>
	<li><i>builddsl.lib</i> is the standard library that is shipped
	with the language</li>
	<li><i>builddsl.ui</i> contains the Eclipse UI parts (like content
	assist)</li>
	<li><i>builddsl.tests</i> contains all tests for the language,
	including UI tests</li>
	<li><i>builddsl.target</i> is a target platform definition for the
	language. If you don't use one, then your language will be built
	against whatever is installed in your host workbench. This makes
	builds highly platform dependent. So every project should have a
	target platform.</li>
	<li><i>builddls.repository</i> assembles the plugins into a p2
	repository, from which the plugins can be installed</li>
	</ul>

	<h2>The minimalist approach</h2>

	<p>
	Next up we will create the minimal required POMs to build the
	language with Maven. You can find these projects under <i>language/1-simple</i>.
	The most important addition is the <i>builddsl.parent</i> project.
	This is the main entrypoint for Maven. It lists all other projects
	which should be built and contains common configuration for them.
	The crucial part is enabling the Tycho Maven plugin.
	</p>

	<div>
	<script
	src="https://gist.github.com/oehme/9bc23b6f192aeac5f6f1.js"></script>
	<span style="font-family: Arial;"><span
	style="font-size: 15px; line-height: 17.25px; white-space: pre-wrap;"><br /></span></span>
	</div>

	<p>Tycho is a Maven extension that can read dependency information
	from Manifest files and fetch these dependencies from p2
	repositories. It also teaches Maven which build steps are required
	for Eclipse plugins. The second important bit of configuration is
	the target platform definition. We just reference our target
	project and list the environments for which we want to build.
	Without specifying these, Tycho would just use the currently
	running system as the default, which would make the build platform
	dependent.</p>

	<div>
	<span
	style="font-family: Arial; font-size: 15px; line-height: 17.25px; white-space: pre-wrap;"><br /></span>
	<script
	src="https://gist.github.com/oehme/d1ee67076c3613c29a43.js"></script>
	</div>
	<br />
	<p>The POMs for the individual projects are all very
	straightforward. Each inherits from the parent and sepcifies a
	packaging type. Tycho adds packaging types for Eclipse plugins,
	target platforms and repositories. One of the more interesting
	POMs is that of the core language plugin. It needs a dependency to
	the library plugin, even though that dependency is already present
	in the Manifest.</p>

	<div>
	<span style="font-family: Arial;"><span
	style="font-size: 15px; line-height: 17.25px; white-space: pre-wrap;"><br /></span></span>
	<script
	src="https://gist.github.com/oehme/dbdddcecd7e28e3b0cee.js"></script>
	</div>


	<p>This is the downside of using Tycho. If you want Maven clients to
	understand your dependencies, they always need to be in the POM
	But Tycho does not "enhance" the POM with the dependency
	information from the Manifest. Luckily, this is the only bit of
	duplication we need. All Xtext runtime dependencies are already on
	Maven Central and part of the Xtext plugins for Maven and Gradle,
	which we will use later.</p>

	<p>The other more elaborate POM is that of the tests plugin. You
	will see that we enable the UI harness so that we can run the
	content assist tests contained in the project. Sadly, we need to
	add some platform specific handling for OSX, as otherwise SWT will
	not run. This is the price you pay if you want your build to run
	well on all platforms.</p>

	<p>
	Try running <i>mvn install</i> from the parent folder to install
	the language into your local Maven repository. Deploying the
	artifacts to some other repository like Maven Central is out of
	scope for this post, but well documented and no different from any
	other Maven project. Also, you will find a p2 repository in the <i>/target</i>
	folder of the <i>builddsl.repository</i> project. You could
	transfer this to some public server so that other people can add
	your language to their Eclipse installation.
	</p>


	<h2>Using your language</h2>

	<p>You are now ready to use your language from the client projects.
	There are two example clients. Each of them contains the same
	code: A builddsl file which references an Xtend file. This is just
	to show that referencing existing Java code works as desired. As a
	bonus, both projects also showcase how to compile Xtend files.
	Note that both clients lack any Eclipse configuration, because all
	the information is present in the build scripts. If you want to
	look at them inside Eclipse, you need to use their respective
	Eclipse integration.</p>

	<p>
	For Maven users, there is the xtext-maven-plugin. You just add
	your core language artifact as a dependency to the plugin and add
	the language configuration. This configuration contains the name
	of the StandaloneSetup class and the output folders. If your
	project has a large classpath, you can also filter which jars to
	search for model files, greatly speeding up your build. As of now,
	you manually have to add the output directory to the list of Java
	source folders in order for the generated code to be picked up by
	the Java compiler. This is done using the build-helper plugin. Try
	running the build with <i>mvn clean verify</i>. If you want to see
	the project in Eclipse. use the "Import existing Maven Project"
	wizard.
	</p>

	<div>
	<span style="font-family: Arial;"><span
	style="font-size: 15px; line-height: 17.25px; white-space: pre-wrap;"><br /></span></span>
	<script
	src="https://gist.github.com/oehme/36377efb3d43d0556fa9.js"></script>
	<span style="font-family: Arial;"><span
	style="font-size: 15px; line-height: 17.25px; white-space: pre-wrap;"><br /></span></span>
	</div>

	<p>
	For Gradle users, there is the even more convenient
	xtext-gradle-plugin. Similar to Maven, you add your core language
	artifact to the xtextTooling dependency configuration. Then you
	configure the StandaloneSetup and output folders. You can directly
	specify that an output folder produces Java code, which will
	automatically make it a source folder. Try running the build with
	<i>gradlew clean build</i>. If you want to see the project in
	Eclipse. run <i>gradle eclipse</i> on the command line and then
	use the normal "Import existing Project" wizard.
	</p>


	<div>
	<span style="font-family: Arial;"><span
	style="font-size: 15px; line-height: 17.25px; white-space: pre-wrap;"><br /></span></span>
	<script
	src="https://gist.github.com/oehme/1156982157729406a9ce.js"></script>
	</div>

	<h2>The fully mavenized way</h2>

	<p>
	Now for the final part of our journey: Building the language
	infrastructure from scratch using Maven. You can find the projects
	under <i>language/2-full</i>. In the parent POM you can see that
	we have added common configuration for Xtend. This is because part
	of the language infrastructure and tests are implemented using
	Xtend. Subprojects now only need to add a two-line configuration
	to use Xtend and inherit everything else from the parent. The
	other big change happened in the core language plugin. In order to
	run the mwe2 workflow to generate the language infrastructure, we
	use the exec-maven plugin.
	</p>


	<span
	style="font-family: Arial; font-size: 15px; line-height: 17.25px; white-space: pre-wrap;"><br /></span>
	<script src="https://gist.github.com/oehme/019186d7b6f2ac1e720c.js"></script>

	<p>
	We add <i>org.eclipse.xtext.xtext</i> as a dependency, which
	contains the grammar language and generator including all
	dependencies like mwe2. We also depend on Xbase because our
	language inherits from it. Then we just point the Mwe2Launcher at
	our workflow file. For the proper cleanup, we configure the
	clean-plugin to throw away everything that is generated by the
	workflow. By default the clean-plugin only cleans the /target
	folder. Last but not least there is a one-line change to the
	workflow itself. The grammarUri cannot be a classpath Uri, since
	at the time the workflow is run in Maven, the grammar file is not
	on the classpath. Instead we switch to a platform resource Uri.
	</p>

	<span
	style="font-family: Arial; font-size: 15px; line-height: 17.25px; white-space: pre-wrap;"><br /></span>
	<script src="https://gist.github.com/oehme/67b6d1773d3c0896f2df.js"></script>

	<p>Now try running mvn clean to see how everything but the
	handwritten source gets deleted and then mvn install to build the
	language from scratch. You could now .gitignore the generated
	code.</p>

	<h2>Conclusion</h2>

	<p>
	I hope this guide has helped you set up your prefered build style.
	If you have any further questions, just ask them in the <a
	href="https://www.eclipse.org/forums/index.php?t=thread&frm_id=27">Xtext
	forum</a>. There are lots of helpful people who will probably
	have an answer =)
	</p>

	<div class="bottomitem">
	<h3>About the Authors</h3>

	<div class="row">
	<div class="col-sm-12">
	<div class="row">
	<div class="col-sm-8">
	<img class="author-picture"
	src="/community/eclipse_newsletter/2014/august/images/stefan1.jpg"
	width="75" alt="Stefan Oehme" />
	</div>
	<div class="col-sm-16">
	<p class="author-name">
	Stefan Oehme<br />
	<a target="_blank" href="http://www.itemis.com/">itemis</a>
	</p>
	<ul class="author-link">
	<li><a target="_blank" href="http://mnmlst-dvlpr.blogspot.de/">Blog</a></li>
	<!-- <li><a target="_blank" href="http://twitter.com/zxiiro">Twitter</a></li> -->
	<li><a target="_blank"
	href="https://plus.google.com/+StefanOehme/posts">Google +</a></li>
	<?php echo $og; ?>
	</ul>
	</div>
	</div>
	</div>
	</div>
	</div>