core/howto/generate parser/generateParser.html - gerrit/www.eclipse.org/jdt - Git at Google

 <!doctype html public "-//w3c//dtd html 4.0 transitional//en">
 <html>
 	<head>
 		<title>How to: Generate the Parser</title>
 		<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
 		<link rel="stylesheet" href="http://dev.eclipse.org/default_style.css" type="text/css">
 	</head>
 	<body bgcolor="#FFFFFF" text="#000000">

 	<div style="background: yellow; border: solid 2px orange">&#x26A0;&#xFE0F; The content of this document is partly outdated. Up-to-date information is available at <a href="https://wiki.eclipse.org/JDT_Core_Programmer_Guide/ECJ/Parse">https://wiki.eclipse.org/JDT_Core_Programmer_Guide/ECJ/Parse</a></div>

 	<table border=0 cellspacing=5 cellpadding=2 width="100%" >

 <tr>
   <td align=LEFT valign=TOP colspan="2" bgcolor="#0080C0"><b><font color="#FFFFFF">
    JDT Core / HowTo: Generate the Parser </font></b></td>
 </tr>

 <tr>
 	<td ALIGN=RIGHT VALIGN=TOP WIDTH="2%"><img SRC="http://dev.eclipse.org/images/Adarrow.gif" NOSAVE BORDER=0 height=16 width=16></td>
 	<td WIDTH="98%"><b>Where to get the parser generator</b><br>
 	<blockquote>The parser files and resources are automatically generated using the Jikespg parser generator. This tools has
 	been renamed Jikes Parser generator. You can find more information and latest releases at this <A HREF="http://sourceforge.net/project/showfiles.php?group_id=128803&package_id=144579">link</A>.
 	The latest tool is provided in source format. We don't provide any help for compiling these
 	source files. Refer to the link above if you have trouble to get binaries.
     <br>Our grammar is generated using the version 1.3 of <a href="jikespg.exe">Jikespg</a>. If newer versions fail to generate resources from our
 	grammar, please send request to the Jikes Parser Generator team.
 	</blockquote>
 	</td>
 </tr>
 <tr>
 	<td ALIGN=RIGHT VALIGN=TOP WIDTH="2%"><img SRC="http://dev.eclipse.org/images/Adarrow.gif" NOSAVE BORDER=0 height=16 width=16></td>
 	<td WIDTH="98%"><b>Where to get the grammar</b><br>
 	<blockquote>The latest grammar is always located in the <code>java.g</code> file.
 	(N.B.: Older versions of the grammar, such as java_1_4.g and java_1_5.g are not supported and are no longer functional.). Go to the org.eclipse.jdt.core plugins
 	directory (<font color="#3366FF">eclipse\plugins\org.eclipse.jdt.core</font> where eclipse is the root of your eclipse installation) and open the
 	<font color="#3366FF">grammar</font> directory. Then search for the <font color="#3366FF">java.g</font> file inside the
 	folder <font color="#3366FF">grammar</font>. Then copy its contents from:
 <PRE>--main options
 %options ACTION, AN=JavaAction.java, GP=java,
 ....
 $end
 -- need a carriage return after the $end
 </PRE>
 	into a file called <font color="#3366FF">java.g</font>. It is important to add a carriage return at the end of the last line.
 	You can save this file where you want, we will assume from thereon you saved it in <code>d:\temp\</code>.
 	If the end-of-line of the environment of the C compiler used to build jikespg is different from the one used when running the executable, you might end up with an exception at runtime.
 	Be sure to used consistent end-of-line characters.
 	</blockquote>
 	</td>
 </tr>
 <tr>
 	<td ALIGN=RIGHT VALIGN=TOP WIDTH="2%"><img SRC="http://dev.eclipse.org/images/Adarrow.gif" NOSAVE BORDER=0 height=16 width=16></td>
 	<td WIDTH="98%"><b>What to do with these files and update the parser class...</b><br>
 	<blockquote>
 Assuming the Jikespg executable (<code>jikespg.exe</code>) is located in <code>d:\jikespg</code>.
  <ol>
  <li>First in a console, run:
  <PRE>
  	d:
  	cd \temp
  	d:\jikespg\jikespg.exe java.g
  </PRE>
  </li>
  <li>You will get an output that looks like this:
  <PRE>
  IBM Research Jikes Parser Generator            Fri Oct 14 09:22:32 2005
 %OPTIONS ACTION, AN=JavaAction.java, GP=java,
 %OPTIONS FILE-PREFIX=java, ESCAPE=$, PREFIX=TokenName, OUTPUT-SIZE=125 ,
 %OPTIONS NOGOTO-DEFAULT, SINGLE-PRODUCTIONS, LALR=1 , TABLE,
 %OPTIONS ERROR_MAPS
 %OPTIONS first follow
 %OPTIONS TRACE=FULL ,
 %OPTIONS VERBOSE
 %OPTIONS DEFERRED
 %OPTIONS NAMES=MAX
 %OPTIONS SCOPES
 Options in effect:
     ACTION  ACTFILE-NAME=JavaAction.java  BLOCKB=/.  BLOCKE=./  BYTE  CONFLI
     DEFAULT=5  NODEBUG  DEFERRED  NOEDIT  ERROR-MAPS  ESCAPE=$
     FILE-PREFIX=java  FIRST  FOLLOW  GENERATE-PARSER=JAVA  NOGOTO-DEFAULT
     HACTFILE-NAME=javahdr.java  HBLOCKB=/:  HBLOCKE=:/  LALR=1  LIST
     MAX-DISTANCE=30  MIN-DISTANCE=3  NAMES=MAXIMUM  NONT-CHECK  ORMARK=|
     OUTPUT-SIZE=125  PREFIX=TokenName  READ-REDUCE  SCOPES  NOSHIFT-DEFAULT
     SINGLE-PRODUCTIONS  STACK-SIZE=128  STATES  SUFFIX=  TABLE=SPACE
     TRACE=FULL  VERBOSE  WARNINGS  XREF

 *** The following Terminals are useless:
         const  goto

 This grammar is LALR(1).

 Number of Terminals: 110
 Number of Nonterminals: 306
 Number of Productions: 693
 Number of Single Productions: 232
 Number of Items: 2165
 Number of Scopes: 132
 Number of States: 954
 Number of Shift actions: 6057
 Number of Goto actions: 7493
 Number of Shift/Reduce actions: 593
 Number of Goto/Reduce actions: 1041
 Number of Reduce actions: 13922
 Number of Shift-Reduce conflicts: 0
 Number of Reduce-Reduce conflicts: 0
 Reallocating storage for SPACE table, adding 2820 entries

 Length of base Action Table: 12268
 Number of entries in base Action Table: 9488
 Percentage of increase: 29.3%
 Storage required for base Tables: 24536 Bytes, 24K
 Storage required for Rules: 2076 Bytes
 Number of unique terminal states: 727
 Number of Shift actions saved by merging: 4113
 Number of Reduce actions saved by merging: 967
 Number of Reduce saved by default: 8997

 Length of Terminal Check Table: 7913
 Length of Terminal Action Table: 7902
 Number of entries in Terminal Action Table: 7222
 Percentage of increase: 9.4%
 Storage required for Terminal Tables: 23717 Bytes, 24K
 Total storage required for Tables: 48253 Bytes, 48K


 Actions in Compressed Tables:
      Number of Shifts: 2211
      Number of Shift/Reduces: 326
      Number of Gotos: 7493
      Number of Goto/Reduces: 1041
      Number of Reduces: 3958
      Number of Defaults: 475

 Error maps storage:
     Storage required for ACTION_SYMBOLS_BASE map: 1908 Bytes
     Storage required for ACTION_SYMBOLS_RANGE map: 1709 Bytes
     Storage required for NACTION_SYMBOLS_BASE map: 1908 Bytes
     Storage required for NACTION_SYMBOLS_RANGE map: 960 Bytes
     Storage required for TERMINAL_INDEX map: 220 Bytes
     Storage required for NON_TERMINAL_INDEX map: 614 Bytes
     Storage required for STRING_BUFFER map: 12004 Bytes

 ***Warning: Base Check vector contains value > 127. 16-bit words used.
 Escaped symbol $eof is an invalid C variable.

 Escaped symbol $error is an invalid C variable.
  </PRE>
  It can be quite different if the output changed since version 1.3 of jikespg. The important part is:<br>
  <blockquote><b>This grammar is LALR(1).</b></blockquote>
  This creates some java source files and information files in the current directory.
  <blockquote><table BORDER=1 CELLSPACING=2 CELLPADDING=10>
  <tr>
  <th VALIGN=top align=left>java.l</th>
  <td VALIGN=top>Information generated by jikespg. Enumarate all the states created for the automaton, etc.</td>
  </tr>
  <tr>
  <th VALIGN=top align=left>JavaAction.java</th>
  <td>Contains the method consumeRule(int) of class org.eclipse.jdt.internal.compiler.parser.Parser, which handles all semantic action dispatches.</td>
  </tr>
  <tr>
 <th VALIGN=top align=left>javahdr.java</th>
  <td>Used to generate the resources files.</td>
  </tr>
  <tr>
 <th VALIGN=top align=left>javadcl.java</th>
  <td>Used to generate the resources files.</td>
  </tr>
  <tr>
 <th VALIGN=top align=left>javasym.java</th>
  <td>Contents of the class org.eclipse.jdt.core.compiler.TerminalTokens.</td>
  </tr>
  <tr>
 <th VALIGN=top align=left>javadef.java</th>
  <td>Contents of the class org.eclipse.jdt.internal.compiler.parser.ParserBasicInformation.</td>
  </tr>
  <tr>
 <th VALIGN=top align=left>javaprs.java</th>
  <td>You don't need this file. Its contents is already inlined in the Parser class.</td>
  </tr>
  </table></blockquote>
  </li>
  <li><blockquote>Now we need to update the different classes and resource files.
  </blockquote>
  <ol>
  <li>Copy the contents of the <font color="#3366FF">JavaAction.java</font> file into the <font color="#3366FF">consumeRule(int)</font> method of the org.eclipse.jdt.internal.compiler.parser.Parser class.
  </li>
  <li>The class <font color="#3366FF">org.eclipse.jdt.internal.compiler.parser.ParserBasicInformation</font> needs to be updated with the content of the file <font color="#3366FF">javadef.java</font>. Don't copy the
  interface name. Simply copy the field declarations. The actual source of this class will guide you.</li>
  <li>Similarly, update the class <font color="#3366FF">org.eclipse.jdt.internal.compiler.parser.TerminalTokens</font> with the contents of the file <font color="#3366FF">javasym.java</font>.</li>
  <li>The last step is to update the resource files:<br>
  Copy the jdtcore.jar file in d:\temp. Compile this <A HREF="UpdateParserFiles.java">source</A> inside d:\temp. You will have a file UpdateParserFiles.class.
  Then run the following command-line:
   <PRE>
  D:\temp>java -classpath jdtcore.jar:. UpdateParserFiles javadcl.java javahdr.java
  </PRE>
 Once this done, you will end up with 25 new files inside d:\temp. They are called parser&lt;n&gt;.rsc, with n equals to 1..24 and readableNames.properties.
 All these files need to be moved to the org\eclipse\jdt\internal\compiler\parser folder. Now you are ready to execute and test
 the new parser.
  </li>
  </ol>
  </li>
  </ol>
  <p><b>Warning:</b> <blockquote>If the number of terminals is over 128, then the parser term_check field needs to be converted
  to a <code>char[]</code>. Also the generation of the parser resources files needs to be changed to support
  an array of <code>char[]</code>.
  </blockquote></p>
  <P>
  <b>NOTE:</b>  <blockquote>Changing the parser is a risky operation if you miss one of the steps above. The resulting parser can be completely
  unpredictable. It can go from crashing to reporting invalid errors. Be sure that you followed all the steps and that all the
  files are updated and recompiled before you run it. </blockquote>
  </P>
  </blockquote>
 </td>
 </tr>
 </table>
 </body>
 </html>
	<!doctype html public "-//w3c//dtd html 4.0 transitional//en">
	<html>
	<head>
	<title>How to: Generate the Parser</title>
	<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
	<link rel="stylesheet" href="http://dev.eclipse.org/default_style.css" type="text/css">
	</head>
	<body bgcolor="#FFFFFF" text="#000000">

	<div style="background: yellow; border: solid 2px orange">⚠️ The content of this document is partly outdated. Up-to-date information is available at <a href="https://wiki.eclipse.org/JDT_Core_Programmer_Guide/ECJ/Parse">https://wiki.eclipse.org/JDT_Core_Programmer_Guide/ECJ/Parse</a></div>

	<table border=0 cellspacing=5 cellpadding=2 width="100%" >

	<tr>
	<td align=LEFT valign=TOP colspan="2" bgcolor="#0080C0"><b><font color="#FFFFFF">
	JDT Core / HowTo: Generate the Parser </font></b></td>
	</tr>

	<tr>
	<td ALIGN=RIGHT VALIGN=TOP WIDTH="2%"><img SRC="http://dev.eclipse.org/images/Adarrow.gif" NOSAVE BORDER=0 height=16 width=16></td>
	<td WIDTH="98%"><b>Where to get the parser generator</b><br>
	<blockquote>The parser files and resources are automatically generated using the Jikespg parser generator. This tools has
	been renamed Jikes Parser generator. You can find more information and latest releases at this <A HREF="http://sourceforge.net/project/showfiles.php?group_id=128803&package_id=144579">link</A>.
	The latest tool is provided in source format. We don't provide any help for compiling these
	source files. Refer to the link above if you have trouble to get binaries.
	<br>Our grammar is generated using the version 1.3 of <a href="jikespg.exe">Jikespg</a>. If newer versions fail to generate resources from our
	grammar, please send request to the Jikes Parser Generator team.
	</blockquote>
	</td>
	</tr>
	<tr>
	<td ALIGN=RIGHT VALIGN=TOP WIDTH="2%"><img SRC="http://dev.eclipse.org/images/Adarrow.gif" NOSAVE BORDER=0 height=16 width=16></td>
	<td WIDTH="98%"><b>Where to get the grammar</b><br>
	<blockquote>The latest grammar is always located in the <code>java.g</code> file.
	(N.B.: Older versions of the grammar, such as java_1_4.g and java_1_5.g are not supported and are no longer functional.). Go to the org.eclipse.jdt.core plugins
	directory (<font color="#3366FF">eclipse\plugins\org.eclipse.jdt.core</font> where eclipse is the root of your eclipse installation) and open the
	<font color="#3366FF">grammar</font> directory. Then search for the <font color="#3366FF">java.g</font> file inside the
	folder <font color="#3366FF">grammar</font>. Then copy its contents from:
	<PRE>--main options
	%options ACTION, AN=JavaAction.java, GP=java,
	....
	$end
	-- need a carriage return after the $end
	</PRE>
	into a file called <font color="#3366FF">java.g</font>. It is important to add a carriage return at the end of the last line.
	You can save this file where you want, we will assume from thereon you saved it in <code>d:\temp\</code>.
	If the end-of-line of the environment of the C compiler used to build jikespg is different from the one used when running the executable, you might end up with an exception at runtime.
	Be sure to used consistent end-of-line characters.
	</blockquote>
	</td>
	</tr>
	<tr>
	<td ALIGN=RIGHT VALIGN=TOP WIDTH="2%"><img SRC="http://dev.eclipse.org/images/Adarrow.gif" NOSAVE BORDER=0 height=16 width=16></td>
	<td WIDTH="98%"><b>What to do with these files and update the parser class...</b><br>
	<blockquote>
	Assuming the Jikespg executable (<code>jikespg.exe</code>) is located in <code>d:\jikespg</code>.
	<ol>
	<li>First in a console, run:
	<PRE>
	d:
	cd \temp
	d:\jikespg\jikespg.exe java.g
	</PRE>
	</li>
	<li>You will get an output that looks like this:
	<PRE>
	IBM Research Jikes Parser Generator Fri Oct 14 09:22:32 2005
	%OPTIONS ACTION, AN=JavaAction.java, GP=java,
	%OPTIONS FILE-PREFIX=java, ESCAPE=$, PREFIX=TokenName, OUTPUT-SIZE=125 ,
	%OPTIONS NOGOTO-DEFAULT, SINGLE-PRODUCTIONS, LALR=1 , TABLE,
	%OPTIONS ERROR_MAPS
	%OPTIONS first follow
	%OPTIONS TRACE=FULL ,
	%OPTIONS VERBOSE
	%OPTIONS DEFERRED
	%OPTIONS NAMES=MAX
	%OPTIONS SCOPES
	Options in effect:
	ACTION ACTFILE-NAME=JavaAction.java BLOCKB=/. BLOCKE=./ BYTE CONFLI
	DEFAULT=5 NODEBUG DEFERRED NOEDIT ERROR-MAPS ESCAPE=$
	FILE-PREFIX=java FIRST FOLLOW GENERATE-PARSER=JAVA NOGOTO-DEFAULT
	HACTFILE-NAME=javahdr.java HBLOCKB=/: HBLOCKE=:/ LALR=1 LIST
	MAX-DISTANCE=30 MIN-DISTANCE=3 NAMES=MAXIMUM NONT-CHECK ORMARK=\|
	OUTPUT-SIZE=125 PREFIX=TokenName READ-REDUCE SCOPES NOSHIFT-DEFAULT
	SINGLE-PRODUCTIONS STACK-SIZE=128 STATES SUFFIX= TABLE=SPACE
	TRACE=FULL VERBOSE WARNINGS XREF

	*** The following Terminals are useless:
	const goto

	This grammar is LALR(1).

	Number of Terminals: 110
	Number of Nonterminals: 306
	Number of Productions: 693
	Number of Single Productions: 232
	Number of Items: 2165
	Number of Scopes: 132
	Number of States: 954
	Number of Shift actions: 6057
	Number of Goto actions: 7493
	Number of Shift/Reduce actions: 593
	Number of Goto/Reduce actions: 1041
	Number of Reduce actions: 13922
	Number of Shift-Reduce conflicts: 0
	Number of Reduce-Reduce conflicts: 0
	Reallocating storage for SPACE table, adding 2820 entries

	Length of base Action Table: 12268
	Number of entries in base Action Table: 9488
	Percentage of increase: 29.3%
	Storage required for base Tables: 24536 Bytes, 24K
	Storage required for Rules: 2076 Bytes
	Number of unique terminal states: 727
	Number of Shift actions saved by merging: 4113
	Number of Reduce actions saved by merging: 967
	Number of Reduce saved by default: 8997

	Length of Terminal Check Table: 7913
	Length of Terminal Action Table: 7902
	Number of entries in Terminal Action Table: 7222
	Percentage of increase: 9.4%
	Storage required for Terminal Tables: 23717 Bytes, 24K
	Total storage required for Tables: 48253 Bytes, 48K


	Actions in Compressed Tables:
	Number of Shifts: 2211
	Number of Shift/Reduces: 326
	Number of Gotos: 7493
	Number of Goto/Reduces: 1041
	Number of Reduces: 3958
	Number of Defaults: 475

	Error maps storage:
	Storage required for ACTION_SYMBOLS_BASE map: 1908 Bytes
	Storage required for ACTION_SYMBOLS_RANGE map: 1709 Bytes
	Storage required for NACTION_SYMBOLS_BASE map: 1908 Bytes
	Storage required for NACTION_SYMBOLS_RANGE map: 960 Bytes
	Storage required for TERMINAL_INDEX map: 220 Bytes
	Storage required for NON_TERMINAL_INDEX map: 614 Bytes
	Storage required for STRING_BUFFER map: 12004 Bytes

	***Warning: Base Check vector contains value > 127. 16-bit words used.
	Escaped symbol $eof is an invalid C variable.

	Escaped symbol $error is an invalid C variable.
	</PRE>
	It can be quite different if the output changed since version 1.3 of jikespg. The important part is:<br>
	<blockquote><b>This grammar is LALR(1).</b></blockquote>
	This creates some java source files and information files in the current directory.
	<blockquote><table BORDER=1 CELLSPACING=2 CELLPADDING=10>
	<tr>
	<th VALIGN=top align=left>java.l</th>
	<td VALIGN=top>Information generated by jikespg. Enumarate all the states created for the automaton, etc.</td>
	</tr>
	<tr>
	<th VALIGN=top align=left>JavaAction.java</th>
	<td>Contains the method consumeRule(int) of class org.eclipse.jdt.internal.compiler.parser.Parser, which handles all semantic action dispatches.</td>
	</tr>
	<tr>
	<th VALIGN=top align=left>javahdr.java</th>
	<td>Used to generate the resources files.</td>
	</tr>
	<tr>
	<th VALIGN=top align=left>javadcl.java</th>
	<td>Used to generate the resources files.</td>
	</tr>
	<tr>
	<th VALIGN=top align=left>javasym.java</th>
	<td>Contents of the class org.eclipse.jdt.core.compiler.TerminalTokens.</td>
	</tr>
	<tr>
	<th VALIGN=top align=left>javadef.java</th>
	<td>Contents of the class org.eclipse.jdt.internal.compiler.parser.ParserBasicInformation.</td>
	</tr>
	<tr>
	<th VALIGN=top align=left>javaprs.java</th>
	<td>You don't need this file. Its contents is already inlined in the Parser class.</td>
	</tr>
	</table></blockquote>
	</li>
	<li><blockquote>Now we need to update the different classes and resource files.
	</blockquote>
	<ol>
	<li>Copy the contents of the <font color="#3366FF">JavaAction.java</font> file into the <font color="#3366FF">consumeRule(int)</font> method of the org.eclipse.jdt.internal.compiler.parser.Parser class.
	</li>
	<li>The class <font color="#3366FF">org.eclipse.jdt.internal.compiler.parser.ParserBasicInformation</font> needs to be updated with the content of the file <font color="#3366FF">javadef.java</font>. Don't copy the
	interface name. Simply copy the field declarations. The actual source of this class will guide you.</li>
	<li>Similarly, update the class <font color="#3366FF">org.eclipse.jdt.internal.compiler.parser.TerminalTokens</font> with the contents of the file <font color="#3366FF">javasym.java</font>.</li>
	<li>The last step is to update the resource files:<br>
	Copy the jdtcore.jar file in d:\temp. Compile this <A HREF="UpdateParserFiles.java">source</A> inside d:\temp. You will have a file UpdateParserFiles.class.
	Then run the following command-line:
	<PRE>
	D:\temp>java -classpath jdtcore.jar:. UpdateParserFiles javadcl.java javahdr.java
	</PRE>
	Once this done, you will end up with 25 new files inside d:\temp. They are called parser<n>.rsc, with n equals to 1..24 and readableNames.properties.
	All these files need to be moved to the org\eclipse\jdt\internal\compiler\parser folder. Now you are ready to execute and test
	the new parser.
	</li>
	</ol>
	</li>
	</ol>
	<p><b>Warning:</b> <blockquote>If the number of terminals is over 128, then the parser term_check field needs to be converted
	to a <code>char[]</code>. Also the generation of the parser resources files needs to be changed to support
	an array of <code>char[]</code>.
	</blockquote></p>
	<P>
	<b>NOTE:</b> <blockquote>Changing the parser is a risky operation if you miss one of the steps above. The resulting parser can be completely
	unpredictable. It can go from crashing to reporting invalid errors. Be sure that you followed all the steps and that all the
	files are updated and recompiled before you run it. </blockquote>
	</P>
	</blockquote>
	</td>
	</tr>
	</table>
	</body>
	</html>