From: Dirk Rossow <rossow@foni.net>
Date: 04/19/2005 01:00 PM

Subject: FixedFormParser


Hi Jeff,

i have worked on the fixed form lexer and sending you my first results. It
looks very promising that far. The details:

FixedFormLexerPrepass preprocesses the input stream. It discards all
whitespaces and comment lines and concatenates continuation lines.
Additional it holds a mapping to the character-positions in the file (for
correct start/end line/col in the Token objects).

The modified InputStream is feed to FixedFormLexerPhase1, which is a
modified copy of your FreeFormLexerPhase1. The mods are:

Handling of hollerith constants (This is missing in your
FreeFormLexerPhase1)
A new state which is active at the start of line. Needed this for
hollerith-handling
A new state for IDENT-matching. It only gets active if no keyword matches. I
added this because there are no whitespaces delimiting the tokens and for
example callsubname needs to get matched as T_CALL T_IDENT. So the lexer
tries first to match a token ignoring IDENTs. If there is no match, an IDENT
is matched and returned.


FixedFormLexerPhase1 feeds FixedFormLexerPhase2, which uses your
FreeFormLexerPhase2.
After your phase2lexer has processed the tokens, it checks if IDENT-Tokens
need to get concatenated. This can happen if for example:
callsubname=0
T_CALL T_IDENT T_EQUAL T_ICON
gets converted by your phase2lexer to
T_IDENT T_IDENT T_EQUAL T_ICON

Finally i modified your FreeFormLexerPhase2 and the generated Parser to
support different Lexers. I simply added an Interface Ilexer.

I did not commit anything to the repository, i thought you first want to
have a look.
There are still some bugs in the FreeFormLexerPhase2. You can find examples
for them in the fixedTestFiles/regressionTestFiles folder. They are
slashesInFormat.f, statusEqAfterIf.f, typeSpecAsIdent.f and impliedLoop.f.

I added some of our fortran-code for testing my work, please dont add it to
the repository.

I hope my explanations are understandable and you like what i did :-) Dont
hesitate to contact me if you have questions.

Have a nice day
  Dirk