|
1
|
1 <html>
|
|
|
2 <head>
|
|
|
3 <title>MayaChemTools:Documentation:Parsers::Lexer.pm</title>
|
|
|
4 <meta http-equiv="content-type" content="text/html;charset=utf-8">
|
|
|
5 <link rel="stylesheet" type="text/css" href="../../css/MayaChemTools.css">
|
|
|
6 </head>
|
|
|
7 <body leftmargin="20" rightmargin="20" topmargin="10" bottommargin="10">
|
|
|
8 <br/>
|
|
|
9 <center>
|
|
|
10 <a href="http://www.mayachemtools.org" title="MayaChemTools Home"><img src="../../images/MayaChemToolsLogo.gif" border="0" alt="MayaChemTools"></a>
|
|
|
11 </center>
|
|
|
12 <br/>
|
|
|
13 <div class="DocNav">
|
|
|
14 <table width="100%" border=0 cellpadding=0 cellspacing=2>
|
|
|
15 <tr align="left" valign="top"><td width="33%" align="left"><a href="./PackageInfo.html" title="PackageInfo.html">Previous</a> <a href="./index.html" title="Table of Contents">TOC</a> <a href="./SimpleCalcParser.html" title="SimpleCalcParser.html">Next</a></td><td width="34%" align="middle"><strong>Parsers::Lexer.pm</strong></td><td width="33%" align="right"><a href="././code/Lexer.html" title="View source code">Code</a> | <a href="./../pdf/Lexer.pdf" title="PDF US Letter Size">PDF</a> | <a href="./../pdfgreen/Lexer.pdf" title="PDF US Letter Size with narrow margins: www.changethemargins.com">PDFGreen</a> | <a href="./../pdfa4/Lexer.pdf" title="PDF A4 Size">PDFA4</a> | <a href="./../pdfa4green/Lexer.pdf" title="PDF A4 Size with narrow margins: www.changethemargins.com">PDFA4Green</a></td></tr>
|
|
|
16 </table>
|
|
|
17 </div>
|
|
|
18 <p>
|
|
|
19 </p>
|
|
|
20 <h2>NAME</h2>
|
|
|
21 <p>Parsers::Lexer</p>
|
|
|
22 <p>
|
|
|
23 </p>
|
|
|
24 <h2>SYNOPSIS</h2>
|
|
|
25 <p>use Parsers::Lexer;</p>
|
|
|
26 <p>use Parsers::Lexer qw(:all);</p>
|
|
|
27 <p>
|
|
|
28 </p>
|
|
|
29 <h2>DESCRIPTION</h2>
|
|
|
30 <p><strong>Lexer</strong> class provides the following methods:</p>
|
|
|
31 <p> <a href="#new">new</a>, <a href="#getlex">GetLex</a>, <a href="#lex">Lex</a>, <a href="#next">Next</a>, <a href="#peek">Peek</a>, <a href="#stringifylexer">StringifyLexer</a>
|
|
|
32 </p><p>The object oriented chained <strong>Lexer</strong> is implemented based on examples available in
|
|
|
33 Higher-order Perl [ Ref 126 ] book by Mark J. Dominus. It is designed to be used
|
|
|
34 both in standalone mode or as a base class for <strong>YYLexer</strong>.</p>
|
|
|
35 <p>A chained lexer is created by generating a lexer for for the first specified token
|
|
|
36 specification using specified input and chaining it with other lexers generated for all
|
|
|
37 subsequent token specifications. The lexer generated for the first token specification
|
|
|
38 uses input iterator to retrieve any available input text; the subsequent chained lexeres
|
|
|
39 for rest of the token specifications use lexers generated for previous token
|
|
|
40 specifications to get next input, which might be unmatched input text or a reference
|
|
|
41 to an array containing token and matched text pair.</p>
|
|
|
42 <p>
|
|
|
43 </p>
|
|
|
44 <h2>METHODS</h2>
|
|
|
45 <dl>
|
|
|
46 <dt><strong><a name="new" class="item"><strong>new</strong></a></strong></dt>
|
|
|
47 <dd>
|
|
|
48 <div class="OptionsBox">
|
|
|
49 $Lexer = new Parsers::Lexer($Input, @TokensSpec);</div>
|
|
|
50 <p>Using specified <em>Input</em> and <em>TokensSpec</em>, <strong>new</strong> method generates a new lexer
|
|
|
51 and returns a reference to newly created <strong>Lexer</strong> object.</p>
|
|
|
52 <p>Example:</p>
|
|
|
53 <div class="OptionsBox">
|
|
|
54 # Tokens specifications supplied by the caller. It's an array containing references
|
|
|
55 <br/> # to arrays with each containing TokenLabel and TokenMatchRegex pair along with
|
|
|
56 <br/> # an option reference to code to be executed after a matched.
|
|
|
57 <br/> #
|
|
|
58 <br/> @LexerTokensSpec = (
|
|
|
59 <br/> [ 'LETTER', qr/[a-zA-Z]/ ],
|
|
|
60 [ 'NUMBER', qr/\d+/ ],
|
|
|
61 [ 'SPACE', qr/[ ]*/,
|
|
|
62 <br/> sub { my($This, $TokenLabel, $MatchedText) = @_; return ''; }
|
|
|
63 <br/> ],
|
|
|
64 [ 'NEWLINE', qr/(?:\r\n|\r|\n)/,
|
|
|
65 <br/> sub { my($This, $TokenLabel, $MatchedText) = @_; return "\n"; }
|
|
|
66 <br/> ],
|
|
|
67 [ 'CHAR', qr/./ ]
|
|
|
68 <br/> );</div>
|
|
|
69 <div class="OptionsBox">
|
|
|
70 # Input string...
|
|
|
71 <br/> $InputText = 'y = 3 + 4';
|
|
|
72 <br/> $Lexer = new Parsers::Lexer($InputText, @LexerTokensSpec);</div>
|
|
|
73 <div class="OptionsBox">
|
|
|
74 # Process input stream...
|
|
|
75 <br/> while (defined($Token = $Lexer->Lex())) {
|
|
|
76 <br/> print "Token: " . ((ref $Token) ? "@{$Token}" : "$Token") . "\n";
|
|
|
77 <br/> }</div>
|
|
|
78 <div class="OptionsBox">
|
|
|
79 # Input file...
|
|
|
80 <br/> $InputFile = "Input.txt";
|
|
|
81 <br/> open INPUTFILE, "$InputFile" or die "Couldn't open $InputFile: $!\n";
|
|
|
82 <br/> $Lexer = new Parsers::Lexer(\*INPUTFILE, @LexerTokensSpec);</div>
|
|
|
83 <div class="OptionsBox">
|
|
|
84 # Input file iterator...
|
|
|
85 <br/> $InputFile = "TestSimpleCalcParser.txt";
|
|
|
86 <br/> open INPUTFILE, "$InputFile" or die "Couldn't open $InputFile: $!\n";
|
|
|
87 <br/> $InputIterator = sub { return <INPUTFILE>; };
|
|
|
88 <br/> $Lexer = new Parsers::Lexer($InputIterator, @LexerTokensSpec);</div>
|
|
|
89 <div class="OptionsBox">
|
|
|
90 @LexerTokensSpec = (
|
|
|
91 <br/> [ 'VAR', qr/[[:alpha:]]+/ ],
|
|
|
92 [ 'NUM', qr/\d+/ ],
|
|
|
93 [ 'OP', qr/[-+=\/]/,
|
|
|
94 <br/> sub { my($This, $Label, $Value) = @_;
|
|
|
95 <br/> $Value .= "; ord: " . ord $Value;
|
|
|
96 <br/> return [$Label, $Value];
|
|
|
97 <br/> }
|
|
|
98 <br/> ],
|
|
|
99 [ 'NEWLINE', qr/(?:\r\n|\r|\n)/, sub { return [$_[1], 'NewLine']; } ],
|
|
|
100 [ 'SPACE', qr/\s*/, sub { return [$_[1], 'Space']; } ],
|
|
|
101 <br/> );</div>
|
|
|
102 <div class="OptionsBox">
|
|
|
103 # Look ahead without removing...
|
|
|
104 <br/> $Token = $Lexer->Lex('Peek');
|
|
|
105 <br/> if (defined $Token && ref $Token) {
|
|
|
106 <br/> print "PEEK: Token: @{$Token}\n\n";
|
|
|
107 <br/> }</div>
|
|
|
108 <div class="OptionsBox">
|
|
|
109 # Process input stream...
|
|
|
110 <br/> while (defined($Token = $Lexer->Lex())) {
|
|
|
111 <br/> print "Token: " . ((ref $Token) ? "@{$Token}" : "$Token") . "\n";
|
|
|
112 <br/> }</div>
|
|
|
113 </dd>
|
|
|
114 <dt><strong><a name="getlex" class="item"><strong>GetLex</strong></a></strong></dt>
|
|
|
115 <dd>
|
|
|
116 <div class="OptionsBox">
|
|
|
117 $LexerRef = $Lexer->GetLex();</div>
|
|
|
118 <p>Returns a refernece to <em>Lexer</em> method to the caller for use in a specific <strong>YYLexer</strong>.</p>
|
|
|
119 </dd>
|
|
|
120 <dt><strong><a name="lex" class="item"><strong>Lex</strong></a></strong></dt>
|
|
|
121 <dd>
|
|
|
122 <div class="OptionsBox">
|
|
|
123 $TokenRefOrText = $Lexer->Lex($Mode);
|
|
|
124 <br/> if (ref $TokenRefOrText) {
|
|
|
125 <br/> ($TokenLabel, $TokenValue) = @{$TokenRefOrText};
|
|
|
126 <br/> }
|
|
|
127 <br/> else {
|
|
|
128 <br/> $TokenText = $TokenRefOrText;
|
|
|
129 <br/> }</div>
|
|
|
130 <p>Get next available token label and value pair as an array reference or unrecognized
|
|
|
131 text from input stream by either removing it from the input or simply peeking ahead
|
|
|
132 and without removing it from the input stream.</p>
|
|
|
133 <p>Possible <em>Mode</em> values: <em>Peek, Next</em>. Default: <em>Next</em>.</p>
|
|
|
134 </dd>
|
|
|
135 <dt><strong><a name="next" class="item"><strong>Next</strong></a></strong></dt>
|
|
|
136 <dd>
|
|
|
137 <div class="OptionsBox">
|
|
|
138 $TokenRefOrText = $Lexer->Next();</div>
|
|
|
139 <p>Get next available token label and value pair as an array reference or unrecognized
|
|
|
140 text from input stream by removing it from the input stream.</p>
|
|
|
141 </dd>
|
|
|
142 <dt><strong><a name="peek" class="item"><strong>Peek</strong></a></strong></dt>
|
|
|
143 <dd>
|
|
|
144 <div class="OptionsBox">
|
|
|
145 $TokenRefOrText = $Lexer->Peek();</div>
|
|
|
146 <p>Get next available token label and value pair as an array reference or unrecognized
|
|
|
147 text from input stream by by simply peeking ahead and without removing it from the
|
|
|
148 input stream.</p>
|
|
|
149 </dd>
|
|
|
150 <dt><strong><a name="stringifylexer" class="item"><strong>StringifyLexer</strong></a></strong></dt>
|
|
|
151 <dd>
|
|
|
152 <div class="OptionsBox">
|
|
|
153 $LexerString = $Lexer->StringifyLexer();</div>
|
|
|
154 <p>Returns a string containing information about <em>Lexer</em> object.</p>
|
|
|
155 </dd>
|
|
|
156 </dl>
|
|
|
157 <p>
|
|
|
158 </p>
|
|
|
159 <h2>AUTHOR</h2>
|
|
|
160 <p><a href="mailto:msud@san.rr.com">Manish Sud</a></p>
|
|
|
161 <p>
|
|
|
162 </p>
|
|
|
163 <h2>SEE ALSO</h2>
|
|
|
164 <p><a href="./YYLexer.html">YYLexer.pm</a>, <a href="./SimpleCalcYYLexer.html">SimpleCalcYYLexer.pm</a>, <a href="./SimpleCalcParser.html">SimpleCalcParser.yy</a>
|
|
|
165 </p>
|
|
|
166 <p>
|
|
|
167 </p>
|
|
|
168 <h2>COPYRIGHT</h2>
|
|
|
169 <p>Copyright (C) 2015 Manish Sud. All rights reserved.</p>
|
|
|
170 <p>This file is part of MayaChemTools.</p>
|
|
|
171 <p>MayaChemTools is free software; you can redistribute it and/or modify it under
|
|
|
172 the terms of the GNU Lesser General Public License as published by the Free
|
|
|
173 Software Foundation; either version 3 of the License, or (at your option)
|
|
|
174 any later version.</p>
|
|
|
175 <p> </p><p> </p><div class="DocNav">
|
|
|
176 <table width="100%" border=0 cellpadding=0 cellspacing=2>
|
|
|
177 <tr align="left" valign="top"><td width="33%" align="left"><a href="./PackageInfo.html" title="PackageInfo.html">Previous</a> <a href="./index.html" title="Table of Contents">TOC</a> <a href="./SimpleCalcParser.html" title="SimpleCalcParser.html">Next</a></td><td width="34%" align="middle"><strong>March 29, 2015</strong></td><td width="33%" align="right"><strong>Parsers::Lexer.pm</strong></td></tr>
|
|
|
178 </table>
|
|
|
179 </div>
|
|
|
180 <br />
|
|
|
181 <center>
|
|
|
182 <img src="../../images/h2o2.png">
|
|
|
183 </center>
|
|
|
184 </body>
|
|
|
185 </html>
|
