Mercurial > repos > deepakjadmin > mayatool3_test2
comparison docs/modules/html/Lexer.html @ 0:4816e4a8ae95 draft default tip
Uploaded
| author | deepakjadmin |
|---|---|
| date | Wed, 20 Jan 2016 09:23:18 -0500 |
| parents | |
| children |
comparison
equal
deleted
inserted
replaced
| -1:000000000000 | 0:4816e4a8ae95 |
|---|---|
| 1 <html> | |
| 2 <head> | |
| 3 <title>MayaChemTools:Documentation:Parsers::Lexer.pm</title> | |
| 4 <meta http-equiv="content-type" content="text/html;charset=utf-8"> | |
| 5 <link rel="stylesheet" type="text/css" href="../../css/MayaChemTools.css"> | |
| 6 </head> | |
| 7 <body leftmargin="20" rightmargin="20" topmargin="10" bottommargin="10"> | |
| 8 <br/> | |
| 9 <center> | |
| 10 <a href="http://www.mayachemtools.org" title="MayaChemTools Home"><img src="../../images/MayaChemToolsLogo.gif" border="0" alt="MayaChemTools"></a> | |
| 11 </center> | |
| 12 <br/> | |
| 13 <div class="DocNav"> | |
| 14 <table width="100%" border=0 cellpadding=0 cellspacing=2> | |
| 15 <tr align="left" valign="top"><td width="33%" align="left"><a href="./PackageInfo.html" title="PackageInfo.html">Previous</a> <a href="./index.html" title="Table of Contents">TOC</a> <a href="./SimpleCalcParser.html" title="SimpleCalcParser.html">Next</a></td><td width="34%" align="middle"><strong>Parsers::Lexer.pm</strong></td><td width="33%" align="right"><a href="././code/Lexer.html" title="View source code">Code</a> | <a href="./../pdf/Lexer.pdf" title="PDF US Letter Size">PDF</a> | <a href="./../pdfgreen/Lexer.pdf" title="PDF US Letter Size with narrow margins: www.changethemargins.com">PDFGreen</a> | <a href="./../pdfa4/Lexer.pdf" title="PDF A4 Size">PDFA4</a> | <a href="./../pdfa4green/Lexer.pdf" title="PDF A4 Size with narrow margins: www.changethemargins.com">PDFA4Green</a></td></tr> | |
| 16 </table> | |
| 17 </div> | |
| 18 <p> | |
| 19 </p> | |
| 20 <h2>NAME</h2> | |
| 21 <p>Parsers::Lexer</p> | |
| 22 <p> | |
| 23 </p> | |
| 24 <h2>SYNOPSIS</h2> | |
| 25 <p>use Parsers::Lexer;</p> | |
| 26 <p>use Parsers::Lexer qw(:all);</p> | |
| 27 <p> | |
| 28 </p> | |
| 29 <h2>DESCRIPTION</h2> | |
| 30 <p><strong>Lexer</strong> class provides the following methods:</p> | |
| 31 <p> <a href="#new">new</a>, <a href="#getlex">GetLex</a>, <a href="#lex">Lex</a>, <a href="#next">Next</a>, <a href="#peek">Peek</a>, <a href="#stringifylexer">StringifyLexer</a> | |
| 32 </p><p>The object oriented chained <strong>Lexer</strong> is implemented based on examples available in | |
| 33 Higher-order Perl [ Ref 126 ] book by Mark J. Dominus. It is designed to be used | |
| 34 both in standalone mode or as a base class for <strong>YYLexer</strong>.</p> | |
| 35 <p>A chained lexer is created by generating a lexer for for the first specified token | |
| 36 specification using specified input and chaining it with other lexers generated for all | |
| 37 subsequent token specifications. The lexer generated for the first token specification | |
| 38 uses input iterator to retrieve any available input text; the subsequent chained lexeres | |
| 39 for rest of the token specifications use lexers generated for previous token | |
| 40 specifications to get next input, which might be unmatched input text or a reference | |
| 41 to an array containing token and matched text pair.</p> | |
| 42 <p> | |
| 43 </p> | |
| 44 <h2>METHODS</h2> | |
| 45 <dl> | |
| 46 <dt><strong><a name="new" class="item"><strong>new</strong></a></strong></dt> | |
| 47 <dd> | |
| 48 <div class="OptionsBox"> | |
| 49 $Lexer = new Parsers::Lexer($Input, @TokensSpec);</div> | |
| 50 <p>Using specified <em>Input</em> and <em>TokensSpec</em>, <strong>new</strong> method generates a new lexer | |
| 51 and returns a reference to newly created <strong>Lexer</strong> object.</p> | |
| 52 <p>Example:</p> | |
| 53 <div class="OptionsBox"> | |
| 54 # Tokens specifications supplied by the caller. It's an array containing references | |
| 55 <br/> # to arrays with each containing TokenLabel and TokenMatchRegex pair along with | |
| 56 <br/> # an option reference to code to be executed after a matched. | |
| 57 <br/> # | |
| 58 <br/> @LexerTokensSpec = ( | |
| 59 <br/> [ 'LETTER', qr/[a-zA-Z]/ ], | |
| 60 [ 'NUMBER', qr/\d+/ ], | |
| 61 [ 'SPACE', qr/[ ]*/, | |
| 62 <br/> sub { my($This, $TokenLabel, $MatchedText) = @_; return ''; } | |
| 63 <br/> ], | |
| 64 [ 'NEWLINE', qr/(?:\r\n|\r|\n)/, | |
| 65 <br/> sub { my($This, $TokenLabel, $MatchedText) = @_; return "\n"; } | |
| 66 <br/> ], | |
| 67 [ 'CHAR', qr/./ ] | |
| 68 <br/> );</div> | |
| 69 <div class="OptionsBox"> | |
| 70 # Input string... | |
| 71 <br/> $InputText = 'y = 3 + 4'; | |
| 72 <br/> $Lexer = new Parsers::Lexer($InputText, @LexerTokensSpec);</div> | |
| 73 <div class="OptionsBox"> | |
| 74 # Process input stream... | |
| 75 <br/> while (defined($Token = $Lexer->Lex())) { | |
| 76 <br/> print "Token: " . ((ref $Token) ? "@{$Token}" : "$Token") . "\n"; | |
| 77 <br/> }</div> | |
| 78 <div class="OptionsBox"> | |
| 79 # Input file... | |
| 80 <br/> $InputFile = "Input.txt"; | |
| 81 <br/> open INPUTFILE, "$InputFile" or die "Couldn't open $InputFile: $!\n"; | |
| 82 <br/> $Lexer = new Parsers::Lexer(\*INPUTFILE, @LexerTokensSpec);</div> | |
| 83 <div class="OptionsBox"> | |
| 84 # Input file iterator... | |
| 85 <br/> $InputFile = "TestSimpleCalcParser.txt"; | |
| 86 <br/> open INPUTFILE, "$InputFile" or die "Couldn't open $InputFile: $!\n"; | |
| 87 <br/> $InputIterator = sub { return <INPUTFILE>; }; | |
| 88 <br/> $Lexer = new Parsers::Lexer($InputIterator, @LexerTokensSpec);</div> | |
| 89 <div class="OptionsBox"> | |
| 90 @LexerTokensSpec = ( | |
| 91 <br/> [ 'VAR', qr/[[:alpha:]]+/ ], | |
| 92 [ 'NUM', qr/\d+/ ], | |
| 93 [ 'OP', qr/[-+=\/]/, | |
| 94 <br/> sub { my($This, $Label, $Value) = @_; | |
| 95 <br/> $Value .= "; ord: " . ord $Value; | |
| 96 <br/> return [$Label, $Value]; | |
| 97 <br/> } | |
| 98 <br/> ], | |
| 99 [ 'NEWLINE', qr/(?:\r\n|\r|\n)/, sub { return [$_[1], 'NewLine']; } ], | |
| 100 [ 'SPACE', qr/\s*/, sub { return [$_[1], 'Space']; } ], | |
| 101 <br/> );</div> | |
| 102 <div class="OptionsBox"> | |
| 103 # Look ahead without removing... | |
| 104 <br/> $Token = $Lexer->Lex('Peek'); | |
| 105 <br/> if (defined $Token && ref $Token) { | |
| 106 <br/> print "PEEK: Token: @{$Token}\n\n"; | |
| 107 <br/> }</div> | |
| 108 <div class="OptionsBox"> | |
| 109 # Process input stream... | |
| 110 <br/> while (defined($Token = $Lexer->Lex())) { | |
| 111 <br/> print "Token: " . ((ref $Token) ? "@{$Token}" : "$Token") . "\n"; | |
| 112 <br/> }</div> | |
| 113 </dd> | |
| 114 <dt><strong><a name="getlex" class="item"><strong>GetLex</strong></a></strong></dt> | |
| 115 <dd> | |
| 116 <div class="OptionsBox"> | |
| 117 $LexerRef = $Lexer->GetLex();</div> | |
| 118 <p>Returns a refernece to <em>Lexer</em> method to the caller for use in a specific <strong>YYLexer</strong>.</p> | |
| 119 </dd> | |
| 120 <dt><strong><a name="lex" class="item"><strong>Lex</strong></a></strong></dt> | |
| 121 <dd> | |
| 122 <div class="OptionsBox"> | |
| 123 $TokenRefOrText = $Lexer->Lex($Mode); | |
| 124 <br/> if (ref $TokenRefOrText) { | |
| 125 <br/> ($TokenLabel, $TokenValue) = @{$TokenRefOrText}; | |
| 126 <br/> } | |
| 127 <br/> else { | |
| 128 <br/> $TokenText = $TokenRefOrText; | |
| 129 <br/> }</div> | |
| 130 <p>Get next available token label and value pair as an array reference or unrecognized | |
| 131 text from input stream by either removing it from the input or simply peeking ahead | |
| 132 and without removing it from the input stream.</p> | |
| 133 <p>Possible <em>Mode</em> values: <em>Peek, Next</em>. Default: <em>Next</em>.</p> | |
| 134 </dd> | |
| 135 <dt><strong><a name="next" class="item"><strong>Next</strong></a></strong></dt> | |
| 136 <dd> | |
| 137 <div class="OptionsBox"> | |
| 138 $TokenRefOrText = $Lexer->Next();</div> | |
| 139 <p>Get next available token label and value pair as an array reference or unrecognized | |
| 140 text from input stream by removing it from the input stream.</p> | |
| 141 </dd> | |
| 142 <dt><strong><a name="peek" class="item"><strong>Peek</strong></a></strong></dt> | |
| 143 <dd> | |
| 144 <div class="OptionsBox"> | |
| 145 $TokenRefOrText = $Lexer->Peek();</div> | |
| 146 <p>Get next available token label and value pair as an array reference or unrecognized | |
| 147 text from input stream by by simply peeking ahead and without removing it from the | |
| 148 input stream.</p> | |
| 149 </dd> | |
| 150 <dt><strong><a name="stringifylexer" class="item"><strong>StringifyLexer</strong></a></strong></dt> | |
| 151 <dd> | |
| 152 <div class="OptionsBox"> | |
| 153 $LexerString = $Lexer->StringifyLexer();</div> | |
| 154 <p>Returns a string containing information about <em>Lexer</em> object.</p> | |
| 155 </dd> | |
| 156 </dl> | |
| 157 <p> | |
| 158 </p> | |
| 159 <h2>AUTHOR</h2> | |
| 160 <p><a href="mailto:msud@san.rr.com">Manish Sud</a></p> | |
| 161 <p> | |
| 162 </p> | |
| 163 <h2>SEE ALSO</h2> | |
| 164 <p><a href="./YYLexer.html">YYLexer.pm</a>, <a href="./SimpleCalcYYLexer.html">SimpleCalcYYLexer.pm</a>, <a href="./SimpleCalcParser.html">SimpleCalcParser.yy</a> | |
| 165 </p> | |
| 166 <p> | |
| 167 </p> | |
| 168 <h2>COPYRIGHT</h2> | |
| 169 <p>Copyright (C) 2015 Manish Sud. All rights reserved.</p> | |
| 170 <p>This file is part of MayaChemTools.</p> | |
| 171 <p>MayaChemTools is free software; you can redistribute it and/or modify it under | |
| 172 the terms of the GNU Lesser General Public License as published by the Free | |
| 173 Software Foundation; either version 3 of the License, or (at your option) | |
| 174 any later version.</p> | |
| 175 <p> </p><p> </p><div class="DocNav"> | |
| 176 <table width="100%" border=0 cellpadding=0 cellspacing=2> | |
| 177 <tr align="left" valign="top"><td width="33%" align="left"><a href="./PackageInfo.html" title="PackageInfo.html">Previous</a> <a href="./index.html" title="Table of Contents">TOC</a> <a href="./SimpleCalcParser.html" title="SimpleCalcParser.html">Next</a></td><td width="34%" align="middle"><strong>March 29, 2015</strong></td><td width="33%" align="right"><strong>Parsers::Lexer.pm</strong></td></tr> | |
| 178 </table> | |
| 179 </div> | |
| 180 <br /> | |
| 181 <center> | |
| 182 <img src="../../images/h2o2.png"> | |
| 183 </center> | |
| 184 </body> | |
| 185 </html> |