記錄讀英文文檔的過程 主要爲了自己方便複習回顧 同時希望能幫到你看懂文檔到底在說啥任務到底要怎麼做 如果有錯誤/模糊的地方歡迎指出
每次作業完成一個編譯器的一個部分：詞法分析、語法分析、語義分析、代碼生成
PA1 寫一個詞法分析器 利用cpp的flex工具或者java的jlex工具把Cool的tokens轉換成cpp或java代碼
Flex compiles your rule file (e.g., “lexer.l”) to C (or, if you are using JLex, Java) source code implementing a finite automaton recognizing the regular expressions that you specify in your rule file.
flex的框架
{definitions}
%%
{rules}
%%
{user subroutines}
definitions部分和user subroutines部分可以沒有 第二個%%可以沒有 第一個必須有
%%
是一個合法的最小的lex程序 從輸入到輸出什麼都沒做
這個作業最主要的任務就是寫rules rules裏面是一個字符串表 左邊是正則表達式 右邊是actions, program fragments to be executed when the expressions are recognized.
下面介紹正則表達式裏的一些符號
\ - ^這三個字符在正則表達式裏的作用特殊
-表示範圍 如果要表示-這個字符的話必須放在最前面或者最後面
//If it is desired to include the character - in a character class, it should be first or last;
^必須放在最前面 表示後面這個表達式的補集
The operator ? indicates an optional element of an expression. Thus
ab?c
matches either ac or abc.
$只能放在最後 並且只能在一行的末尾進行匹配
/可以表示後面的上下文。
<x>可以表示這個規則是以條件x爲開始的
{x}x如果是一個變量名 那麼就會從前面找到定義好的x替代掉現在這個x//此處不確定理解是否正確
x如果是數字 就表示重複的次數
//a{1,5}looks for 1 to 5 occurrences of a.
下面介紹與表達式對應的動作裏的一些操作
lex會把字符數組存到yytext這個變量裏
[a-z]+ printf("%s", yytext); <<==>> [a-z]+ ECHO;
這個字符數組的長度是yyleng
提供兩個方法來確定yytext的長度
yymore()把下一個輸入串接在當前這串的後面 而不是將其替代
Example: Consider a language which defines a string as a set of characters between quotation (") marks, and provides that to include a " in a string it must be preceded by a \. The regular expression which matches that is somewhat confusing, so that it might be preferable to write
\"[^"]* {
if (yytext[yyleng-1] == '\\')
yymore();
else
... normal user processing
}
yyless (n)把當前這串的末尾n位返回給輸入串 和下次的輸入串一起處理
1) input() which returns the next input character;
2) output(c) which writes the character c on the output; and
3) unput(c) pushes the character c back onto the input stream to be read later by input().
yywrap() 可以重寫 一般默認在遇到EOF的時候return 1 但是在跨越源文件的情況下可以讓其return 0 讓解析繼續進行
規則的匹配原則：
1. 最長匹配
2. 一樣長久匹配先出現的
每個字符都只能匹配一個規則 如果要多次匹配的話可以使用REJECT 即完成了當前的action之後將這個輸入串與下一個可匹配的規則進行匹配
she {s++; REJECT;}
he {h++; REJECT;}
\n |
. ;
比如這個規則可以統計所有地方出現的"he"這個字符串的次數
讀完了lex的文檔接下來讀cool_manual的section10 Lexical Structure
10.1 Integers, Identifiers, and Special Notation
Integers are non-empty strings of digits 0-9.
Identifiers are strings (other than keywords) consisting of letters, digits, and the underscore character.
Type identifiers begin with a capital letter; 類名由大寫字母開頭
object identifiers begin with a lower case letter. 對象名由小寫字母開頭
There are two other identifiers, self and SELF TYPE that are treated specially by Cool but are not treated as keywords.這倆是特殊的標識符但是不是關鍵字
10.2 Strings
字符串由雙引號包圍 字符串裏的\c表示字符c 除了
\b backspace \t tab \n newline \f formfeed這幾個特例
可以換行 但是要在行尾加\
可以沒有EOF和\0 但是不能跨文件
10.3 Comments
單行註釋-- 包起來的註釋(*...*) 後者可以嵌套 不能跨文件
10.4 Keywords
The keywords of cool are: class, else, false, fi, if, in, inherits, isvoid, let, loop, pool, then, while, case, esac, new, of, not, true. Except for the constants true and false, keywords are case insensitive. To conform to the rules for other objects, the first letter of true and false must be lowercase; the trailing letters may be upper or lower case.
true和false的首字母是小寫的 其他的關鍵字以及true和false之後的字符都是大小寫不敏感的
10.5 White Space
White space consists of any sequence of the characters: blank (ascii 32), \n (newline, ascii 10), \f (form feed, ascii 12), \r (carriage return, ascii 13), \t (tab, ascii 9), \v (vertical tab, ascii 11).
BEGIN的用法
To handle the same problem with start conditions, each start condition must be introduced to Lex in the definitions section with a line reading
%Start name1 name2 ...
where the conditions may be named in any order. The word Start may be abbreviated to s or S. The conditions may be referenced at the head of a rule with the <> brackets:
<name1>expression
is a rule which is only recognized when Lex is in the start condition name1. To enter a start condition, execute the action statement
BEGIN name1;
which changes the start condition to name1. To resume the normal state,
BEGIN 0;
resets the initial condition of the Lex automaton interpreter. A rule may be active in several start conditions: <name1,name2,name3> is a legal prefix. Any rule not beginning with the <> prefix operator is always active.
There is also a special default state called INITIAL which is active unless you explicitly indicate the beginning of a new state.
下面是抄作業的過程
首先從最簡單的單行註釋開始
狀態開頭就是單行註釋
<INITIAL>"--" {BEGIN INLINE_CONMENTS;}
上面的單行註釋後面跟了除了newline之外的字符
<INLINE_COMENTS>[^\n]* {}
單行註釋遇上了newline
<INLINE_COMENTS>\n {
curr_lineno++;
BEGIN 0;
}
接着寫多行註釋的規則
/* Nested comments */
開始符號之後跟着一個註釋開始 那麼層數++ 並且往下尋找可以匹配的規則
<INITIAL,COMMENTS,INLINE_COMMENTS>"(*" {
comment_layer++;
BEGIN COMMENTS;
}
不是這三個字符的情況都不用處理
<COMMENTS>[^\n(*]* { }
如果遇到這三個字符也不同處理 最長匹配的機制可以保證當"(*"或者"*)"出現的時候一定是走comment_layer變化的規則
<COMMENTS>[(*] { }
註釋層數--
<COMMENTS>"*)" {
comment_layer--;
if (comment_layer == 0) {
BEGIN 0;
}
}
註釋跨文件了
<COMMENTS><<EOF>> {
yylval.error_msg = "EOF in comment";
BEGIN 0;
return ERROR;
}
不匹配的結束符
"*)" {
yylval.error_msg = "Unmatched *)";
return ERROR;
}
接着寫keyword
?i:表示大小寫不敏感 其他關鍵字類似 不表
/* NOT */
(?i:not) { return NOT; }
常量
/* INT_CONST */
{DIGIT}+ {
cool_yylval.symbol = inttable.add_string(yytext);
return INT_CONST;
}
true和false首字母小寫 後面的字母大小寫不敏感
/* BOOL_CONST */
t(?i:rue) {
cool_yylval.boolean = 1;
return BOOL_CONST;
}
f(?i:alse) {
cool_yylval.boolean = 0;
return BOOL_CONST;
}
無視代碼中國的空白
/* White Space */
[ \f\r\t\v]+ { }
類名
/* TYPEID */
[A-Z][A-Za-z0-9_]* {
cool_yylval.symbol = idtable.add_string(yytext);
return TYPEID;
}
更新行號
/* To treat lines. */
"\n" {
curr_lineno++;
}
對象名
/* OBJECTID */
[a-z][A-Za-z0-9_]* {
cool_yylval.symbol = idtable.add_string(yytext);
return OBJECTID;
}
運算符
/* ASSIGN */
"<-" { return ASSIGN; }
/* LE */
"<=" { return LE; }
/* DARROW */
"=>" { return DARROW; }
"+" { return int('+'); }
最後大頭 字符串
看到單個雙引號了就開始了String的內容
<INITIAL>(\") {
BEGIN STRING;
yymore();
}
字符串的內容裏面不能出現這三個字符
/* Cannot read '\\' '\"' '\n' */
<STRING>[^\\\"\n]* { yymore(); }
如果一定要出現單個\的話就匹配這一條
/* normal escape characters, not \n */
<STRING>\\[^\n] { yymore(); }
如果既要\又要換行
/* seen a '\\' at the end of a line, the string continues */
<STRING>\\\n {
curr_lineno++;
yymore();
}
這裏注意yyrestart方法 把掃描文件的指針重新放到文件開頭
/* meet EOF in the middle of a string, error */
<STRING><<EOF>> {
yylval.error_msg = "EOF in string constant";
BEGIN 0;
yyrestart(yyin);
return ERROR;
}
/* meet a '\n' in the middle of a string without a '\\', error */
<STRING>\n {
yylval.error_msg = "Unterminated string constant";
BEGIN 0;
curr_lineno++;
return ERROR;
}
/* meet a "\\0" ??? */
<STRING>\\0 {
yylval.error_msg = "Unterminated string constant";
BEGIN 0;
//curr_lineno++;
return ERROR;
}
字符串結尾
/* string ends, we need to deal with some escape characters */
<STRING>\" {
std::string input(yytext, yyleng);
// remove the '\"'s on both sizes.
input = input.substr(1, input.length() - 2);
std::string output = "";
std::string::size_type pos;
std::string::npos是一個常數，它等於size_type類型可以表示的最大值，用來表示一個不存在的位置,類型一般是std::container_type::size_type。
if (input.find_first_of('\0') != std::string::npos) {
yylval.error_msg = "String contains null character";
BEGIN 0;
return ERROR;
}
while ((pos = input.find_first_of("\\")) != std::string::npos) {
output += input.substr(0, pos);
switch (input[pos + 1]) {
case 'b':
output += "\b";
break;
case 't':
output += "\t";
break;
case 'n':
output += "\n";
break;
case 'f':
output += "\f";
break;
default:
output += input[pos + 1];
break;
}
input = input.substr(pos + 2, input.length() - 2);
}
output += input;
if (output.length() > 1024) {
yylval.error_msg = "String constant too long";
BEGIN 0;
return ERROR;
}
cool_yylval.symbol = stringtable.add_string((char*)output.c_str());
BEGIN 0;
return STR_CONST;
}