std/jule/token
Index
Variables
fn IsUnaryOp(id: int): bool
fn IsBinOp(id: int): bool
fn IsWeakOp(id: int): bool
fn IsStr(k: str): bool
fn IsRawStr(k: str): bool
fn IsRune(k: str): bool
fn IsNil(k: str): bool
fn IsBool(k: str): bool
fn IsFloat(k: str): bool
fn IsNum(k: str): bool
fn IsLit(k: str): bool
fn IsPunct(r: rune): bool
fn IsSpace(r: rune): bool
fn IsLetter(r: rune): bool
fn IsNameRune(s: str): bool
fn IsKeyword(s: str): bool
fn IsDecimal(r: rune): bool
fn IsBinary(r: rune): bool
fn IsOctal(r: rune): bool
fn IsHex(r: rune): bool
fn IsAssign(id: int): bool
fn IsPostfixOp(id: int): bool
fn IsAssignOp(id: int): bool
fn Lex(mut f: &Fileset, mode: int): []log::Log
struct Token
fn Prec(self): byte
struct Fileset
static fn New(path: str): &Fileset
fn Fill(mut self, data: []byte)
unsafe fn FillMut(mut self, mut data: []byte)
fn Dir(self): str
fn Name(self): str
fn Addr(self): uintptr
fn GetRow(self, row: int): str
Variables
static Puncts: [...]rune = [ ... ]
Punctuations.
static Spaces: [...]rune = [ ... ]
Space characters.
static UnaryOps: [...]int = [ ... ]
Kind list of unary operators.
static BinOps: [...]int = [ ... ]
Kind list of binary operators.
static WeakOps: [...]int = [ ... ]
Kind list of weak operators. These operators are weak, can used as part of expression.
static PostfixOps: [...]int = [ ... ]
List of postfix operators.
static AssignOps: [...]int = [ ... ]
List of assign operators.
const (
Illegal = iota
Name
Ret
Semicolon
Lit
Comma
Const
Type
Colon
For
Break
Cont
In
If
Else
Comment
Use
Dot
Goto
DblColon
Enum
Struct
Co
Match
Self
Trait
Impl
Chan
Cpp
Fall
Fn
Let
Unsafe
Mut
Defer
Static
Hash
Error
Map
ColonEq
TripleDot
PlusEq
MinusEq
StarEq
SolidusEq
PercentEq
ShlEq
ShrEq
CaretEq
AmperEq
VlineEq
DblEq
NotEq
GtEq
LtEq
DblAmper
DblVline
Shl
Shr
DblPlus
DblMinus
Plus
Minus
Star
Solidus
Percent
Amper
Vline
Caret
Excl
Lt
Gt
Eq
LBrace
RBrace
LParent
RParent
LBracket
RBracket
RArrow
Select
)
Token identities.
const (
Standard = 1 << iota // Standard mode.
Comments // Standard mode + comments.
)
Lexer mode.
IsUnaryOp
fn IsUnaryOp(id: int): bool
Reports whether kind is unary operator.
IsBinOp
fn IsBinOp(id: int): bool
Reports whether kind is binary operator.
IsWeakOp
fn IsWeakOp(id: int): bool
Reports whether kind is weak operator.
IsStr
fn IsStr(k: str): bool
Reports whether kind is string literal.
IsRawStr
fn IsRawStr(k: str): bool
Reports whether kind is raw string literal.
IsRune
fn IsRune(k: str): bool
Reports whether kind is rune literal. Literal value can be byte or rune.
IsNil
fn IsNil(k: str): bool
Reports whether kind is nil literal.
IsBool
fn IsBool(k: str): bool
Reports whether kind is boolean literal.
IsFloat
fn IsFloat(k: str): bool
Reports whether kind is float.
IsNum
fn IsNum(k: str): bool
Reports whether kind is numeric.
IsLit
fn IsLit(k: str): bool
Reports whether kind is literal.
IsPunct
fn IsPunct(r: rune): bool
Reports whether rune is punctuation.
IsSpace
fn IsSpace(r: rune): bool
Reports whether rune is whitespace.
IsLetter
fn IsLetter(r: rune): bool
Reports whether rune is letter.
IsNameRune
fn IsNameRune(s: str): bool
Reports whether first rune of string is allowed to first rune for identifier.
IsKeyword
fn IsKeyword(s: str): bool
Reports whether s is keyword.
IsDecimal
fn IsDecimal(r: rune): bool
Reports whether rune is decimal sequence.
IsBinary
fn IsBinary(r: rune): bool
Reports whether rune is binary sequence.
IsOctal
fn IsOctal(r: rune): bool
Reports whether rune is octal sequence.
IsHex
fn IsHex(r: rune): bool
Reports whether rune is hexadecimal sequence.
IsAssign
fn IsAssign(id: int): bool
Reports given token id is allow for assignment left-expression or not.
IsPostfixOp
fn IsPostfixOp(id: int): bool
Reports whether operator kind is postfix operator.
IsAssignOp
fn IsAssignOp(id: int): bool
Reports whether operator kind is assignment operator.
Lex
fn Lex(mut f: &Fileset, mode: int): []log::Log
Lex source code into fileset. Returns nil if f == nil. Returns nil slice for errors if no any error.
Token
struct Token {
File: &Fileset
Row: int
Column: int
Kind: str
Id: int
}
Token is lexer token.
Prec
fn Prec(self): byte
Returns operator precedence of token. Returns 0 if token is not operator or invalid operator for operator precedence.
Accepts assignment tokens (like equals [=]) as precedenced operator to handle expression assignments.
Fileset
struct Fileset {
Path: str
Tokens: []&Token
// NOTE: contains filtered hidden or unexported fields
}
Fileset for lexing.
New
static fn New(path: str): &Fileset
Returns new Fileset with path.
Fill
fn Fill(mut self, data: []byte)
Fills data. Not uses mutable copy of data, allocates new copy.
FillMut
unsafe fn FillMut(mut self, mut data: []byte)
Fills data. Uses mutable copy of data, not allocated new copy. But it is unsafe, because any mutation on the data may cause inconsistent results. However, it is efficient way to use already allocated data.
Dir
fn Dir(self): str
Returns directory of file's path.
Name
fn Name(self): str
Returns filename.
Addr
fn Addr(self): uintptr
Returns self as uintptr.
GetRow
fn GetRow(self, row: int): str
Returns line (not include new-line char) by row. Returns empty string if line is not buffer.