Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix performance issue w.r.t. not_out_end #311

Merged
merged 5 commits into from
Oct 15, 2024
Merged

Fix performance issue w.r.t. not_out_end #311

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
7a2ae16
Fix performance issue w.r.t. not_out_end
bkiers Oct 13, 2024
6c867c0
Remove another `~TagEnd+`, remove unused `other_than_tag_start`
bkiers Oct 13, 2024
fe19822
Fix comment
bkiers Oct 13, 2024
2312fa4
Include correct tokens in `not_out_end` and `other_than_tag_end`
bkiers Oct 13, 2024
2b8b615
Update LiquidParser.g4
bkiers Oct 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions src/main/antlr4/liquid/parser/v4/LiquidLexer.g4
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,8 @@ fragment LineBreak : '\r'? '\n' | '\r';
fragment Letter : [a-zA-Z];
fragment Digit : [0-9];

// Note that when adding tokens to this `IN_TAG` mode, be sure to include them in the parser
// rules `not_out_end` and/or `other_tag_parameters` as well!
mode IN_TAG;

OutStart2 : '{{' -> pushMode(IN_TAG);
Expand Down
25 changes: 19 additions & 6 deletions src/main/antlr4/liquid/parser/v4/LiquidParser.g4
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,10 +124,6 @@ comment_tag
: TagStart CommentStart TagEnd .*? TagStart CommentEnd TagEnd
;

other_than_tag_start
: ~( TagStart )*
;

if_tag
: TagStart IfStart expr TagEnd block elsif_tag* else_tag? TagStart IfEnd TagEnd
;
Expand Down Expand Up @@ -230,8 +226,21 @@ output
| {isWarn() || isLax()}? outStart term filter* unparsed=not_out_end? OutEnd
;

// When doing `( ~OutEnd )+`, it appears ANTLR is much slower on large input text. Even when `isStrict() == true` the
// parser will never get here, but the prediction algorithm still tries this branch and takes too much time when the
// much too large set `( ~OutEnd )+` is used. The tokens below are all tokens that are possible when the lexer is in
// the `IN_TAG` mode.
//
// The input from https://github.com/bkiers/Liqp/issues/310 is tested by parsing it 100 times. When this rule contains
// `( ~OutEnd )+`, it ran in about 8000-8500 ms on average. With the individual tokens specified in the `IN_TAG` mode,
// the average runtime was around 3000-3200 ms.
//
// All tokens in the `IN_TAG` mode _except_ the `OutEnd` token
not_out_end
: ( ~OutEnd )+
: ( TagEnd | OutStart2 | Str | DotDot | Dot | NEq | Eq | EqSign | GtEq | Gt | LtEq | Lt | Minus | Pipe
| Col | Comma | OPar | CPar | OBr | CBr | QMark | PathSep | DoubleNum | LongNum | Contains | In | And
| Or | True | False | Nil | With | Offset | Continue | Reversed | Empty | Blank | IdChain | Id
)+
;

filter
Expand Down Expand Up @@ -333,8 +342,12 @@ other_tag_parameters
: other_than_tag_end
;

// All tokens in the `IN_TAG` mode _except_ the `TagEnd` token
other_than_tag_end
: ~TagEnd+
: ( OutEnd | OutStart2 | Str | DotDot | Dot | NEq | Eq | EqSign | GtEq | Gt | LtEq | Lt | Minus | Pipe
| Col | Comma | OPar | CPar | OBr | CBr | QMark | PathSep | DoubleNum | LongNum | Contains | In | And
| Or | True | False | Nil | With | Offset | Continue | Reversed | Empty | Blank | IdChain | Id
)+
;

filename
Expand Down
Loading