hxuanyu/intellij-sdk-code-samples
1
0
Fork 0
mirror of https://github.com/JetBrains/intellij-sdk-code-samples.git synced 2025-07-30 18:27:49 +08:00
Code Issues Packages Projects Releases Wiki Activity

implementing_parser_and_psi.md: note TokenSet.WHITE_SPACE

Browse Source
This commit is contained in:
Yann Cébron 2022-09-05 18:44:54 +02:00
parent 07559afb9a
commit d7705a405d
1 changed files with 1 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
topics/reference_guide/custom_language_support/implementing_parser_and_psi.md
View File

@ -60,6 +60,7 @@ An essential feature of `PsiBuilder` is its handling of whitespace and comments.
The types of tokens which are treated as whitespace or comments are defined by `getWhitespaceTokens()` and `getCommentTokens()` in [`ParserDefinition`](%gh-ic%/platform/core-api/src/com/intellij/lang/ParserDefinition.java).
`PsiBuilder` automatically omits whitespace and comment tokens from the stream of tokens it passes to `PsiParser` and adjusts the token ranges of AST nodes so that leading and trailing whitespace tokens are not included in the node.
Most languages will not need to override `getWhitespaceTokens()` which returns the language-agnostic [`TokenSet.WHITE_SPACE`](%gh-ic%/platform/core-api/src/com/intellij/psi/tree/TokenSet.java) by default.
The token set returned from [`ParserDefinition.getCommentTokens()`](%gh-ic%/platform/core-api/src/com/intellij/lang/ParserDefinition.java) is also used to search for [TODO items](https://www.jetbrains.com/help/idea/using-todo.html).
To better understand the process of building a PSI tree for a simple expression, you can refer to the following diagram: