hxuanyu/intellij-sdk-code-samples
1
0
Fork 0
mirror of https://github.com/JetBrains/intellij-sdk-code-samples.git synced 2025-07-28 01:07:49 +08:00
Code Issues Packages Projects Releases Wiki Activity

implementing_lexer.md: more TokenSet

Browse Source
This commit is contained in:
Yann Cébron 2022-09-07 17:41:37 +02:00
parent f6b00c94d3
commit 301d53d7ae
1 changed files with 6 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
topics/reference_guide/custom_language_support/implementing_lexer.md
View File

@ -67,11 +67,15 @@ Custom language plugins should reuse these token types wherever applicable.
For all other token types, the plugin needs to create new [`IElementType`](%gh-ic%/platform/core-api/src/com/intellij/psi/tree/IElementType.java) instances and associate with the language in which the token type is used.
The same [`IElementType`](%gh-ic%/platform/core-api/src/com/intellij/psi/tree/IElementType.java) instance should be returned every time a particular token type is encountered by the lexer.
Groups of related types (e.g., keywords) can be defined using [`TokenSet`](%gh-ic%/platform/core-api/src/com/intellij/psi/tree/TokenSet.java).
**Example:**
[Token types](%gh-ic%/plugins/properties/properties-psi-api/src/com/intellij/lang/properties/parsing/PropertiesTokenTypes.java) for [Properties language plugin](%gh-ic%/plugins/properties)
Groups of related types (e.g., keywords) can be defined using [`TokenSet`](%gh-ic%/platform/core-api/src/com/intellij/psi/tree/TokenSet.java).
All `TokenSet` for a language should be grouped in a dedicated `$Language$TokenSets` class for re-use.
**Example:**
[`GroovyTokenSets`](%gh-ic%/plugins/groovy/groovy-psi/src/org/jetbrains/plugins/groovy/lang/psi/GroovyTokenSets.java)
### Embedded Language
An important feature that can be implemented at the lexer level is mixing languages within a file, such as embedding fragments of Java code in some template language.