From 301d53d7ae28c711b70a45a6ae3ddccdd5cb40ee Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Yann=20C=C3=A9bron?= Date: Wed, 7 Sep 2022 17:41:37 +0200 Subject: [PATCH] implementing_lexer.md: more TokenSet --- .../custom_language_support/implementing_lexer.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/topics/reference_guide/custom_language_support/implementing_lexer.md b/topics/reference_guide/custom_language_support/implementing_lexer.md index 9b41e7054..7aa32c422 100644 --- a/topics/reference_guide/custom_language_support/implementing_lexer.md +++ b/topics/reference_guide/custom_language_support/implementing_lexer.md @@ -67,11 +67,15 @@ Custom language plugins should reuse these token types wherever applicable. For all other token types, the plugin needs to create new [`IElementType`](%gh-ic%/platform/core-api/src/com/intellij/psi/tree/IElementType.java) instances and associate with the language in which the token type is used. The same [`IElementType`](%gh-ic%/platform/core-api/src/com/intellij/psi/tree/IElementType.java) instance should be returned every time a particular token type is encountered by the lexer. -Groups of related types (e.g., keywords) can be defined using [`TokenSet`](%gh-ic%/platform/core-api/src/com/intellij/psi/tree/TokenSet.java). - **Example:** [Token types](%gh-ic%/plugins/properties/properties-psi-api/src/com/intellij/lang/properties/parsing/PropertiesTokenTypes.java) for [Properties language plugin](%gh-ic%/plugins/properties) +Groups of related types (e.g., keywords) can be defined using [`TokenSet`](%gh-ic%/platform/core-api/src/com/intellij/psi/tree/TokenSet.java). +All `TokenSet` for a language should be grouped in a dedicated `$Language$TokenSets` class for re-use. + +**Example:** +[`GroovyTokenSets`](%gh-ic%/plugins/groovy/groovy-psi/src/org/jetbrains/plugins/groovy/lang/psi/GroovyTokenSets.java) + ### Embedded Language An important feature that can be implemented at the lexer level is mixing languages within a file, such as embedding fragments of Java code in some template language.