more info about read/write actions, transactions, modality states, dumb mode (IJSDK-200), gists (IJSDK-211)

basics/architectural_overview/general_threading_rules.md

@@ -2,10 +2,36 @@
 title: General Threading Rules
 ---
 
-In general, data structures in the *IntelliJ Platform* are covered by a single reader/writer lock.
+## Read/write lock
 
-Reading data is allowed from any thread.  Reading data from the UI thread does not require any special effort. However, read operations performed from any other thread need to be wrapped in a read action by using `ApplicationManager.getApplication().runReadAction()`.
+In general, code-related data structures in the *IntelliJ Platform* are covered by a single reader/writer lock. This applies to PSI, VFS and project root model.
 
-Writing data is only allowed from the UI thread, and write operations always need to be wrapped in a write action with `ApplicationManager.getApplication().runWriteAction()`.
+Reading data is allowed from any thread.  Reading data from the UI thread does not require any special effort. However, read operations performed from any other thread need to be wrapped in a read action by using `ApplicationManager.getApplication().runReadAction()` or, shorter, `ReadAction.run/compute`.
 
-To pass control from a background thread to the event dispatch thread, instead of the standard `SwingUtilities.invokeLater()`, plugins should use `ApplicationManager.getApplication().invokeLater()`. The latter API allows specifying the _modality state_ for the call, i.e. the stack of modal dialogs under which the call is allowed to execute. Passing `ModalityState.NON_MODAL` means that the operation will be executed after all modal dialogs are closed. Passing `ModalityState.stateForComponent()` means that the operation may be executed while the specified component (part of a dialog) is still visible.
+Writing data is only allowed from the UI thread, and write operations always need to be wrapped in a write action with `ApplicationManager.getApplication().runWriteAction()` or, shorter, `WriteAction.run/compute`.
+
+In addition, modifying the model is only allowed from write-safe contexts, which include user actions, `invokeLater` calls from them (see the next section), and transactions (`TransactionGuard.submitTransaction`). You may not modify PSI, VFS or project model from inside UI renderers or `SwingUtilities.invokeLater` calls. See `TransactionGuard` documentation for more details.
+
+You must not access the model outside a read or write action. The corresponding objects are not guaranteed to survive between several consecutive read actions. So as a rule of thumb, whenever you start a read action, you should check if your PSI/VFS/project/module are still valid.
+
+## invokeLater
+
+To pass control from a background thread to the event dispatch thread, instead of the standard `SwingUtilities.invokeLater()`, plugins should use `ApplicationManager.getApplication().invokeLater()`. The latter API allows specifying the _modality state_ for the call, i.e. the stack of modal dialogs under which the call is allowed to execute. 
+* Passing `ModalityState.NON_MODAL` means that the operation will be executed after all modal dialogs are closed. Note that this state is almost never appropriate, because if any of the open (unrelated) project displays a per-project modal dialog, the action will be executed after it's closed. 
+* Passing `ModalityState.stateForComponent()` means that the operation can be executed when the topmost shown dialog is the one that contains the specified component, or is one of its parent dialogs. 
+* If no modality state is passed, `ModalityState.defaultModalityState()` will be used. In most cases, this is the optimal choice, that uses current modality state when invoked from UI thread, and has a special handling for background processes started with `ProgressManager`: `invokeLater` from such a process may run in the same dialog that the process started.
+* `ModalityState.any()` means that the runnable will be executed as soon as possible regardless of modal dialogs. Please note that modifying PSI, VFS or project model is prohibited from such runnables. See `TransactionGuard` documentation for more details.
+
+If your UI thread activity needs to access [file-based index](../indexing_and_psi_stubs.md) (e.g. it's doing any kind of project-wide PSI analysis, resolves references, etc), please use `DumbService#smartInvokeLater`. That way, your activity will be run after all possible indexing processes have been completed.
+
+## Preventing UI freezes
+
+Background threads shouldn't take read actions for a long time. The reason is that if the UI thread needs a write action (e.g. the user types something), it must acquire it as soon as possible, otherwise the UI will freeze until all background threads have released their read actions.
+
+The best known approach to that is to cancel background read actions whenever there's a write action about to occur, and restart that background read action later from the scratch. Editor highlighting, code completion, Goto Class/File/etc actions all work like this. There are two recommended ways of doing this:
+* If you're on UI thread, create a `ReadTask` and pass it to one of `ProgressIndicatorUtils.schedule*` methods. Inside `onCanceled` method, schedule it again if the activity should be restarted. 
+* If you're already in a background thread, use `ProgressManager.getInstance().runInReadActionWithWriteActionPriority()` in a loop, until it passes or the whole activity becomes obsolete.
+
+In both approaches, you should always check at the start of each read action, if the objects you're working with are still valid, and if the whole operation still makes sense (i.e. not canceled by the user, the project isn't closed, etc.)
+
+If the activity you're doing has to access [file-based index](../indexing_and_psi_stubs.md) (e.g. it's doing any kind of project-wide PSI analysis, resolves references, etc), you should override `ReadTask#runBackgroundProcess` and use "smart-mode" read action there: `DumbService.getInstance(project).runReadActionInSmartMode(() -> performInReadAction(indicator))`

basics/indexing_and_psi_stubs.md

@@ -2,14 +2,30 @@
 title: Indexing and PSI Stubs
 ---
 
+## Indices
 
 The indexing framework provides a quick way to locate certain elements, e.g. files containing a certain word or methods with a particular name, in large code bases. Plugin developers can use the existing indexes built by the IDE itself, as well as build and use their own indexes.
 
 It supports two main types of indexes:
 
-* [File-based indexes](/basics/indexing_and_psi_stubs/file_based_indexes.md)
-* [Stub indexes](/basics/indexing_and_psi_stubs/stub_indexes.md)
+* [File-based indices](/basics/indexing_and_psi_stubs/file_based_indexes.md)
+* [Stub indices](/basics/indexing_and_psi_stubs/stub_indexes.md)
 
 File-based indexes are built directly over the content of files. Stub indexes are built over serialized *stub trees*. A stub tree for a source file is a subset of its PSI tree which contains only externally visible declarations and is serialized in a compact binary format.
 
 Querying a file-based index gets you the set of files matching a certain condition. Querying a stub index gets you the set of matching PSI elements. Therefore, custom language plugin developers should typically use stub indexes in their plugin implementations.
+
+## Dumb mode
+
+Indexing is a potentially long process. It's performed in background, and during this time, IDE's features are restricted to the ones that don't require index: basic text editing, version control etc. This restriction is managed by [DumbService](upsource:///platform/core-api/src/com/intellij/openapi/project/DumbService.java).
+
+`DumbService` provides API to query whether the IDE is currently in "dumb" mode (where index access is not allowed) or "smart" mode (with all index built and ready to use). It also provides ways of delaying code execution until indices are ready. Please see its javadoc for more details.
+
+## Gists
+
+Sometimes, the following conditions hold:
+* the aggregation functionality of file-based indices is not needed. One just needs to calculate some data based on particular file's contents, and cache it on disk
+* eagerly calculating the data for the entire project during indexing isn't needed (e.g. it slows down the indexing, and/or this data probably will ever be needed for a minor subset of all project files)
+* the data can be recalculated lazily on request without major performance penalties
+
+In such cases, file-based index can be used, but file gists provide a way to perform data calculation lazily, caching on disk, and a more lightweight API. Please see [VirtualFileGist](upsource:///platform/indexing-api/src/com/intellij/util/gist/VirtualFileGist.java) and [VirtualFileGist](upsource:///platform/indexing-api/src/com/intellij/util/gist/PsiFileGist.java) documentation.