Add comprehensive MainLoop architecture deep dive analysis

Co-authored-by: tig <[email protected]>

MAINLOOP_DEEP_DIVE.md

@@ -0,0 +1,593 @@
+# Deep Dive: Terminal.Gui MainLoop Architecture Analysis
+
+## Executive Summary
+
+After independently analyzing the Terminal.Gui codebase, I've discovered Terminal.Gui uses a **dual-architecture system** with modern and legacy mainloop implementations. The architecture separates concerns into distinct threads and phases, with the "MainLoop" actually referring to multiple different concepts depending on context.
+
+## Key Discovery: Two MainLoop Implementations
+
+Terminal.Gui has **TWO distinct MainLoop architectures**:
+
+1. **Modern Architecture** (`ApplicationMainLoop<T>`) - Used by v2 drivers (DotNet, Windows, Unix)
+2. **Legacy Architecture** (`MainLoop`) - Marked obsolete, used only by FakeDriver for backward compatibility
+
+This explains the confusion around terminology - "MainLoop" means different things in different contexts.
+
+## Architecture Overview
+
+### The Modern Architecture (v2)
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│                     APPLICATION LAYER                            │
+│                                                                  │
+│  Application.Run(toplevel)                                      │
+│    ├─> Begin(toplevel) → RunState token                         │
+│    ├─> LOOP: while(toplevel.Running)                            │
+│    │     └─> Coordinator.RunIteration()                         │
+│    └─> End(runState)                                            │
+│                                                                  │
+└──────────────────────────┬───────────────────────────────────────┘
+                           │
+┌──────────────────────────▼───────────────────────────────────────┐
+│                  MAINLOOP COORDINATOR                            │
+│                                                                  │
+│  Two separate threads:                                          │
+│                                                                  │
+│  ┌────────────────────┐        ┌─────────────────────────┐     │
+│  │   INPUT THREAD     │        │    MAIN UI THREAD        │     │
+│  │                    │        │                          │     │
+│  │  ConsoleInput.Run()│        │ ApplicationMainLoop      │     │
+│  │  (blocking read)   │        │   .Iteration()           │     │
+│  │         │          │        │                          │     │
+│  │         ▼          │        │  1. ProcessQueue()       │     │
+│  │  InputBuffer       │───────>│  2. Check for changes    │     │
+│  │  (ConcurrentQueue) │        │  3. Layout if needed     │     │
+│  │                    │        │  4. Draw if needed       │     │
+│  │                    │        │  5. RunTimers()          │     │
+│  │                    │        │  6. Throttle             │     │
+│  └────────────────────┘        └─────────────────────────┘     │
+│                                                                  │
+└──────────────────────────┬───────────────────────────────────────┘
+                           │
+┌──────────────────────────▼───────────────────────────────────────┐
+│                    DRIVER LAYER                                  │
+│                                                                  │
+│  IConsoleInput<T>          IConsoleOutput                       │
+│  IInputProcessor           OutputBuffer                         │
+│  IComponentFactory<T>      IWindowSizeMonitor                   │
+│                                                                  │
+└──────────────────────────────────────────────────────────────────┘
+```
+
+## Detailed Component Analysis
+
+### 1. Application.Run() - The Complete Lifecycle
+
+**Location:** `Terminal.Gui/App/Application.Run.cs`
+
+```csharp
+public static void Run(Toplevel view, Func<Exception, bool>? errorHandler = null)
+{
+    RunState rs = Application.Begin(view);     // Setup phase
+    
+    while (toplevel.Running)                   // The actual "run loop"
+    {
+        Coordinator.RunIteration();            // One iteration
+    }
+    
+    Application.End(rs);                       // Cleanup phase
+}
+```
+
+**Purpose:** High-level convenience method that orchestrates the complete lifecycle.
+
+**Key insight:** `Run()` is a WRAPPER, not the loop itself.
+
+### 2. Application.Begin() - Setup Phase
+
+**What it does:**
+1. Creates `RunState` token (handle for Begin/End pairing)
+2. Pushes Toplevel onto `TopLevels` stack
+3. Sets `Application.Top` to the new toplevel
+4. Initializes and lays out the toplevel
+5. Draws initial screen
+6. Fires `NotifyNewRunState` event
+
+**Purpose:** Prepares a Toplevel for execution but does NOT start the loop.
+
+### 3. The Loop Layer - Where Things Get Interesting
+
+There are actually **THREE different "loop" concepts**:
+
+#### A. The Application Loop (`while (toplevel.Running)`)
+
+**Location:** `ApplicationImpl.Run()`
+
+```csharp
+while (_topLevels.TryPeek(out found) && found == view && view.Running)
+{
+    _coordinator.RunIteration();  // Call coordinator
+}
+```
+
+**Purpose:** The main control loop at the Application level. Continues until `RequestStop()` is called.
+
+**This is NOT a "RunLoop"** - it's the application's execution loop.
+
+#### B. The Coordinator Iteration (`Coordinator.RunIteration()`)
+
+**Location:** `MainLoopCoordinator<T>.RunIteration()`
+
+```csharp
+public void RunIteration()
+{
+    _loop.Iteration();  // Delegates to ApplicationMainLoop
+}
+```
+
+**Purpose:** Thin wrapper that delegates to ApplicationMainLoop.
+
+#### C. The ApplicationMainLoop Iteration (`ApplicationMainLoop<T>.Iteration()`)
+
+**Location:** `ApplicationMainLoop<T>.Iteration()` and `.IterationImpl()`
+
+```csharp
+public void Iteration()
+{
+    DateTime dt = Now();
+    int timeAllowed = 1000 / MaximumIterationsPerSecond;
+    
+    IterationImpl();  // Do the actual work
+    
+    // Throttle to respect MaximumIterationsPerSecond
+    TimeSpan sleepFor = TimeSpan.FromMilliseconds(timeAllowed) - (Now() - dt);
+    if (sleepFor.Milliseconds > 0)
+        Task.Delay(sleepFor).Wait();
+}
+
+internal void IterationImpl()
+{
+    InputProcessor.ProcessQueue();           // 1. Process buffered input
+    ToplevelTransitionManager.RaiseReadyEventIfNeeded();
+    ToplevelTransitionManager.HandleTopMaybeChanging();
+    
+    // 2. Check if any views need layout or drawing
+    bool needsDrawOrLayout = AnySubViewsNeedDrawn(...);
+    bool sizeChanged = WindowSizeMonitor.Poll();
+    
+    if (needsDrawOrLayout || sizeChanged)
+    {
+        Application.LayoutAndDraw(true);     // 3. Layout and draw
+        Out.Write(OutputBuffer);             // 4. Flush to console
+    }
+    
+    SetCursor();                             // 5. Update cursor
+    TimedEvents.RunTimers();                 // 6. Run timeout callbacks
+}
+```
+
+**Purpose:** This is the REAL "main loop iteration" that does all the work.
+
+### 4. The Legacy MainLoop (for FakeDriver)
+
+**Location:** `Terminal.Gui/App/MainLoop/LegacyMainLoopDriver.cs`
+
+```csharp
+[Obsolete("This class is for legacy FakeDriver compatibility only")]
+public class MainLoop : IDisposable
+{
+    internal void RunIteration()
+    {
+        RunAnsiScheduler();
+        MainLoopDriver?.Iteration();
+        TimedEvents.RunTimers();
+    }
+}
+```
+
+**Purpose:** Backward compatibility with v1 FakeDriver. Marked obsolete.
+
+**Key difference:** This is what `Application.RunLoop()` calls when using legacy drivers.
+
+### 5. Application.RunLoop() - The Misnamed Method
+
+**Location:** `Application.Run.cs`
+
+```csharp
+public static void RunLoop(RunState state)
+{
+    var firstIteration = true;
+    
+    for (state.Toplevel.Running = true; state.Toplevel?.Running == true;)
+    {
+        if (MainLoop is { })
+            MainLoop.Running = true;
+        
+        if (EndAfterFirstIteration && !firstIteration)
+            return;
+        
+        firstIteration = RunIteration(ref state, firstIteration);
+    }
+    
+    if (MainLoop is { })
+        MainLoop.Running = false;
+}
+```
+
+**Purpose:** For legacy code that manually controls the loop.
+
+**Key insight:** This is a LEGACY compatibility method that calls either:
+- Modern: The application loop (via `RunIteration`)
+- Legacy: `MainLoop.RunIteration()` when legacy driver is used
+
+**The name "RunLoop" is misleading** because:
+1. It doesn't "run" a loop - it IS a loop
+2. It's actually the application-level control loop
+3. The real "mainloop" work happens inside `RunIteration()`
+
+### 6. Application.RunIteration() - One Cycle
+
+**Location:** `Application.Run.cs`
+
+```csharp
+public static bool RunIteration(ref RunState state, bool firstIteration = false)
+{
+    // If driver has events pending, do an iteration of the driver MainLoop
+    if (MainLoop is { Running: true } && MainLoop.EventsPending())
+    {
+        if (firstIteration)
+            state.Toplevel.OnReady();
+        
+        MainLoop.RunIteration();  // LEGACY path
+        Iteration?.Invoke(null, new());
+    }
+    
+    firstIteration = false;
+    
+    if (Top is null)
+        return firstIteration;
+    
+    LayoutAndDraw(TopLevels.Any(v => v.NeedsLayout || v.NeedsDraw));
+    
+    if (PositionCursor())
+        Driver?.UpdateCursor();
+    
+    return firstIteration;
+}
+```
+
+**Purpose:** Process ONE iteration - either legacy or modern path.
+
+**Modern path:** Called by `ApplicationImpl.Run()` via coordinator
+**Legacy path:** Calls `MainLoop.RunIteration()` (obsolete)
+
+### 7. Application.End() - Cleanup Phase
+
+**What it does:**
+1. Fires `Toplevel.OnUnloaded()` event
+2. Pops Toplevel from `TopLevels` stack
+3. Fires `Toplevel.OnClosed()` event
+4. Restores previous `Top`
+5. Clears RunState.Toplevel
+6. Disposes RunState token
+7. Forces final layout/draw
+
+**Purpose:** Balances `Begin()` - cleans up after execution.
+
+## The Input Threading Model
+
+One of the most important aspects is the **two-thread architecture**:
+
+### Input Thread (Background)
+
+```
+IConsoleInput<T>.Run(CancellationToken)
+    └─> Infinite loop:
+        1. Read from console (BLOCKING)
+        2. Parse raw input
+        3. Push to InputBuffer (thread-safe ConcurrentQueue)
+        4. Repeat until cancelled
+```
+
+**Purpose:** Dedicated thread for blocking console I/O operations.
+
+**Platform-specific implementations:**
+- `NetInput` (DotNet driver) - Uses `Console.ReadKey()`
+- `WindowsInput` - Uses Win32 API `ReadConsoleInput()`
+- `UnixDriver.UnixInput` - Uses Unix terminal APIs
+
+### Main UI Thread (Foreground)
+
+```
+ApplicationMainLoop<T>.IterationImpl()
+    └─> One iteration:
+        1. InputProcessor.ProcessQueue()
+           └─> Drain InputBuffer
+           └─> Translate to Key/Mouse events
+           └─> Fire KeyDown/KeyUp/MouseEvent
+        2. Check for view changes
+        3. Layout if needed
+        4. Draw if needed
+        5. Update output buffer
+        6. Flush to console
+        7. Run timeout callbacks
+```
+
+**Purpose:** Main thread where all UI operations happen.
+
+**Thread safety:** `ConcurrentQueue<T>` provides thread-safe handoff between threads.
+
+## The RequestStop Flow
+
+```
+User Action (e.g., Ctrl+Q)
+    │
+    ▼
+InputProcessor processes key
+    │
+    ▼
+Fires KeyDown event
+    │
+    ▼
+Application.Keyboard handles QuitKey
+    │
+    ▼
+Application.RequestStop(toplevel)
+    │
+    ▼
+Sets toplevel.Running = false
+    │
+    ▼
+while(toplevel.Running) loop exits
+    │
+    ▼
+Application.End() cleans up
+```
+
+## Key Terminology Issues Discovered
+
+### Issue 1: "RunLoop" is Ambiguous
+
+The term "RunLoop" refers to:
+1. **`Application.RunLoop()` method** - Application-level control loop (legacy compatibility)
+2. **`MainLoop` class** - Legacy main loop driver (obsolete)
+3. **`MainLoop.RunIteration()`** - Legacy iteration method
+4. **The actual application loop** - `while(toplevel.Running)`
+5. **The coordinator iteration** - `Coordinator.RunIteration()`
+6. **The main loop iteration** - `ApplicationMainLoop.Iteration()`
+
+**Problem:** Six different concepts all contain "Run" or "Loop"!
+
+### Issue 2: "RunState" Doesn't Hold State
+
+`RunState` is actually a **token** or **handle** for the Begin/End pairing:
+
+```csharp
+public class RunState : IDisposable
+{
+    public Toplevel Toplevel { get; internal set; }
+    // That's it - no "state" data!
+}
+```
+
+**Purpose:** Ensures `End()` is called with the same Toplevel that `Begin()` set up.
+
+### Issue 3: "RunIteration" Has Two Paths
+
+`Application.RunIteration()` does different things depending on the driver:
+
+**Legacy path** (FakeDriver):
+```
+RunIteration() 
+    → MainLoop.EventsPending()
+    → MainLoop.RunIteration()
+    → LayoutAndDraw()
+```
+
+**Modern path** (Normal use):
+```
+ApplicationImpl.Run()
+    → Coordinator.RunIteration()
+    → ApplicationMainLoop.Iteration()
+    → (includes LayoutAndDraw internally)
+```
+
+## The True MainLoop Architecture
+
+The **actual mainloop** in modern Terminal.Gui is:
+
+```
+ApplicationMainLoop<T>
+    ├─ InputBuffer (ConcurrentQueue<T>)
+    ├─ InputProcessor (processes queue)
+    ├─ OutputBuffer (buffered drawing)
+    ├─ ConsoleOutput (writes to terminal)
+    ├─ TimedEvents (timeout callbacks)
+    ├─ WindowSizeMonitor (detects resizing)
+    └─ ToplevelTransitionManager (handles Top changes)
+```
+
+**Coordinated by:**
+```
+MainLoopCoordinator<T>
+    ├─ Input thread: IConsoleInput<T>.Run()
+    ├─ Main thread: ApplicationMainLoop<T>.Iteration()
+    └─ Synchronization via ConcurrentQueue
+```
+
+**Exposed via:**
+```
+ApplicationImpl
+    └─ IMainLoopCoordinator (hides implementation details)
+```
+
+## Complete Execution Flow Diagram
+
+```
+Application.Init()
+    ├─> Create Coordinator
+    ├─> Start input thread (IConsoleInput.Run)
+    ├─> Initialize ApplicationMainLoop
+    └─> Return to caller
+
+Application.Run(toplevel)
+    │
+    ├─> Application.Begin(toplevel)
+    │   ├─> Create RunState token
+    │   ├─> Push to TopLevels stack
+    │   ├─> Initialize & layout toplevel
+    │   ├─> Initial draw
+    │   └─> Fire NotifyNewRunState
+    │
+    ├─> while (toplevel.Running)  ← APPLICATION CONTROL LOOP
+    │   │
+    │   └─> Coordinator.RunIteration()
+    │       │
+    │       └─> ApplicationMainLoop.Iteration()
+    │           │
+    │           ├─> Throttle based on MaxIterationsPerSecond
+    │           │
+    │           └─> IterationImpl():
+    │               │
+    │               ├─> 1. InputProcessor.ProcessQueue()
+    │               │     └─> Drain InputBuffer
+    │               │     └─> Fire Key/Mouse events
+    │               │
+    │               ├─> 2. ToplevelTransitionManager events
+    │               │
+    │               ├─> 3. Check if layout/draw needed
+    │               │
+    │               ├─> 4. If needed:
+    │               │     ├─> Application.LayoutAndDraw()
+    │               │     └─> Out.Write(OutputBuffer)
+    │               │
+    │               ├─> 5. SetCursor()
+    │               │
+    │               └─> 6. TimedEvents.RunTimers()
+    │
+    └─> Application.End(runState)
+        ├─> Fire OnUnloaded event
+        ├─> Pop from TopLevels stack
+        ├─> Fire OnClosed event
+        ├─> Restore previous Top
+        ├─> Clear & dispose RunState
+        └─> Final layout/draw
+
+Application.Shutdown()
+    ├─> Coordinator.Stop()
+    │   └─> Cancel input thread
+    └─> Cleanup all resources
+```
+
+## Parallel: Input Thread Flow
+
+```
+Input Thread (runs concurrently):
+
+IConsoleInput<T>.Run(token)
+    │
+    └─> while (!token.IsCancellationRequested)
+        │
+        ├─> Read from console (BLOCKING)
+        │   ├─ DotNet: Console.ReadKey()
+        │   ├─ Windows: ReadConsoleInput() API
+        │   └─ Unix: Terminal read APIs
+        │
+        ├─> Parse raw input to <T>
+        │
+        ├─> InputBuffer.Enqueue(input)  ← Thread-safe handoff
+        │
+        └─> Loop back
+```
+
+## Summary of Discoveries
+
+### 1. Dual Architecture
+- Modern: `ApplicationMainLoop<T>` with coordinator pattern
+- Legacy: `MainLoop` class for FakeDriver only (obsolete)
+
+### 2. Two-Thread Model
+- Input thread: Blocking console reads
+- Main UI thread: Event processing, layout, drawing
+
+### 3. The "MainLoop" Misnomer
+- There's no single "MainLoop" - it's distributed across:
+  - Application control loop (`while (toplevel.Running)`)
+  - Coordinator iteration
+  - ApplicationMainLoop iteration
+  - Legacy MainLoop (obsolete)
+
+### 4. RunState is a Token
+- Not state data
+- Just a handle to pair Begin/End
+- Contains only the Toplevel reference
+
+### 5. Iteration Hierarchy
+```
+Application.RunLoop()          ← Legacy compatibility method (the LOOP)
+    └─> Application.RunIteration()  ← One iteration (modern or legacy)
+        └─> Modern: Coordinator.RunIteration()
+            └─> ApplicationMainLoop.Iteration()
+                └─> IterationImpl()  ← The REAL mainloop work
+```
+
+### 6. The Real MainLoop
+- Is `ApplicationMainLoop<T>`
+- Runs on the main thread
+- Does one iteration per call to `Iteration()`
+- Is throttled to MaximumIterationsPerSecond
+- Processes input, layouts, draws, runs timers
+
+## Terminology Recommendations Based on Analysis
+
+Based on this deep dive, here are the terminology issues:
+
+### Critical Issues
+
+1. **`RunState` should be `RunToken`**
+   - It's a token, not state
+   - Analogy: `CancellationToken`
+
+2. **`EndAfterFirstIteration` should be `StopAfterFirstIteration`**
+   - Controls loop stopping, not lifecycle cleanup
+   - "End" suggests `End()` method
+   - "Stop" aligns with `RequestStop()`
+
+### Less Critical (But Still Confusing)
+
+3. **`Application.RunLoop()` is misnamed**
+   - It IS a loop, not something that "runs" a loop
+   - Legacy compatibility method
+   - Better name would describe it's the application control loop
+
+4. **"MainLoop" is overloaded**
+   - `MainLoop` class (obsolete)
+   - `ApplicationMainLoop` class (modern)
+   - The loop concept itself
+   - Used in variable names throughout
+
+### What Works Well
+
+- `Begin` / `End` - Clear lifecycle pairing
+- `RequestStop` - "Request" conveys non-blocking
+- `RunIteration` - Clear it processes one iteration
+- `Coordinator` - Good separation of concerns pattern
+- `ApplicationMainLoop<T>` - Descriptive class name
+
+## Conclusion
+
+Terminal.Gui's mainloop is actually a sophisticated multi-threaded architecture with:
+- Separate input/UI threads
+- Thread-safe queue for handoff
+- Coordinator pattern for lifecycle management
+- Throttled iterations for performance
+- Clean separation between drivers and application logic
+
+The confusion stems from:
+1. Legacy compatibility with FakeDriver
+2. Overloaded "Run" terminology
+3. Misleading "RunState" name
+4. The term "MainLoop" meaning different things
+
+The modern architecture is well-designed, but the naming reflects the evolution from v1 to v2, carrying forward legacy terminology that no longer accurately describes the current implementation.