FjUdZddlmZddlmZmZddlmZmZm Z m Z ddl Z ddl Z dZ dId ZeGd dZeGddZdJdZdKdZdLdZdMdZdNdZdOdZdPd"ZeeeeeeeegeefZdQd$ZdRd&ZdSd)ZdTd,ZdUd.Z dUd/Z!dUd0Z"dUd1Z#dUd2Z$dUd3Z%dUd4Z&dUd5Z'e e!e"e#e$e%e&e'gZ(d6e)d7<d8Z*d9d9d9d:d;d<Z+dVd>Z,dWd?Z-ddd@dXdEZ.dYdHZ/dS)ZuLKanban diagnostics — structured, actionable distress signals for tasks. A ``Diagnostic`` is a machine-readable description of something that's wrong with a kanban task: a hallucinated card id, a spawn crash-loop, a task stuck blocked for too long, etc. Each one carries: * A **kind** (canonical code; UI/tests match on this). * A **severity** (``warning`` / ``error`` / ``critical``). * A **title** (one-line human description) and **detail** (longer text). * A list of **suggested actions** — structured entries the dashboard turns into buttons and the CLI turns into hints. Rules run over (task, recent events, recent runs) and emit diagnostics. They are stateless and read-only — no DB writes. Callers compute diagnostics on demand (on ``/board`` load, ``/tasks/:id`` fetch, or ``hermes kanban diagnostics``). Design goals: * Fixable-on-the-operator's-side signals only (missing config, phantom ids, crash loop). Not "the provider returned 502 once" — that's a transient runtime blip, not a diagnostic. * Recoverable: every diagnostic comes with at least one suggested recovery action the operator can actually take from the UI. * Auto-clearing: when the underlying failure mode resolves (a clean ``completed`` event arrives, a spawn succeeds, the task gets unblocked), the diagnostic stops firing. The audit event trail stays. ) annotations) dataclassfield)AnyCallableIterableOptionalN)warningerrorcriticalseverity Optional[str] thresholdreturnboolc|dS|tvs |tvrdSt|t|kS)z=Return True when ``severity`` meets or exceeds ``threshold``.NTF)SEVERITY_ORDERindex)r rs auth``). No HTTP side effect. * ``open_docs`` — deep-link to the docs URL named in ``payload.url``. * ``comment`` — nudge the operator to add a comment (for stuck-blocked tasks that need human input). ``suggested=True`` marks the action as the recommended first step; the UI highlights it. Multiple actions can be suggested if they're equally valid. strkindlabeldefault_factorydictpayloadFr suggestedrc8|j|j|j|jdS)Nrrr r!r#selfs rto_dictzDiagnosticAction.to_dictOs%IZ|    rNrr) __name__ __module__ __qualname____doc____annotations__rrr r!r&rrrr5sw&IIIJJJE$///G////I      rrceZdZUdZded<ded<ded<ded<eeZded <d Zd ed <d Z d ed <dZ d ed<dZ ded<ee Z ded<ddZdS) Diagnosticz%One active distress signal on a task.rrr titledetailrlist[DiagnosticAction]actionsrint first_seen_at last_seen_atcountN Optional[int]run_idrdatarc |j|j|j|jd|jD|j|j|j|j|j d S)Nc6g|]}|Sr-)r&).0as r z&Diagnostic.to_dict..os ::: :::r rr r0r1r3r5r6r8r:r;rAr$s rr&zDiagnostic.to_dictisSI Zk::T\:::!/ -ZkI   rr')r(r)r*r+r,rlistr3r5r6r8r:rr;r&r-rrr/r/Xs// IIIMMMJJJKKK&+eD&A&A&AGAAAAMLENNNN F    t,,,D,,,,       rr/c||S t|dr||vr||Sn#t$rYnwxYwt|tr|||St |||S)a'Read a field from a task regardless of representation. Callers pass sqlite3.Row (dict-like with [] but no attribute access), kanban_db.Task dataclasses (attribute access), or plain dicts (both). This normalises them so rule functions don't have to branch on type each time. Nkeys)hasattrrD Exception isinstancergetgetattr)tasknamedefaults r _task_fieldrM|s |  4  TTYY[[%8%8:       $'xxg&&& 4w ' ''s-5 AArct|dd}|iSt|tr|St|tr) t j|piS#t $ricYSwxYwiS)zedited completed)rWclearappend)r\ractiverQks r_active_hallucination_eventsrfs_F OO ' ' ' LLNNNN $YY MM"    Mrctd}|D]2}t|dvrt|}t||}3|S)zTimestamp of the most recent clean completion / edit event. Kept for general "has this task ever been successfully completed" lookups; hallucination rules use ``_active_hallucination_events`` instead because they need strict ordering. r>r`ra)rWr[max)r\latestrQrZs r_latest_clean_event_tsrjsHF$$ r??5 5 5" A^^F MrrJrrunningr2cg}|r%|tddi|tddd|i|S)Nreclaimz Reclaim taskrrr reassignzReassign to different profile reclaim_first)rcr)rJrkouts r_generic_recovery_actionsrrs"$C  #       JJ - '* JrslotcTt|tsdSt|dpd}|r|dkrdSdD];}t||pdrdS>HH&&t/ txx}}" # # ) ) + + 44  5r raw_configc2t|tsdS|d}t|trt|dpd}t|dp+|dp|dpd}t |o|St t|pdS)aBest-effort check that a main model is configured. Diagnostics runs in the dashboard process which may not share the CLI's runtime state, so we read the raw config dict. If we cannot prove the main model is set, we err on the side of NOT firing the diagnostic. FrwrurUrLrK)rGrrHrrzr)r~ model_cfgrurws r_main_model_visiblers j$ ' 'uw''I)T""(y}}Z006B77==?? MM) $ $ }}W%% }}V$$    %''  H&''' IO$$**,, - --rconfigOptional[dict]ct|tsdS|d}t|tr|S|d}t|dtr|dni}t|ts|sd|vrdSd}d}t|trDt|d}t|d}d }t|tr&d |vr"t |d }|||t |d S) u)Inspect raw config and report whether triage paths look configured. Returns ``None`` when config context is unavailable (suppress diagnostic to avoid noisy false positives in tests / low-level callers). Otherwise returns a dict with: - ``auto_decompose``: bool — whether the dispatcher auto-runs decompose - ``decomposer_explicit``: bool — user-supplied decomposer slot - ``specifier_explicit``: bool — user-supplied specifier slot - ``main_model_visible``: bool — main model can serve as auto fallback Ntriage_aux_status auxiliarykanbanrwFkanban_decomposertriage_specifierTauto_decompose)rdecomposer_explicitspecifier_explicitmain_model_visible)rGrrHr}rr)rexplicitaux kanban_cfgrrrs rrrsn fd # #tzz-..H(D!! **[ ! !C)3FJJx4H4H$)O)OWH%%%UWJ sD ! ! 6 ! !t#tM09L1M1MNN/8J0K0KLLN*d##@(8J(F(Fjnn-=>>??)201&99   rvaluerLcj t|}n#ttf$r|cYSwxYw|dkr|n|S)Nr7)r4 TypeError ValueError)rrLparseds r _positive_intrNsOU z "q[[66g-s ((list[Diagnostic]c  t|d}|sgSg}t|d}t|d}|D]E} t| } | dgpgD]} | |vr|| Ft |ddk} g} | t ddd | t|| td d dd| ||t|d|i gS)aIBlocked-hallucination gate fires: a worker called kanban_complete with created_cards that didn't exist or weren't created by the completing profile. Task stayed in its prior state; the operator needs to decide how to proceed. Auto-clears when a successful completion (or edit) follows the blocked event. completion_blocked_hallucinationr phantom_cardsstatusrkcommentz#Add a comment explaining what to doFrrr!rkhallucinated_cardsr z%Worker claimed cards that don't exista The completing worker declared created_cards that either didn't exist or weren't created by its profile. The completion was blocked and the task stayed in its prior state. Usually means the worker hallucinated ids instead of capturing return values from kanban_create. phantom_ids rr r0r1r3r5r6r8r;) rfr[rSrHrcrMrextendrrr/len)rJr\runsnowcfghitsrfirstlastrQr pidrkr3s r_rule_hallucinated_cardsrVs_ (0R S SD  K d1g  E T"X  D(( $$;;339r ( (C+%%""3''' ($))Y6G&(G NN# 3  NN,T7CCCDDD  !5 1 $ii[ )    rct|ddkrgSt|}|gSt|d}t|d}t|d}t|d} |r d} |} d } |} d }d }n d } |} d} |} d }d }| s| rgSt|dpd}t dd| dd| didg}| s0| s.|t dd| dd| di|s-|t dd|dd|it ddd|dd|d |||d!||| | d"# gS)$uA triage task cannot leave triage without an auxiliary helper. With the auto-decompose dispatcher (kanban.auto_decompose, default True), triage tasks fan out via ``auxiliary.kanban_decomposer`` and fall back to ``auxiliary.triage_specifier`` when the decomposer returns ``fanout=false``. With auto-decompose off, the user must run ``hermes kanban specify``, which only needs ``auxiliary.triage_specifier``. The default slot is ``provider: auto`` → auto-falls back to the main model, so this rule only fires when: - the relevant slot is explicitly set to something broken, OR - the auto fallback has no main model to fall back to. Config context is required; pass {} from tests to keep the rule silent. rtriageNrrrrzauxiliary.kanban_decomposerzauxiliary.triage_specifier decomposerzAuto-decompose is on, so the dispatcher needs auxiliary.kanban_decomposer (with auxiliary.triage_specifier as a fallback for non-fan-out tasks). specifierzkAuto-decompose is off, so triage tasks need `hermes kanban specify`, which uses auxiliary.triage_specifier.idz cli_hintz Configure commandzhermes config set z.provider autoTr#zOr configure fallback rnz(Specify manually: hermes kanban specify zhermes kanban specify triage_aux_unavailabler zTriage z has no usable modelzZThis task is still in triage and no working auxiliary model is visible to the dispatcher. z The default slot uses `provider: auto` which falls back to the main model, but no main model is configured either. Configure the slot directly or set a main model so the auto fallback can take over.r7)task_idr primary_slotrr)rMrrrHrrcr/)rJr\rrrrrrr main_visiblerprimary_explicit fallback_slotfallback_explicit primary_desc detail_pathrr3s r_rule_triage_aux_unavailablers"4""h.. s # #F ~ &**%56677Nvzz*?@@AAfjj)=>>?? #78899L 4 .4 .#  1  4 -5 /"  N < $%%4G-|--EEEE   G   \  ':=::FFFF        'FWFF B B BC       %: ::: >*5 > > > ,(".       rc t|d}|sgSg}|D]C}t|dgpgD]}||vr||Dt |ddk} t ddddt || t|d t|d t|d|i gS) zAdvisory prose-scan: the completion summary mentions ``t_`` ids that don't resolve. Non-blocking; surfaced as a warning only. Auto-clears when a fresh clean completion arrives AFTER the suspected event. !suspected_hallucinated_references phantom_refsrrkprose_phantom_refsr z.Completion summary references unknown task idszThe completion summary mentions task ids that don't resolve in this board's database. The completion itself succeeded, but downstream consumers parsing the summary may be pointed at cards that never existed.rrrr) rfrSrHrcrMr/rrr[r) rJr\rrrrrrQrrks r_rule_prose_phantom_refsrs (0S T TD   L))!"%%))."==C ) )C,&&##C((( )$))Y6G  !> + *$@@@Q((tBx(($iil +    rct|d|ddd}t|d|}t|ddt|ddnt|dd}|||krgSt|d dt|d dnt|d d}t|d } t|d } d} t | D]} t| d} | dvr| } ng}| dkrh| rf| dkr`|t dd| ddd| did|t dd| ddd| dinD| dvr@t|d}|r.|t dd|dd|id|t|t|d d!k"||d#zkrd$nd%}|r|pd& nd&}|r"|dd't|d'krd(nd&znd&}d)d*d+d,| pd&d-}|r;d.|d/|d0| ddd1}d2|d3|d4|d5|d6 }nd.|d/|d7}d2|d3|d8}td9||||||||| |||d:; gS)z)_rule_repeated_failures..@sk!T1.E.Err|outcome>crashed timed_out spawn_failedrrLrzVerify profile: hermes -p z doctorrz hermes -p Tr#zFix profile auth: hermes -p z authrn>rrrCheck logs: hermes kanban log hermes kanban log rrkrr r rU…spawntimeoutcrash)rrrfailurezAgent z xz: zThis task has failed z times in a row (most recent: z). Full last error: z3 The dispatcher circuit breaker is configured for z_ consecutive non-success attempts. Fix the root cause and reclaim or unblock the task to retry.z (no error recorded)zP) but no error text was captured. Check the suggested command or the worker log.repeated_failures)rmost_recent_outcome last_errorrrr) rrHrMsortedreversedrcrrrrrzr splitlinesr/)rJr\rrrrrfailureslast_errr ordered_runsrrocr3rr err_text err_snippet outcome_labelr0r1s r_rule_repeated_failuresrs cgg )1--   I"#''/":":IFFM t3T : : F D0$777 / 3 3 8i//  t14 8 8 D D.555 14 8 8 4,,H $$E$EFFFL l # # I & & 9 9 9"$  E :')Gn,,,h)>S>S'@x@@@ >X > > >?       '@@@@ &Q66zzGH+3;B%%'''HMU](4C4.S]]S-@-@EEbII[]K c  #Y//   YYY(YYk6L6L6N6Nq6QRVSVRV6WYY DH D D* D D D D  D D D IHH(HHH HH H H* H H H   $,#6"!**       rc t|d|dd}t|ddpd}||krgSt|dd}t|d }d} d } t |D]6} t| d } | d kr| d z } | t| d} 0| dvrn7| |krgSt|d} g}| r.|t dd| dd| idt|ddk}|t||| |dzkrdnd}| r| pd nd}|r"|d dt|dkrdndznd}|r1d| d| dd d }d!| d"|}n d| d#}d!| d$}td%||||||| | | d&' gS)(uQThe worker spawns fine but keeps crashing mid-run. Check the last N runs' outcomes; N consecutive ``crashed`` without a successful ``completed`` means something about the task + profile combo is broken (OOM, missing dependency, tool it needs is down). Threshold: cfg["crash_threshold"] (default 2). Narrower than ``repeated_failures`` — fires earlier (2 crashes vs 3 total failures) so the operator gets a crash-specific heads-up before the unified rule kicks in. Suppresses itself when the unified rule is also about to fire, to avoid double-flagging. rrrrrcrash_thresholdrc$t|ddSrrVrs rrz(_rule_repeated_crashes..sQa)@)@rrNrrr7r >ra reclaimedrrrrrTr#rrkrr rUrrzAgent crashed zx: rz The last z4 runs ended with outcome=crashed. Full last error: zx (no error recorded)z_ runs ended with outcome=crashed but no error text was captured. Check the worker log for more.repeated_crashes)consecutive_crashesrr) r4rHrMrrrcrrrrrzrrr/)rJr\rrrrunified_counterrordered consecutiverrrrr3rkr rrr0r1s r_rule_repeated_crashesrsCGG )1-- D0!449+++ CGG-q1122IT@@AAAGKH g    a++ i   1 K&q'22 2 2 2 E Y $%%G&(G '<7<< >W > >?       $))Y6G NN,T7CCCDDD(IM99zzwH,4;B%%'''HMU](4C4.S]]S-@-@EEbII[]K  TTT1G1G1I1I!1LTcT1RTT 1  1 1#. 1 1  DCCC J  J J J   %0 I I     rc2t|dd}t|d}|dkrgSd}|D]4}t|dkrt |} t || }5|dkrgS||z dz } | |krgS|D]*}t|dvrt ||krgcS+t dd d g} td d dt| ddt| d| ||d|t| dd gS)zTask has been in ``blocked`` status for too long without a comment. Threshold: cfg["blocked_stale_hours"] (default 24). Surfaced as a warning so humans know there's a pending unblock. blocked_stale_hoursrblockedrg @> commented unblockedrz Add a comment / unblock the taskTrstuck_in_blockedr zTask has been blocked for hz"This task transitioned to blocked uh ago and has had no comments or unblock attempts since. Blocked tasks are waiting for human input — check the block reason and either unblock with feedback or answer with a comment.r7) blocked_at age_hoursr) floatrHrMrWr[rhrr/r4round) rJr\rrrhoursrlast_blocked_tsrQrZrr3s r_rule_stuck_in_blockedrs #''/44 5 5E x ( (F  O66 r??i ' '" A!/155O! &&0I5  r??8 8 8Yr]]_=\=\III4   'G  <3y>><<< FY F F F %$+% 1:M:M N N    rct|dd}t|dd}||z }d}d} d} d} |D]I} t| } | |krt | }|dkr| dkr| } | r |dz }| } d} A|d krd } J||krgSt |d }g}|r.|td d |dd|id tddd|dt|dz dd|d|| rt| nt|| rt| nt|||t|d gS)utTask has cycled through blocked → unblocked many times — the ``unblock`` is not fixing the underlying problem and the worker keeps re-blocking for substantially the same reason. ``_rule_stuck_in_blocked`` resets its timer on any ``commented`` / ``unblocked`` event, so a task that cycles every few minutes is invisible to it regardless of how many times it cycles (#29747 gap 1). This rule complements that one by counting block→unblock cycles in a sliding window. Threshold: cfg["block_cycle_threshold"] (default 3) cycles within cfg["block_cycle_window_seconds"] (default 24h). block_cycle_thresholdrblock_cycle_window_secondsiQrFrr7rTrrz*Check block reasons: hermes kanban events rzhermes kanban events r#block_unblock_cyclingr uTask block→unblock cycled zx in rzThis task has been blocked z times after being unblocked, suggesting the unblock is not addressing the root cause and the worker keeps hitting the same wall. Review the block reasons in the event history; a different intervention (reassign, change scope, archive) may be needed.)cycleswindow_secondsr) rrHrr[rWrMrcrr/r4)rJr\rrrrr cycle_cutoffr seen_unblock_since_last_cycleinitial_blocked_tslast_cycle_blocked_tsrQtsrrr3s r_rule_block_unblock_cyclingrscgg&=>>BBI377#?KKLLN'LF$)! 1 1 r]]   2 9  !Q&&%'", 6! (*%05- [ ,0 )  $%%G&(G 'HwHH A A AB        $UVUU#nT>Q:R:RUUU L& L L L 1CQc,---S3HVS.///cRUhh!.11      rc.t|dd}t|d}|dkrgSt|drgSt|dpd}|sgShd}d } |D]2} t | |vrt | } t | | } 3| d kr!tt|d d pd } | d krgS|| z } | |krgS| d kr | d z d d} nt| dz d} | |dzkrd}n| |dzkrd}nd}tddd|itddddig}td|d| d d!| d"|d#|| | d$| t| |t|d%& gS)'uTask has been in ``ready`` status for too long without any worker claiming it. Threshold: cfg["stranded_threshold_seconds"] (default 1800 = 30 min). Catches every "task waiting for a worker that never comes" case without caring WHY: * Operator typo'd the assignee — no profile or external worker matches. * Profile was deleted, leaving its tasks stranded. * External worker pool (Codex CLI, Claude Code lane, custom daemon) is down, hung, or wasn't started. * Dispatcher is misconfigured (wrong board, wrong HERMES_HOME). Pre-rule, all of these silently rotted in ``skipped_nonspawnable`` — the dispatcher correctly skipped them (good — no respawn loop) but nobody surfaced the fact that operator-actionable work was accumulating. The rule fires when a ready task's promoted-to-ready timestamp is older than the threshold AND the assignee is non-empty (truly unassigned tasks have their own ``skipped_unassigned`` signal on the dispatcher and a different operator response). The signal is age-based on purpose: it's identity-agnostic, so it works for Hermes profiles, registered lanes, external workers, and typos uniformly. No registry to curate, no per-board allowlist. stranded_threshold_secondsrready claim_lockrrU>createdpromotedrrrrY)rLr z.1fr<mr rr r rozReassign to a different workercurrent_assigneernrzCheck dispatcher statusrzhermes kanban diagnosticsstranded_in_readyz Ready for z with no workerzThis task has been ready for z5 but nothing has claimed it. Common causes: assignee z is misspelled, the profile was deleted, or the external worker pool for this lane is down. Confirm the assignee is correct and that a worker is actually polling for it.r7) ready_since age_secondsrthreshold_secondsr) rrHrMrzrWr[rhr4rr/)rJr\rrrr#rrREADY_TRANSITION_KINDS last_ready_tsrQrZr"age_strr r3s r_rule_stranded_in_readyr'gs6 ,g66x ( (F  4&& 4,,2H >>   M22 r??4 4 4" A q11M KlAFFFK!LL   %K&&& d 4'....r)**--- '!+++ )A- - - 2'2   + ;<   G  37333 HG H H3; H H H #"({++ !$%6!7!7       rz list[RuleFn]_RULES)rrrrrrr r rrr)rrrrrrc|pi}t|dpi}|d|dtdd|vrd|vr |d|d<|S)aFBuild diagnostics config from the runtime ``kanban`` config section. ``kanban.diagnostics.failure_threshold`` remains an explicit override. Otherwise, derive the repeated-failure threshold from ``kanban.failure_limit`` so CLI/dashboard diagnostics match the dispatcher's actual circuit-breaker threshold. diagnosticsrrr)rrH setdefaultDEFAULT_CONFIG)rdiag_cfgs rconfig_from_kanban_configr. s!rJJNN=117R88H 7J(KLL 8++ %X 5 5(0(A$% Orc$|pi}t|tsiSi}|d}t|tr'|t |||d<dD]}||}||||<|S)a]Build diagnostics config from the full Hermes runtime config. Carries through ``kanban``, ``auxiliary``, and ``model`` keys so triage- aware rules can inspect the active aux-helper and main-model state. Folds the ``kanban`` block through ``config_from_kanban_config`` so the repeated-failure threshold derivation still applies. r)rrw)rGrrHupdater.)r~rrr|rs rconfig_from_runtime_configr1#s!rJ j$ ' ' C))J*d### ,Z88999"H %s##  CH Jr)rrrBrrr9c  t||ntj}|pi}it|}d|vr9d|vr5d|vr1t|dtd|d<g}t D]5} |||||||&#t$rY2wxYwdttD | fd|S)zRun every rule against a single task's state and return a severity-sorted list of active diagnostics. Sorting: critical first, then error, then warning; ties broken by most-recent ``last_seen_at``. Nrrrci|]\}}|| Sr-r-)r>iss r z,compute_task_diagnostics..]s???TQAq???rcP|jd |jpd fS)Nrr)rHr r6)d severity_idxs rrz*compute_task_diagnostics.._s/   aj" - - -n! " rr) r4timer,rrHr(rrF enumeratersort) rJr\rrrnow_tsrrqruler9s @rcompute_task_diagnosticsr?:s>TY[[ 9 9F \rF &^ &v &C6)) %V 3 3 v % %#0 JJ ' ' . /$ $   C  JJttD&$<< = = = =    H   @?Y~%>%>???LHH      Js>"B!! B.-B.r*Iterable[Diagnostic]cd}d}|D]@}|jtvrt|jnd}||kr |}|j}A|S)zlHighest severity present in the list, or None if empty. Useful for card badges that need a single color.rN)r rr)r* highest_idxhighestr8idxs rseverity_of_highestrEgsaKG !!23*2N2Nn""1:...TV   KjG Nr)r rrrrr)Nr')rr)rr4)r\r]rrrr^)r\r]rr4)rJrrkrrr2)rsrrr)r~rrr)rrrr)rrrLr4rr4)rr)rrrr)r~rrr) r\rBrrBrr9rrrr)r*r@rr)0r+ __future__r dataclassesrrtypingrrrr rOr:rrrr/rMrSrWr[rfrjrrrBr4rRuleFnr}rrrrrrrrrrr'r(r,DIAGNOSTIC_KINDSr,r.r1r?rEr-rrrKs:#"""""((((((((444444444444 2MMMM          D          F((((0    ---- 0    $    2 3S 49c48$z:JJ K&....,0000f....,,,,^mmmm`DwwwwtTTTTn////dJJJJZxxxx|        #*  ,8! ******Z      r