{'>'}_
English 繁體中文 日本語 한국어 Deutsch Español Français
Login

A Claude Code skill for detecting flaky test patterns that cause intermittent CI failures.

0 stars
0 forks
TypeScript
5 views
View on GitHub Add to Favorites

SKILL.md

Flaky Test Detector Skill

A Claude Code skill for detecting flaky test patterns that cause intermittent CI failures.

Trigger Phrases

This skill activates when you mention:

  • "flaky test"
  • "intermittent failure"
  • "test sometimes fails"
  • "race condition in test"
  • "timing issue in test"
  • "test passes locally but fails in CI"
  • "non-deterministic test"

Capabilities

Pattern Detection

Scans *.test.ts and *.spec.ts files to detect:

Severity Pattern Risk
HIGH ttl.*:\s*1\b 1-second TTL causes second-boundary race conditions
HIGH Math.floor(Date.now() / 1000) Second truncation creates timing edge cases
HIGH setTimeout([^,]+,\s*\d{1,2}) Very short timeouts (<100ms) are unreliable
MEDIUM Date.now() without mocking Real time in tests causes non-determinism
MEDIUM timeout.*:\s*[1-9]0?\b Low timeout values cause race conditions
MEDIUM new Date() without mocking Real date in tests causes flakiness
LOW Math.random() without seeding Random values cause non-determinism
LOW process.nextTick in assertions Timing-dependent assertions

Context Extraction

For each detected pattern, the skill extracts:

  • File path and line number
  • Surrounding code context (3 lines before/after)
  • Pattern severity classification
  • Specific risk explanation

Fix Suggestions

Generates actionable fix suggestions:

  • vi.useFakeTimers() for time-based flakiness
  • TTL increase recommendations
  • Mock injection patterns
  • Deterministic alternatives

Usage

Scan Current Directory

# Run the detector on all test files
npx tsx scripts/index.ts

# Scan specific directory
npx tsx scripts/index.ts ./src/tests

# Output as JSON
npx tsx scripts/index.ts --json

Integration with CI

Add to your CI pipeline to catch flaky patterns before merge:

- name: Check for flaky test patterns
  run: npx tsx scripts/index.ts --ci

README

Flaky Test Detector

A Claude Code skill for detecting flaky test patterns that cause intermittent CI failures.

Installation

As a Claude Code Skill

# Clone to your Claude skills directory
git clone https://github.com/wrsmith108/claude-skill-flaky-test-detector.git ~/.claude/skills/flaky-test-detector

Standalone Usage

# Run directly with npx
npx tsx ~/.claude/skills/flaky-test-detector/scripts/index.ts [directory] [options]

Trigger Phrases

This skill activates when you mention:

  • "flaky test"
  • "intermittent failure"
  • "test sometimes fails"
  • "race condition in test"
  • "timing issue in test"
  • "test passes locally but fails in CI"
  • "non-deterministic test"

Pattern Detection

Scans *.test.ts and *.spec.ts files to detect:

Severity Pattern Risk
HIGH 1-second TTL Second-boundary race conditions
HIGH Math.floor(Date.now() / 1000) Second truncation timing edge cases
HIGH Very short setTimeout (<100ms) Unreliable timeouts
MEDIUM Date.now() without mocking Real time causes non-determinism
MEDIUM new Date() without mocking Real date causes flakiness
MEDIUM Low timeout values Race conditions in CI
LOW Math.random() without seeding Non-reproducible behavior
LOW process.nextTick in assertions Timing-dependent assertions

Usage

Scan Current Directory

# Run the detector on all test files
npx tsx ~/.claude/skills/flaky-test-detector/scripts/index.ts

# Scan specific directory
npx tsx ~/.claude/skills/flaky-test-detector/scripts/index.ts ./src/tests

# Output as JSON
npx tsx ~/.claude/skills/flaky-test-detector/scripts/index.ts --json

# CI mode: exit with code 1 if HIGH severity issues found
npx tsx ~/.claude/skills/flaky-test-detector/scripts/index.ts --ci

Options

Option Description
--json Output results as JSON
--ci CI mode: exit with code 1 if HIGH severity issues found
--quiet Suppress output except errors
--help Show help message

Output Example

## Flaky Test Analysis

Scanned 94 test files, found 15 potential flaky patterns.

### High Risk
**cache.test.ts:119**
- Pattern: `ttlSeconds: 1`
- Risk: Second boundary crossing causes race condition
- Fix: Use vi.useFakeTimers() or increase TTL to 5+

### Medium Risk
**utils.spec.ts:45**
- Pattern: `Date.now()`
- Risk: Real time creates non-determinism
- Fix: Mock with vi.useFakeTimers() and vi.setSystemTime()

CI Integration

Add to your CI pipeline to catch flaky patterns before merge:

- name: Check for flaky test patterns
  run: npx tsx ~/.claude/skills/flaky-test-detector/scripts/index.ts --ci

Fix Suggestions

The skill provides actionable fix suggestions:

  • vi.useFakeTimers() for time-based flakiness
  • TTL increase recommendations
  • Mock injection patterns
  • Deterministic alternatives

Requirements

  • Node.js 18+
  • TypeScript (tsx for execution)

License

MIT

Related Skills

References