docs: ship private Audnexus chapter analysis

This commit is contained in:
drockhollaback
2026-08-29 16:19:19 -04:00
parent 79e6c9b059
commit af16e51ca7
7 changed files with 1079 additions and 11 deletions
+25
View File
@@ -2,6 +2,31 @@
All notable changes to this project are documented in this file.
## Unreleased
### Added
- Private-LAN Audnexus metadata and chapter retrieval from Discovery at
`192.168.86.168:3333`, with independent metadata/chapter approval.
- Exact-runtime publisher chapters, conservative silence-guided alignment, and
deterministic silence-only chapter candidates with detailed preview reasons.
- Bounded FFmpeg silence analysis across sorted multi-track input and a
deterministic container smoke fixture covering real detector integration.
### Security
- Plain HTTP Audnexus URLs accept only literal RFC1918 IPv4 or loopback
addresses. Public addresses, hostnames, credentials, paths, queries,
fragments, and redirects remain rejected.
- Remote cover downloads remain HTTPS-only and bounded despite private HTTP API
access.
### Operations
- Documented Discovery configuration, chapter thresholds, known limitations,
real-ASIN Enterprise-D acceptance, explicit approval, and absence of invented
percentage or ETA.
## 0.1.0 - 2026-08-26
### Added
+99 -11
View File
@@ -47,7 +47,8 @@ Enterprise-D must provide:
- SSH access with terminal allocation (`ssh -t`)
- a user that can read and write all four audiobook directories
- enough free space for the source, temporary merge, partial M4B, and final M4B
- outbound HTTPS only if optional Audnexus lookup is enabled
- LAN access to Discovery at `http://192.168.86.168:3333` when optional
Audnexus lookup is enabled
- either `linux/amd64` or `linux/arm64` for the default
`sandreas/m4b-tool:0.5.2` image
@@ -144,8 +145,11 @@ any option the planner can emit: `--jobs`, `--output-file`, `--name`,
`--artist`, `--writer`, `--series`, `--series-part`, `--year`,
`--description`, or `--cover`. It creates two synthetic one-second MP3 tracks
and a generated cover, exercises every optional metadata field, runs the real
conversion services, and checks the final duration and two chapter titles. It
does not touch host audiobook paths.
conversion services, and checks the final duration and two chapter titles. A
second deterministic tone/silence/tone fixture runs through the real bounded
silence detector and must report its known midpoint. Temporary media is removed
on every normal, failed, or interrupted exit. The smoke test does not touch host
audiobook paths.
## Launch over SSH
@@ -198,6 +202,16 @@ The directories are the queue database: `pending` means waiting,
For a directory book it must be the direct `chapters.txt` child; for a loose
MP3 it must be the `chapters.txt` sibling. The application derives this path
from the selected book and never accepts arbitrary chapter-file path text.
- **Audnexus exact:** publisher titles and offsets are selectable only when
local and remote runtimes differ by no more than the smaller of 1% or 120
seconds, and every imported offset fits the local audio.
- **Audnexus aligned:** when runtimes differ by no more than 5%, publisher
boundaries are projected onto the local timeline and snapped to nearby
detected silence. At least 90% of nonzero boundaries must match. Preview
shows alignment confidence plus predicted, snapped, and delta offsets.
- **Silence detection:** estimated boundaries come only from qualifying local
silences with minimum chapter spacing. Titles are generic (`Chapter 1`,
`Chapter 2`, and so on), never publisher titles.
The exact manual format is one nonblank chapter per line:
@@ -227,14 +241,35 @@ precedence over an accepted Audnexus cover.
## Audnexus
Audnexus is optional, unauthenticated enrichment. Network failure, malformed
responses, rate limits, and missing results fall back to local/manual metadata.
No API key is accepted. API responses and cover downloads are capped at 10 MiB,
returned ASINs and external text are validated, and only HTTPS cover URLs are
accepted. A remote cover is downloaded only after its Audnexus candidate is
accepted, into persistent work staging; validation or download failure simply
continues without that cover. If the public service becomes paid, credentialed,
or otherwise unsuitable, disable it in `config.toml`:
Audnexus is optional enrichment served by Discovery on private LAN HTTP:
```toml
[audnexus]
enabled = true
base_url = "http://192.168.86.168:3333"
region = "us"
connect_timeout = 3.0
read_timeout = 8.0
```
Book chapter requests use
`/books/{ASIN}/chapters?region=us`. Metadata choice and chapter choice are
independent: operator may keep local metadata while using chapters returned for
entered ASIN. Network failure, malformed responses, rate limits, wrong returned
ASINs, and missing results leave Audnexus modes unavailable while local chapter
modes remain usable. No API key is accepted by this client.
Private HTTP is deliberately narrow. URL host must be literal RFC1918 IPv4 or
literal loopback IPv4/IPv6. HTTP hostnames, public addresses, credentials,
non-root paths, queries, and fragments are rejected; redirects are not followed.
HTTPS remains valid for a normal syntactic host. This rule validates operator
configuration, not Discovery's availability.
API responses and cover downloads are capped at 10 MiB. Returned ASINs and
external text are validated. Cover handling remains HTTPS-only even when API
base URL uses private HTTP. Remote cover is downloaded only after its metadata
candidate is accepted, into persistent work staging; validation or download
failure continues without that cover. To disable enrichment:
```toml
[audnexus]
@@ -243,6 +278,59 @@ enabled = false
The rest of the workflow remains available without network access.
### Chapter-analysis configuration
```toml
[chapters]
exact_runtime_percent = 1.0
exact_runtime_max_seconds = 120
alignment_runtime_percent = 5.0
alignment_required_percent = 90.0
[chapters.silence]
threshold_db = -40
minimum_silence_seconds = 1.5
minimum_chapter_minutes = 5
snap_window_seconds = 45
analysis_timeout_seconds = 1800
max_output_bytes = 10485760
```
Analysis reads sorted tracks once and does not modify source media. Each chapter
candidate includes an availability reason. Selection opens detailed preview and
does not approve conversion; operator must still type `CONVERT` exactly after
reviewing paths, commands, timestamps, titles, and alignment diagnostics.
### Known limitations
- Runtime agreement cannot prove audio is same edition. Confirm title, runtime,
chapter count, and several boundaries before conversion.
- Silence detection estimates structural pauses. Long dramatic pauses may
become chapters; music or continuous narration may hide real boundaries.
- Audnexus-aligned titles are publisher data only after 90% configured match;
silence-only titles remain generic.
- Conversion exposes observable work, but no percentage or ETA can be derived
reliably from `m4b-tool`. No probe runs against growing output.
- Remote enrichment and silence failures fall back; neither failure moves source
data or creates ready output.
### Enterprise-D LAN acceptance
Before trusting remote chapters, replace placeholder with real ASIN from your
Audible library and test Discovery directly from Enterprise-D:
```sh
ASIN='REPLACE_WITH_REAL_ASIN'
curl --fail --show-error --silent \
"http://192.168.86.168:3333/books/${ASIN}/chapters?region=us"
```
Then launch TUI, enter same real ASIN, verify exact or aligned preview against
local audio, inspect silence-only preview, and cancel once to prove analysis
does not approve or move source. For disposable backed-up book, approve one
conversion and verify output chapter count, titles, playback, and pending to
ready-plus-done transition.
## Collisions and recovery
If the selected output name already exists in `ready`, choose another name or
@@ -0,0 +1,169 @@
# Conversion Progress Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Make long m4b-tool conversions visibly active through honest phase, elapsed-time, output-size, exit, and error records in both terminal and persistent log.
**Architecture:** Add one focused progress module that owns formatting and cadence. Converter emits immutable progress events through one callback and serializes all log writes on the conversion thread; workflow emits finalization phase without probing growing media or inventing percent/ETA.
**Tech Stack:** Python 3.12 standard library (`threading`, `time`, `pathlib`), pytest 8.4.2, existing `m4b-tool`/FFmpeg converter.
**Spec:** `docs/superpowers/specs/2026-08-29-audnexus-chapters-progress-design.md`
## Global Constraints
- Default update interval is five seconds and must be finite and positive.
- Never run ffprobe against a growing output.
- Never display percentage or ETA that m4b-tool does not provide.
- Report only observable phase, elapsed duration, output size, active state, exit status, and error.
- Every progress line goes to terminal callback and persistent log.
- Do not block child stdout consumption, race log writes, leak threads, or weaken process-group cancellation.
- Preserve all source, partial-output, work-directory, validation, and recovery invariants.
- Use strict TDD and do not push, merge, publish, or mutate servers.
---
### Task 1: Progress Events and Deterministic Formatting
**Files:**
- Create: `app/progress.py`
- Create: `tests/test_progress.py`
- Modify: `app/models.py`
- Modify: `app/config.py`
- Modify: `tests/test_config.py`
**Interfaces:**
- Produces `ProgressPhase(StrEnum)` values `CONVERTING`, `APPLYING_CHAPTERS`, `VALIDATING`, and `MOVING`.
- Produces immutable `ProgressEvent(phase: ProgressPhase, elapsed_seconds: float, active: bool = True, output_bytes: int | None = None, exit_status: int | None = None, error: str = "")`.
- Produces `format_progress(event: ProgressEvent) -> str`.
- Produces `Settings.progress_update_seconds: float = 5.0`.
- [ ] **Step 1: Add failing formatting tests**
Assert exact lines for zero, hour-plus elapsed time, IEC output sizes, phases without output, inactive success with `exit=0`, and escaped/single-line error text. Reject non-finite/negative elapsed and output values at event construction or formatting boundary.
- [ ] **Step 2: Verify missing module failure**
```sh
pytest -q tests/test_progress.py
```
- [ ] **Step 3: Implement immutable event and pure formatter**
Format elapsed as `HH:MM:SS`, sizes as `B`, `KiB`, `MiB`, or `GiB`, and normalize control/newline characters in errors so log framing cannot be forged.
- [ ] **Step 4: Add/implement configuration tests**
Parse `[progress].update_seconds`, default `5`, and reject boolean, zero, negative, NaN, and infinity with `progress.update_seconds must be positive`.
- [ ] **Step 5: Run focused/full tests and commit**
```sh
pytest -q tests/test_progress.py tests/test_config.py
pytest -q
git add app/progress.py app/models.py app/config.py tests/test_progress.py tests/test_config.py
git commit -m "feat: model honest conversion progress"
```
---
### Task 2: Converter Heartbeats and Phase Transitions
**Files:**
- Modify: `app/converter.py`
- Modify: `tests/test_converter.py`
**Interfaces:**
- Consumes Task 1 types.
- Extends `Converter.__init__` with `on_progress: Callable[[ProgressEvent], None] = lambda event: None`, `clock: Callable[[], float] = time.monotonic`, and injected wait/tick primitive suitable for deterministic tests.
- Keeps `Converter.run(plan, log_path) -> ValidationResult` unchanged.
- [ ] **Step 1: Add failing phase-order test**
Using fake clock/process, assert converting starts, periodic converting heartbeat includes `plan.merge_output.stat().st_size` when regular/non-symlink, applying begins after successful merge, validating begins after chapter application, and terminal events appear in order.
- [ ] **Step 2: Verify expected failure**
```sh
pytest -q tests/test_converter.py -k progress
```
- [ ] **Step 3: Implement nonblocking heartbeat design**
Use one daemon heartbeat thread only while a child runs. Thread computes events and enqueues them; conversion thread drains queue and alone calls terminal callback/writes log, preventing concurrent `TextIO` writes. Stop/join heartbeat in `finally` with a bounded join. Filesystem observation uses `lstat`; symlink/missing/error yields no size rather than failure.
- [ ] **Step 4: Add failing log-mirroring/cadence tests**
Assert each formatted progress event appears once in callback and once in log, interval honors injected clock, rapid child completion has start/final events without heartbeat, and no event occurs after run returns.
- [ ] **Step 5: Add failing exit/error/cancellation tests**
Assert nonzero child, output-handler exception, KeyboardInterrupt, SIGTERM success, and SIGKILL fallback produce final inactive event with status/error while retaining existing `ValidationResult` messages. Assert heartbeat thread joins in every path.
- [ ] **Step 6: Implement failure-finalization events minimally**
Do not change existing subprocess vectors, validation semantics, or cleanup paths. Sanitize error before formatting; retain full trusted internal error in `ValidationResult`.
- [ ] **Step 7: Run focused/full tests and commit**
```sh
pytest -q tests/test_converter.py tests/test_progress.py
pytest -q
git add app/converter.py tests/test_converter.py
git commit -m "feat: track long conversion phases"
```
---
### Task 3: TUI Wiring, Finalization Phase, Docs, and Runtime Gate
**Files:**
- Modify: `app/tui.py`
- Modify: `app/main.py`
- Modify: `tests/test_tui.py`
- Modify: `tests/test_main.py`
- Modify: `README.md`
- Modify: `CHANGELOG.md`
- Modify: `scripts/container-smoke.sh`
- Modify: `tests/test_packaging.py`
**Interfaces:**
- Consumes `ProgressEvent`/`format_progress` and updated Converter.
- Extends `UIAdapter` with `show_progress(event: ProgressEvent) -> None` or routes formatted lines through existing `stream` while tests assert one stable public behavior.
- Emits `MOVING` immediately before `finalize_success` and a final inactive completion/error event afterward.
- [ ] **Step 1: Add failing application wiring test**
Assert normal composition sends converter progress to TerminalUI, conversion log, and that moving phase occurs only after validation success and before finalizer. Assert finalization failure records moving-phase error and source behavior remains unchanged.
- [ ] **Step 2: Implement UI/workflow wiring**
Use plain non-markup terminal output. Do not duplicate converter-owned log lines at application layer. If moving events need same log, inject one scoped progress sink owning current log path and close it before result display.
- [ ] **Step 3: Add failing smoke/docs tests**
Assert README explains observable tracker fields, five-second default, lack of percent/ETA, no growing-file ffprobe, cancellation, and log location. Smoke harness captures at least converting/applying/validating/moving phase records with a short configurable interval without making runtime flaky.
- [ ] **Step 4: Implement smoke/docs/package updates**
Update changelog and audited release inputs consistently with chapter-analysis plan. Keep synthetic media short.
- [ ] **Step 5: Run all gates**
```sh
pytest -q
python -m compileall -q app tests
git diff --check
docker compose --env-file .env config --quiet
```
When Docker exists, rebuild and run container smoke. Otherwise record Enterprise-D runtime gate as pending.
- [ ] **Step 6: Commit and independent review**
```sh
git add app/tui.py app/main.py tests/test_tui.py tests/test_main.py README.md CHANGELOG.md scripts/container-smoke.sh tests/test_packaging.py config.toml
git commit -m "feat: show persistent conversion progress"
```
Dispatch whole-feature review against spec, fix findings with red tests, rerun every gate, and do not push or merge.
@@ -0,0 +1,368 @@
# Private Audnexus and Chapter Analysis Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Let audiobook-tui reach Discovery over narrowly permitted private HTTP and offer exact Audnexus, Audnexus-aligned, or silence-only chapters without weakening existing manual approval.
**Architecture:** Keep transport normalization in `app/audnexus.py`, audio-only silence extraction in a new `app/silence.py`, and deterministic candidate policy in a new `app/chapter_analysis.py`. TUI receives already validated candidates and passes the selected existing `Chapter` tuple into the unchanged planner/converter boundary.
**Tech Stack:** Python 3.12, `httpx==0.28.1`, FFmpeg `silencedetect`, pytest 8.4.2, Docker Compose.
**Spec:** `docs/superpowers/specs/2026-08-29-audnexus-chapters-progress-design.md`
## Global Constraints
- HTTPS remains required except HTTP to literal IPv4 RFC1918 or IPv4/IPv6 loopback addresses.
- Reject HTTP hostnames, public/link-local/multicast/unspecified addresses, credentials, query strings, fragments, and non-root paths.
- Redirects remain disabled and covers remain HTTPS-only.
- Audnexus responses remain bounded to 10 MiB and all network failure remains optional fallback.
- Exact runtime tolerance is `min(local * 1%, 120 seconds)`.
- Guided alignment eligibility is at most 5% runtime mismatch and selectability requires at least 90% nonzero-boundary silence matches.
- Silence defaults are `-40 dB`, `1.5 seconds`, five-minute chapter spacing, and a 45-second snap window.
- Analysis never mutates source media or moves workflow state.
- Final conversion approval remains mandatory.
- Use strict TDD: add one failing behavior test, observe expected failure, implement minimally, then run focused and full tests.
- Do not push, merge, publish, or mutate Enterprise-D/Discovery during implementation.
---
### Task 1: Configuration and Private-HTTP Boundary
**Files:**
- Modify: `app/models.py`
- Modify: `app/config.py`
- Modify: `config.toml`
- Modify: `tests/test_config.py`
**Interfaces:**
- Produces: `valid_audnexus_base_url(value: str) -> bool` in `app/models.py`.
- Produces Settings fields: `audnexus_region: str`, `chapter_exact_runtime_percent: float`, `chapter_exact_runtime_max_seconds: float`, `chapter_alignment_runtime_percent: float`, `chapter_alignment_required_percent: float`, `silence_threshold_db: float`, `silence_minimum_seconds: float`, `silence_minimum_chapter_minutes: float`, `silence_snap_window_seconds: float`, `silence_analysis_timeout_seconds: float`, and `silence_max_output_bytes: int`.
- Consumed by Tasks 2-5.
- [ ] **Step 1: Add failing URL and configuration tests**
Add parameterized cases proving HTTPS works, `http://192.168.86.168:3333`, `http://10.0.0.1`, `http://172.16.0.1`, `http://127.0.0.1`, and `http://[::1]` work, while `http://example.com`, `http://8.8.8.8`, link-local, multicast, unspecified, user-info, path, query, and fragment fail. Add a load-settings test for every exact default and for `region="us"`. Add aggregated validation tests for NaN/infinite/nonpositive timeouts and thresholds, percentages outside `(0, 100]`, and nonpositive output ceiling.
- [ ] **Step 2: Verify tests fail for missing fields/private HTTP support**
Run:
```sh
pytest -q tests/test_config.py
```
Expected: failures state private HTTP is rejected and new Settings attributes/config tables are absent.
- [ ] **Step 3: Implement exact URL predicate and settings**
Use `urllib.parse.urlparse` plus `ipaddress.ip_address`; do not resolve DNS. For HTTP, explicitly accept `address.is_loopback` or membership in `10/8`, `172.16/12`, or `192.168/16`, not broad `is_private`. Require empty username/password/query/fragment and path in `{"", "/"}`. Parse `[chapters]` and `[chapters.silence]` as separate tables. The conversion-progress plan owns `[progress]`.
- [ ] **Step 4: Update deployed configuration**
Set:
```toml
[audnexus]
enabled = true
base_url = "http://192.168.86.168:3333"
region = "us"
connect_timeout = 3.0
read_timeout = 8.0
```
Add all spec defaults without changing path or conversion settings.
- [ ] **Step 5: Run focused and full tests**
```sh
pytest -q tests/test_config.py tests/test_main.py
pytest -q
```
Expected: all pass.
- [ ] **Step 6: Commit**
```sh
git add app/models.py app/config.py config.toml tests/test_config.py
git commit -m "feat: allow private Audnexus configuration"
```
---
### Task 2: Audnexus Chapter Transport and Normalization
**Files:**
- Modify: `app/audnexus.py`
- Modify: `tests/test_audnexus.py`
**Interfaces:**
- Consumes: `Settings.audnexus_region` from Task 1 at composition call sites later.
- Produces immutable `AudnexusChapter(start_ms: int, title: str)`.
- Produces immutable `AudnexusChapters(asin: str, runtime_ms: int, is_accurate: bool, chapters: tuple[AudnexusChapter, ...])`.
- Produces `AudnexusClient.chapters(asin: str, *, region: str) -> AudnexusChapters | None`.
- [ ] **Step 1: Add failing documented-response test**
Use `httpx.MockTransport` to assert `GET /books/B08G9PRS1K/chapters?region=us`, no authorization header, and normalization of `runtimeLengthMs`, `isAccurate`, `startOffsetMs`, and titles.
- [ ] **Step 2: Verify missing method failure**
```sh
pytest -q tests/test_audnexus.py -k chapters
```
Expected: fail because `AudnexusClient.chapters` does not exist.
- [ ] **Step 3: Implement dataclasses and happy-path normalization**
Reuse `_bounded_body`, `_ASIN`, `_external_string`, and client timeouts. Require matching ASIN, positive integer runtime, boolean accuracy, nonempty list, integer nonnegative offsets, zero first offset, strict increase, offsets below runtime, and validated titles. Return `None` for invalid input or network/HTTP/JSON failure.
- [ ] **Step 4: Add adversarial failing tests one behavior at a time**
Cover mismatched/invalid ASIN, oversized body, missing/invalid runtime, empty/non-list chapters, bool-as-int offsets, negative/non-increasing/out-of-range offsets, empty/oversized/control-character titles, timeout, 404, and region propagation. Assert every failure returns `None` and no partial document.
- [ ] **Step 5: Implement minimal validation for each red case**
Keep book metadata normalization unchanged. Do not add authentication—the self-hosted server owns Audible credentials.
- [ ] **Step 6: Run focused and full tests**
```sh
pytest -q tests/test_audnexus.py
pytest -q
```
Expected: all pass.
- [ ] **Step 7: Commit**
```sh
git add app/audnexus.py tests/test_audnexus.py
git commit -m "feat: retrieve bounded Audnexus chapters"
```
---
### Task 3: Bounded FFmpeg Silence Extraction
**Files:**
- Create: `app/silence.py`
- Create: `tests/test_silence.py`
**Interfaces:**
- Consumes: `Book`, silence Settings values from Task 1.
- Produces immutable `Silence(start_ms: int, end_ms: int)` with `midpoint_ms` and `duration_ms` properties.
- Produces `SilenceError`.
- Produces `detect_silences(book: Book, settings: Settings, *, popen_factory: Callable[..., Any] = subprocess.Popen) -> tuple[Silence, ...]`.
- [ ] **Step 1: Add failing single-track command/parser test**
Fake `Popen` and assert explicit vector contains `ffmpeg`, `-nostdin`, input path, `silencedetect=noise=-40dB:d=1.5`, and null output. Feed paired `silence_start`/`silence_end` lines and assert millisecond rounding and midpoint.
- [ ] **Step 2: Verify missing module failure**
```sh
pytest -q tests/test_silence.py
```
Expected: import failure for `app.silence`.
- [ ] **Step 3: Implement one-track bounded parser**
Read merged stdout/stderr incrementally, count encoded bytes before retaining/parsing, enforce ceiling, use monotonic deadline, launch with `start_new_session=True`, and reuse converter-equivalent SIGTERM/10-second/SIGKILL cleanup semantics. Accept only finite paired events within probed track duration.
- [ ] **Step 4: Add failing multi-track timeline test**
Use two tracks and prove second-track silence offsets include cumulative first-track duration. Prove returned values sort and exact duplicate midpoint retains longest silence.
- [ ] **Step 5: Implement deterministic multi-track translation**
Analyze tracks separately in existing sorted order; add cumulative duration to every event; deduplicate only identical midpoints; exclude midpoint `<= 0` or `>= total duration`.
- [ ] **Step 6: Add and satisfy failure/cancellation tests**
Add one red-green cycle each for malformed numeric values, reversed/unmatched events, output overflow, timeout, nonzero exit, `KeyboardInterrupt`, SIGTERM success, and SIGKILL fallback. Assert no source writes and no lingering fake child.
- [ ] **Step 7: Run focused and full tests**
```sh
pytest -q tests/test_silence.py
pytest -q
```
Expected: all pass.
- [ ] **Step 8: Commit**
```sh
git add app/silence.py tests/test_silence.py
git commit -m "feat: detect bounded audiobook silences"
```
---
### Task 4: Exact, Aligned, and Silence-Only Candidate Policy
**Files:**
- Create: `app/chapter_analysis.py`
- Create: `tests/test_chapter_analysis.py`
- Modify: `app/models.py`
**Interfaces:**
- Consumes: `AudnexusChapters`, `Silence`, `Book`, and Settings fields from Tasks 1-3.
- Produces immutable `Alignment(boundary_index: int, predicted_ms: int, selected_ms: int, snapped: bool, delta_ms: int)`.
- Produces immutable `ChapterCandidate(mode: ChapterMode, available: bool, chapters: tuple[Chapter, ...] = (), reason: str = "", local_runtime_ms: int = 0, remote_runtime_ms: int | None = None, confidence_percent: float | None = None, alignments: tuple[Alignment, ...] = ())`.
- Produces `analyze_remote_chapters(book: Book, remote: AudnexusChapters | None, silences: tuple[Silence, ...], settings: Settings) -> tuple[ChapterCandidate, ChapterCandidate]` returning exact then aligned.
- Produces `silence_chapters(book: Book, silences: tuple[Silence, ...], settings: Settings) -> ChapterCandidate`.
- [ ] **Step 1: Add failing exact-match boundary tests**
Prove acceptance at `min(local*1%, 120s)`, rejection one millisecond beyond, preservation of titles/offsets, and rejection when a remote offset reaches local duration.
- [ ] **Step 2: Verify missing analysis failure, then implement exact candidate**
```sh
pytest -q tests/test_chapter_analysis.py -k exact
```
Expected before implementation: import/name failure. After minimal implementation: pass.
- [ ] **Step 3: Add failing guided-alignment tests**
Prove proportional projection, nearest midpoint inside 45 seconds, earlier tie, unused silence rule, unsnapped projected offsets, 5% eligibility edge, 90% confidence edge, zero boundary excluded from denominator, five-minute spacing rejection, strict order, and local-duration bound. Assert low confidence produces no selectable chapters or remote titles.
- [ ] **Step 4: Implement deterministic guided alignment**
Use integer millisecond arithmetic with one documented rounding rule (`round`). Match each nonzero projected boundary independently in order against unused silences; validate complete selected sequence after matching. Emit exact reasons containing mismatch/confidence values for TUI display.
- [ ] **Step 5: Add failing silence-only tests**
Prove chapter zero, earliest eligible midpoint selection, five-minute spacing, duplicate-midpoint longest preference, generic sequential titles, no-candidate unavailability, strict order, and end exclusion.
- [ ] **Step 6: Implement silence-only candidate**
Do not reuse Audnexus titles. Require at least one detected nonzero boundary so silence mode never advertises a guessed one-chapter result.
- [ ] **Step 7: Run focused and full tests**
```sh
pytest -q tests/test_chapter_analysis.py
pytest -q
```
Expected: all pass.
- [ ] **Step 8: Commit**
```sh
git add app/models.py app/chapter_analysis.py tests/test_chapter_analysis.py
git commit -m "feat: analyze exact aligned and silence chapters"
```
---
### Task 5: TUI Integration and Chapter Preview
**Files:**
- Modify: `app/tui.py`
- Modify: `app/main.py`
- Modify: `tests/test_tui.py`
- Modify: `tests/test_main.py`
**Interfaces:**
- Consumes all Task 1-4 interfaces.
- Extends `AudnexusService` with `chapters(asin: str, *, region: str) -> AudnexusChapters | None`.
- Extends `UIAdapter` with `choose_chapter_candidate(book: Book, candidates: tuple[ChapterCandidate, ...]) -> ChapterCandidate | ChapterMode | None` and candidate diagnostics in preview.
- Produces end-to-end selected `tuple[Chapter, ...]` for unchanged planner interface.
- [ ] **Step 1: Add failing metadata-independent remote chapter flow test**
Script UI to enter ASIN, retain local metadata, select exact Audnexus chapters, preview, approve, and assert planner receives remote chapters. Assert one metadata request and one chapter request.
- [ ] **Step 2: Verify test fails at missing protocol/mode integration**
```sh
pytest -q tests/test_tui.py -k audnexus_chapter
```
- [ ] **Step 3: Implement chapter-data caching and candidate composition**
Return entered normalized ASIN alongside metadata lookup result. Fetch chapters once. Run silence detection only when exact is unavailable or silence candidate is requested for display; cache result for one conversion preparation. Preserve existing exception-to-fallback behavior.
- [ ] **Step 4: Add failing menu/reason/preview tests**
Assert exact/aligned/silence availability labels, mismatch and confidence reasons, every chapter timestamp/title, alignment predicted/selected/delta fields, and estimated warning. Assert back/cancel performs no planner/converter/finalizer action.
- [ ] **Step 5: Implement terminal UI and builder routing**
Keep embedded, filename, and manual behavior. Candidate selection returns validated chapters directly; local modes continue through `_build_chapters`. Never auto-select best candidate.
- [ ] **Step 6: Add failure fallback tests**
Cover Audnexus 404/timeout/malformed chapters, silence timeout/error, >5% mismatch, <90% confidence, local metadata selection, and no ASIN. Existing local modes must remain selectable and no analysis failure may move source state.
- [ ] **Step 7: Run focused and full tests**
```sh
pytest -q tests/test_tui.py tests/test_main.py
pytest -q
```
Expected: all pass.
- [ ] **Step 8: Commit**
```sh
git add app/tui.py app/main.py tests/test_tui.py tests/test_main.py
git commit -m "feat: select Audnexus and detected chapters"
```
---
### Task 6: Documentation, Synthetic Smoke Analysis, and Release Gate
**Files:**
- Modify: `README.md`
- Modify: `CHANGELOG.md`
- Modify: `scripts/container-smoke.sh`
- Modify: `tests/test_packaging.py`
- Modify: any audited packaging manifest/version file discovered by existing package tests
**Interfaces:**
- Consumes completed Tasks 1-5.
- Produces operator documentation and release-ready container checks.
- [ ] **Step 1: Add failing documentation/package assertions**
Assert README includes Discovery URL, exact/aligned/silence semantics, configuration keys, private-HTTP restriction, cover HTTPS rule, known limitations, and Enterprise-D acceptance commands. Assert packaged config contains Discovery URL and new tables.
- [ ] **Step 2: Add failing synthetic-silence smoke fixture**
Generate deterministic tone/silence/tone media with FFmpeg, call real silence detector, and assert expected midpoint within a narrow tolerance before existing conversion validation. Keep smoke runtime short and source under `mktemp`.
- [ ] **Step 3: Implement docs and smoke updates**
Document no fake ETA, no automatic approval, alignment confidence, generic silence titles, and real-ASIN LAN acceptance. Update changelog and release/package audit inputs consistently.
- [ ] **Step 4: Run all local gates**
```sh
pytest -q
python -m compileall -q app tests
git diff --check
docker compose --env-file .env config --quiet
```
Run existing packaging/static commands documented in README. If Docker is available, rebuild without cache and run `scripts/container-smoke.sh`; otherwise record Docker runtime gate as pending for Enterprise-D.
- [ ] **Step 5: Commit**
```sh
git add README.md CHANGELOG.md scripts/container-smoke.sh tests/test_packaging.py config.toml
git commit -m "docs: ship private Audnexus chapter analysis"
```
- [ ] **Step 6: Final independent review**
Dispatch whole-branch spec and security review. Fix every load-bearing finding with a failing regression first, rerun all gates, and re-review. Do not push or merge.
@@ -0,0 +1,319 @@
# Private Audnexus, Chapter Analysis, and Conversion Progress Design
## Purpose
Extend audiobook-tui so Enterprise-D can use the self-hosted Audnexus service
at `http://192.168.86.168:3333`, import publisher chapter data when it matches
the local audio, align remote chapter data to local silence when runtimes differ
slightly, generate conservative silence-only chapters as a fallback, and show
honest progress during long conversions.
The workflow remains keyboard-only over SSH. Analysis never modifies source
media, and conversion still requires explicit final approval.
## Scope
This change adds:
- narrowly scoped private-network HTTP validation for the Audnexus base URL;
- an Audnexus chapter endpoint client and strict response normalization;
- exact-runtime Audnexus chapters;
- conservative Audnexus-guided silence alignment;
- silence-only chapter detection with generic titles;
- chapter-source availability reasons and detailed preview data;
- phase, elapsed-time, process-activity, and output-size tracking during
conversion;
- configuration, documentation, unit tests, integration tests, and a synthetic
container smoke case for the new behavior.
This change does not add a desktop GUI, modify source audio during analysis,
infer publisher chapter titles from silence alone, follow HTTP redirects, or
automatically approve a conversion.
## Security Boundary for Audnexus URLs
An Audnexus base URL is valid when all these rules hold:
- `https` may address a syntactically valid host.
- `http` may address only a literal IPv4 or IPv6 loopback address or a literal
IPv4 RFC1918 address in `10.0.0.0/8`, `172.16.0.0/12`, or
`192.168.0.0/16`.
- HTTP hostnames are rejected, even when they currently resolve to a private
address. This avoids DNS rebinding and resolution-time ambiguity.
- Public, link-local, multicast, and unspecified HTTP addresses are rejected.
- User information, query strings, and fragments are rejected.
- A non-root path is rejected so endpoint construction cannot escape or double
an operator-supplied path prefix.
- Redirect following remains disabled.
The deployed configuration uses:
```toml
[audnexus]
enabled = true
base_url = "http://192.168.86.168:3333"
region = "us"
```
Cover URLs remain HTTPS-only and retain existing bounded streaming, media-type,
signature, staging, and no-overwrite checks.
## Domain Model
`ChapterMode` gains three values:
- `AUDNEXUS_EXACT`
- `AUDNEXUS_ALIGNED`
- `SILENCE`
Audnexus normalization produces a chapter document containing requested ASIN,
remote runtime, accuracy flag, and ordered remote chapters. Analysis produces a
chapter candidate containing mode, chapters, availability, reason, local and
remote runtimes when applicable, confidence percentage when applicable, and
per-boundary alignment diagnostics when applicable.
The selected candidate is converted to the existing immutable `Chapter` tuple
before planning. Planner and converter continue to consume one validated
chapter sequence without knowing how it was produced.
## Audnexus Chapter Retrieval
For a valid ten-character ASIN, the client requests:
```text
GET /books/{ASIN}/chapters?region={configured-region}
```
The chapter response uses the existing connection/read timeouts and 10 MiB
response ceiling. Failure returns an unavailable candidate rather than blocking
local chapter modes.
Normalization requires:
- a response object;
- returned ASIN equal to the requested ASIN;
- finite, positive remote runtime;
- a nonempty chapter array;
- finite, nonnegative integer millisecond offsets;
- first offset equal to zero;
- strictly increasing offsets;
- every offset less than remote runtime;
- nonempty titles no longer than 10,000 Unicode characters;
- no C0, DEL, or C1 control characters in titles.
Extra response fields are ignored. Malformed or oversized data is rejected as
one remote failure. Metadata acceptance and chapter acceptance remain
independent: an entered ASIN may supply chapters while the operator retains
local metadata.
## Exact Runtime Matching
Let `L` be scanned local duration in milliseconds and `R` be reported remote
duration. Exact chapters are available when:
```text
abs(L - R) <= min(L * 0.01, 120 seconds)
```
Remote offsets and titles are used unchanged after verifying every offset is
less than `L`. This test intentionally uses the smaller tolerance. It prevents
the absolute allowance from becoming too loose for short books and the
percentage allowance from becoming too loose for long books.
## Silence Analysis
Silence analysis runs FFmpeg against sorted local tracks without modifying
them. Multi-file input is analyzed in track order with cumulative offsets so
results share the same local timeline used by conversion. The subprocess uses
an explicit argument vector, bounded captured output, a configured timeout,
and the existing process-group termination pattern for interruption.
Default configuration:
```toml
[chapters]
exact_runtime_percent = 1.0
exact_runtime_max_seconds = 120
alignment_runtime_percent = 5.0
alignment_required_percent = 90.0
[chapters.silence]
threshold_db = -40
minimum_silence_seconds = 1.5
minimum_chapter_minutes = 5
snap_window_seconds = 45
analysis_timeout_seconds = 1800
max_output_bytes = 10485760
```
FFmpeg silence events must form valid start/end pairs. A candidate boundary is
the midpoint of a qualifying silence. Events before zero, beyond the track,
non-finite values, reversed pairs, malformed lines, unmatched pairs, output
beyond the configured ceiling, timeout, cancellation, and nonzero process exit
make silence analysis unavailable with a concise reason.
Candidates are translated by cumulative track duration, sorted, deduplicated,
and filtered so boundaries are not placed at zero or at/after total local
duration.
## Audnexus-Guided Alignment
Guided alignment is considered only when exact matching is unavailable and:
```text
abs(L - R) / R <= 0.05
```
Each remote boundary after the first is projected onto the local timeline:
```text
predicted_local_offset = remote_offset * L / R
```
The projected boundary snaps to the nearest unused silence midpoint within 45
seconds. Ties resolve to the earlier midpoint. A boundary with no qualifying
silence remains at its projected offset and is marked unsnapped. Snapped and
unsnapped boundaries must remain strictly increasing, respect the five-minute
minimum chapter spacing, and precede local duration. A candidate that violates
those rules is unavailable even when its raw match percentage would pass. The
first chapter remains at zero and does not require a silence match.
Alignment confidence is:
```text
matched nonzero remote boundaries / total nonzero remote boundaries * 100
```
Aligned chapters are available only at confidence of at least 90%. When that
threshold passes, remote titles are retained for both snapped and projected
boundaries, and preview identifies every unsnapped boundary. Otherwise the
candidate is unavailable; the application must not attach remote titles to a
partial or low-confidence alignment.
## Silence-Only Chapters
Silence-only mode remains independent of Audnexus availability and runtime.
It sorts qualifying midpoint candidates by time, then greedily selects the
earliest candidate at least five minutes after the previously selected
boundary. Exact duplicate midpoints retain the longest silence. This makes the
result deterministic without allowing silence duration to pull a boundary far
away from the earliest eligible transition.
The first chapter always starts at zero. Subsequent titles are generated in
sequence:
```text
Chapter 1
Chapter 2
...
```
The UI labels these as estimated chapters and never presents them as publisher
chapter data. An empty qualifying set leaves silence mode unavailable rather
than creating a misleading one-chapter result.
## TUI Flow
After track scan and optional ASIN lookup, the application analyzes applicable
chapter sources. Existing embedded, filename, and manual modes remain
available under their current rules.
The chapter menu reports usable candidates and concise unavailable reasons,
including runtime difference and alignment confidence. Example:
```text
Audnexus exact unavailable: runtime differs by 3.4%
Audnexus aligned unavailable: 76% of boundaries matched silence
Silence detection: 31 estimated chapters
```
Selection preview includes:
- chapter method;
- local and remote runtimes when applicable;
- match confidence when applicable;
- every chapter timestamp and title;
- predicted, snapped, and delta offsets for aligned boundaries;
- existing paths and escaped command vectors.
The operator may go back and choose another source. Final conversion approval
remains exact and mandatory.
## Conversion Progress
`m4b-tool` suppresses useful FFmpeg percentage output, so the application must
not invent a percentage or ETA. A tracker reports observable state every five
seconds by default:
```text
[Converting audio] elapsed 00:37:12 | output 94.3 MiB | active
[Applying chapters] elapsed 00:00:08
[Validating output] elapsed 00:00:03
[Moving completed files]
```
Phases are converting audio, applying chapters, validating output, and moving
completed files. During conversion, output size is read with filesystem
metadata only. The application never runs `ffprobe` against a growing output.
Each update goes to both terminal callback and persistent conversion log.
Tracker implementation must not block subprocess output consumption, leak a
thread after completion, race log writes, or weaken Ctrl+C behavior. Final
phase record includes elapsed time, child exit status, and error when present.
Existing source, partial-output, work-directory, and recovery invariants remain
unchanged.
Configuration:
```toml
[progress]
update_seconds = 5
```
The interval must be finite and positive.
## Error Handling
- Audnexus metadata or chapter failures fall back to local choices.
- Silence-analysis failures fall back to non-silence modes.
- Exact mismatch is informational, not fatal.
- Guided mismatch beyond 5% is unavailable without running alignment.
- Low-confidence alignment never produces a selectable candidate.
- Invalid configuration blocks startup with deterministic aggregated errors.
- Conversion failure leaves source pending and preserves existing controlled
recovery behavior.
- No analysis failure moves source data or creates a ready output.
## Testing
Tests follow red-green-refactor and cover:
- HTTPS and private HTTP acceptance plus public, hostname, credential,
path/query/fragment, link-local, multicast, and unspecified rejection;
- IPv4 and IPv6 loopback cases;
- Audnexus chapter request path, region, timeout, body bound, ASIN matching,
schema, title, and offset validation;
- exact-match tolerance boundaries;
- FFmpeg silence command construction, parsing, multi-track offset translation,
malformed/oversized output, timeout, cancellation, and exit failure;
- deterministic silence filtering and generic titles;
- guided projection, nearest snapping, tie behavior, spacing, confidence, 5%
eligibility, and wrong-edition rejection;
- TUI candidate display, unavailable reasons, selection, preview, backtracking,
and final approval;
- progress cadence, observable fields, terminal/log mirroring, child exit,
cancellation, and cleanup;
- synthetic container smoke audio containing known silences;
- regression coverage for embedded, filenames, manual chapters, metadata,
covers, planning, conversion validation, workflow transitions, and packaging.
Runtime acceptance on Enterprise-D must verify a real ASIN against Discovery,
an exact or aligned chapter import, a silence-only preview, visible progress
during a nontrivial conversion, final chapter count/titles, and playable output.
## Delivery
Implementation stays on `feat/audiobook-tui`, is committed in reviewable tasks,
and is not pushed or merged without explicit approval. README and example
configuration document Discovery at `192.168.86.168:3333`, safe-match rules,
alignment confidence, silence limitations, progress semantics, and recovery.
+43
View File
@@ -53,6 +53,13 @@ ffmpeg -nostdin -hide_banner -loglevel error \
-f lavfi -i 'color=c=navy:s=64x64:d=0.1' \
-frames:v 1 -q:v 2 \
"$smoke_root/pending/Synthetic Book/cover.jpg"
ffmpeg -nostdin -hide_banner -loglevel error \
-f lavfi -i 'sine=frequency=440:duration=0.4' \
-f lavfi -i 'anullsrc=r=44100:cl=mono:d=0.8' \
-f lavfi -i 'sine=frequency=880:duration=0.4' \
-filter_complex '[0:a][1:a][2:a]concat=n=3:v=0:a=1[out]' \
-map '[out]' -q:a 7 \
"$smoke_root/silence-fixture.mp3"
python3 - <<'PY'
import os
@@ -108,6 +115,42 @@ if not (roots.done / source.name).is_dir():
raise SystemExit("source was not moved to done")
PY
python3 - <<'PY'
import os
from pathlib import Path
from app.models import Book, Roots, Settings, Track
from app.silence import detect_silences
root = Path(os.environ["AUDIOBOOK_SMOKE_ROOT"])
roots = Roots(
pending=root / "pending",
ready=root / "ready",
done=root / "done",
review=root / "review",
)
fixture = root / "silence-fixture.mp3"
book = Book(source=fixture, tracks=(Track(path=fixture, duration_ms=1600),))
settings = Settings(
roots=roots,
state_dir=root / "state",
work_dir=root / "state" / "work",
jobs=1,
audnexus_enabled=False,
silence_threshold_db=-35,
silence_minimum_seconds=0.4,
silence_analysis_timeout_seconds=30,
silence_max_output_bytes=1_048_576,
)
silences = detect_silences(book, settings)
if len(silences) != 1:
raise SystemExit(f"silence smoke expected one interval; found {silences!r}")
midpoint_ms = silences[0].midpoint_ms
if abs(midpoint_ms - 800) > 150:
raise SystemExit(f"silence smoke midpoint outside tolerance: {midpoint_ms}ms")
print("silence detector smoke passed")
PY
ffprobe -v error \
-show_entries \
'format=duration:chapter_tags=title:stream=codec_name,codec_tag_string,codec_type:stream_disposition=attached_pic' \
+56
View File
@@ -111,6 +111,13 @@ def _run_container_smoke_with_probe_payload(
" printf '%s\\n' 'ffmpeg fake'\n"
" exit 0\n"
"fi\n"
"case \"$*\" in\n"
" *silencedetect=*)\n"
" printf '%s\\n' '[silencedetect] silence_start: 0.400'\n"
" printf '%s\\n' '[silencedetect] silence_end: 1.200'\n"
" exit 0\n"
" ;;\n"
"esac\n"
"for output do :; done\n"
": > \"$output\"\n",
encoding="utf-8",
@@ -191,6 +198,54 @@ def test_config_defaults_match_container_roots() -> None:
assert root in text
def test_packaged_config_enables_discovery_chapter_analysis() -> None:
config = tomllib.loads(Path("config.toml").read_text(encoding="utf-8"))
assert config["audnexus"] == {
"enabled": True,
"base_url": "http://192.168.86.168:3333",
"region": "us",
"connect_timeout": 3.0,
"read_timeout": 8.0,
}
assert config["chapters"] == {
"exact_runtime_percent": 1.0,
"exact_runtime_max_seconds": 120,
"alignment_runtime_percent": 5.0,
"alignment_required_percent": 90.0,
"silence": {
"threshold_db": -40,
"minimum_silence_seconds": 1.5,
"minimum_chapter_minutes": 5,
"snap_window_seconds": 45,
"analysis_timeout_seconds": 1800,
"max_output_bytes": 10_485_760,
},
}
def test_operator_docs_cover_private_audnexus_chapter_analysis() -> None:
readme = Path("README.md").read_text(encoding="utf-8")
required_operator_guidance = (
"http://192.168.86.168:3333",
"Audnexus exact",
"Audnexus aligned",
"Silence detection",
"exact_runtime_percent",
"alignment_required_percent",
"minimum_silence_seconds",
"literal RFC1918",
"HTTPS-only",
"no percentage or ETA",
"does not approve",
"real ASIN",
"/books/{ASIN}/chapters?region=us",
)
for guidance in required_operator_guidance:
assert guidance in readme
def test_work_directory_is_persistent_in_the_state_mount() -> None:
config = tomllib.loads(Path("config.toml").read_text(encoding="utf-8"))
service = _service()
@@ -430,6 +485,7 @@ def test_container_smoke_accepts_expected_final_stream_layout(
)
assert result.returncode == 0, result.stderr
assert "silence detector smoke passed" in result.stdout
assert "container smoke test passed" in result.stdout