Enhance on demand: run a ComfyUI restore template from the apps, approve in-app, lands as a version #192

New Issue

2026-10-30T13:58:40Z

cameron commented

2026-10-30 13:58:40 +00:00

An Enhance action on every photo in FileViewer and Apollo. ImageApi submits a ComfyUI restore workflow for that photo and the app shows the result. You approve it from the app, and it lands in the library as a version (#191). Nothing reaches the library without an approval.

Depends on #191: approved files only appear in the apps once ImageApi ingests versions. The producer side is Apps/comfyui-imageapi-versions.

Verified against ComfyUI's HTTP API

Checked headlessly against a local ComfyUI 0.11.1 (frontend 1.37.11) with the node pack loaded:

Step	Call	Result
Reachable?	`GET /system_stats`	200
Submit	`POST /prompt` `{prompt: <API-format workflow>, client_id}`	`prompt_id`
Done?	`GET /history/{prompt_id}`	`status.completed`; the save node's outputs include `imageapi_versions: [{id, saved_path}]` and `images`
Preview	`GET /view?filename=…&type=temp`	the staged PNG
Approve	`POST /imageapi/approve` `{id}`	`{path}`: the file is written to the library by ComfyUI
Release VRAM	`POST /free` `{unload_models: true, free_memory: true}`	200

ComfyUI still does the writing, so ImageApi keeps its rule of never writing under library roots.

Decisions so far

The button is on every photo. It isn't limited to low-resolution or scanned photos.
A few named templates, e.g. "Restore", "Restore + colourise", "Upscale only".
Approval happens in the app. ImageApi always submits with require_approval = true.
Apps only talk to ImageApi. ComfyUI has no auth and stays LAN-only; ImageApi relays previews.
ComfyUI is not always running (it holds VRAM), so being unavailable is a normal state, not an error.

Templates

Each template is an API-format workflow file (ComfyUI → Workflow → Export (API)) with a name, label and description, configured alongside llm.toml (#190).
ImageApi finds the nodes it fills in by title, not node id, so re-exporting after editing the graph keeps working:
- imageapi:source (required): the Load Image From Path whose path gets the photo.
- imageapi:save (required): the Save Image Version; ImageApi forces require_approval = true.
- imageapi:prompt (optional): a text node whose text input gets the user's adjustment. The template's default prompt is used when the user leaves it empty. Exposed as accepts_prompt.
- imageapi:reference (optional): a second loader for a context or other-view photo. Exposed as accepts_reference.
Seeds: "randomize after generate" only exists in the ComfyUI page, not over the API. ImageApi must set every seed / noise_seed input to a fresh random value on each submit. Otherwise a retry returns ComfyUI's cached result.
Validate templates at startup and log any that are missing a required title.

Path mapping

ImageApi sees /media/nas/Media/Pictures/... and ComfyUI sees \\nas\Media\Pictures\.... Add a prefix map in config; the longest prefix wins and separators are converted. Use it in both directions: library → ComfyUI on submit, and ComfyUI → library to recognise the path approve returns.

Jobs

enhance_jobs
  id, original_hash, library_id, rel_path, template, prompt, reference_hash,
  comfy_prompt_id, status, error, pending (json: [{id, preview}]),
  saved (json: [paths]), created_at, updated_at
status: queued | running | ready | approved | failed | expired | unavailable

A background loop polls /history/{id} (and /queue for queue position) every few seconds, for active jobs only.
If ComfyUI is back up but has forgotten the prompt_id (it was restarted), the job is expired: unapproved results are cleared on restart, and the app offers "Run again".
After an approve, run a targeted ingest of the saved file so the version shows up immediately rather than after the next quick scan.

Endpoints

GET /enhance/templates returns [{name, label, description, accepts_prompt, accepts_reference}] plus available (a cached /system_stats probe with a short timeout).
POST /enhance with {path, template, prompt?, reference_path?} returns the job.
GET /enhance/jobs/{id} returns status, queue position, error, and the pending results.
GET /enhance/jobs?path= lists a photo's recent jobs, so the viewer can show an enhance in progress after reopening.
GET /enhance/jobs/{id}/preview/{n} relays the staged PNG from ComfyUI's /view.
POST /enhance/jobs/{id}/approve with {n} calls /imageapi/approve, triggers the ingest and returns the version.
"Retry with a tweak" is just a new POST /enhance; no endpoint is needed.

VRAM

ComfyUI keeps models loaded after a run. When ComfyUI has been idle (/queue empty, and the newest /history entry older than idle_free_secs, default 300), ImageApi calls POST /free once. The idle check uses ComfyUI's history, which includes your own interactive runs, so it never unloads in the middle of a session you're working in. That leaves the idle process holding only its CUDA context; it's worth measuring how much that is on the ComfyUI PC.

If even that's too much, the follow-up is starting ComfyUI on demand, e.g. llama-swap on the ComfyUI PC managing it with a ttl, or a small launcher. That's out of scope here.

Acceptance criteria

Templates load from config; one missing imageapi:source or imageapi:save is rejected with a clear log line.
POST /enhance submits the right file (with the path mapped) and a fresh seed; two submits of the same photo produce two different results.
A job moves through queued → running → ready, and its preview is served through ImageApi.
Approve writes the file via ComfyUI, the version appears in /image/versions right away, and approving again doesn't duplicate it.
With ComfyUI stopped, templates report available: false and POST /enhance returns a clear "ComfyUI isn't running" error; a job whose prompt_id was lost to a restart becomes expired.
After idle_free_secs with no activity (including runs from the ComfyUI page), /free is called once.
ComfyUI is never exposed to the apps directly.

Out of scope

Starting and stopping the ComfyUI process (see VRAM).
Batch enhancing many photos from the apps.
Client UI (sections added to FileViewer-React#52 and Apollo#55).

An **Enhance** action on every photo in FileViewer and Apollo. ImageApi submits a ComfyUI restore workflow for that photo and the app shows the result. You approve it from the app, and it lands in the library as a version (#191). Nothing reaches the library without an approval. Depends on **#191**: approved files only appear in the apps once ImageApi ingests versions. The producer side is `Apps/comfyui-imageapi-versions`. ## Verified against ComfyUI's HTTP API Checked headlessly against a local ComfyUI 0.11.1 (frontend 1.37.11) with the node pack loaded: | Step | Call | Result | |---|---|---| | Reachable? | `GET /system_stats` | 200 | | Submit | `POST /prompt` `{prompt: <API-format workflow>, client_id}` | `prompt_id` | | Done? | `GET /history/{prompt_id}` | `status.completed`; the save node's outputs include **`imageapi_versions: [{id, saved_path}]`** and `images` | | Preview | `GET /view?filename=…&type=temp` | the staged PNG | | Approve | `POST /imageapi/approve` `{id}` | `{path}`: the file is written to the library by ComfyUI | | Release VRAM | `POST /free` `{unload_models: true, free_memory: true}` | 200 | ComfyUI still does the writing, so ImageApi keeps its rule of never writing under library roots. ## Decisions so far - **The button is on every photo.** It isn't limited to low-resolution or scanned photos. - **A few named templates**, e.g. "Restore", "Restore + colourise", "Upscale only". - **Approval happens in the app.** ImageApi always submits with `require_approval = true`. - **Apps only talk to ImageApi.** ComfyUI has no auth and stays LAN-only; ImageApi relays previews. - **ComfyUI is not always running** (it holds VRAM), so being unavailable is a normal state, not an error. ## Templates - Each template is an API-format workflow file (ComfyUI → *Workflow → Export (API)*) with a name, label and description, configured alongside `llm.toml` (#190). - ImageApi finds the nodes it fills in **by title**, not node id, so re-exporting after editing the graph keeps working: - `imageapi:source` (required): the Load Image From Path whose `path` gets the photo. - `imageapi:save` (required): the Save Image Version; ImageApi forces `require_approval = true`. - `imageapi:prompt` (optional): a text node whose `text` input gets the user's adjustment. The template's default prompt is used when the user leaves it empty. Exposed as `accepts_prompt`. - `imageapi:reference` (optional): a second loader for a context or other-view photo. Exposed as `accepts_reference`. - **Seeds:** "randomize after generate" only exists in the ComfyUI page, not over the API. ImageApi must set **every `seed` / `noise_seed` input to a fresh random value on each submit**. Otherwise a retry returns ComfyUI's cached result. - Validate templates at startup and log any that are missing a required title. ## Path mapping ImageApi sees `/media/nas/Media/Pictures/...` and ComfyUI sees `\\nas\Media\Pictures\...`. Add a prefix map in config; the longest prefix wins and separators are converted. Use it in both directions: library → ComfyUI on submit, and ComfyUI → library to recognise the path `approve` returns. ## Jobs ``` enhance_jobs id, original_hash, library_id, rel_path, template, prompt, reference_hash, comfy_prompt_id, status, error, pending (json: [{id, preview}]), saved (json: [paths]), created_at, updated_at status: queued | running | ready | approved | failed | expired | unavailable ``` - A background loop polls `/history/{id}` (and `/queue` for queue position) every few seconds, for active jobs only. - If ComfyUI is back up but has forgotten the `prompt_id` (it was restarted), the job is **expired**: unapproved results are cleared on restart, and the app offers "Run again". - After an approve, run a targeted ingest of the saved file so the version shows up immediately rather than after the next quick scan. ## Endpoints - `GET /enhance/templates` returns `[{name, label, description, accepts_prompt, accepts_reference}]` plus `available` (a cached `/system_stats` probe with a short timeout). - `POST /enhance` with `{path, template, prompt?, reference_path?}` returns the job. - `GET /enhance/jobs/{id}` returns status, queue position, error, and the pending results. - `GET /enhance/jobs?path=` lists a photo's recent jobs, so the viewer can show an enhance in progress after reopening. - `GET /enhance/jobs/{id}/preview/{n}` relays the staged PNG from ComfyUI's `/view`. - `POST /enhance/jobs/{id}/approve` with `{n}` calls `/imageapi/approve`, triggers the ingest and returns the version. - "Retry with a tweak" is just a new `POST /enhance`; no endpoint is needed. ## VRAM ComfyUI keeps models loaded after a run. When ComfyUI has been idle (`/queue` empty, and the newest `/history` entry older than `idle_free_secs`, default 300), ImageApi calls `POST /free` once. The idle check uses ComfyUI's history, which includes your own interactive runs, so it never unloads in the middle of a session you're working in. That leaves the idle process holding only its CUDA context; it's worth measuring how much that is on the ComfyUI PC. If even that's too much, the follow-up is **starting ComfyUI on demand**, e.g. llama-swap on the ComfyUI PC managing it with a `ttl`, or a small launcher. That's out of scope here. ## Acceptance criteria - [ ] Templates load from config; one missing `imageapi:source` or `imageapi:save` is rejected with a clear log line. - [ ] `POST /enhance` submits the right file (with the path mapped) and a fresh seed; two submits of the same photo produce two different results. - [ ] A job moves through queued → running → ready, and its preview is served through ImageApi. - [ ] Approve writes the file via ComfyUI, the version appears in `/image/versions` right away, and approving again doesn't duplicate it. - [ ] With ComfyUI stopped, templates report `available: false` and `POST /enhance` returns a clear "ComfyUI isn't running" error; a job whose `prompt_id` was lost to a restart becomes `expired`. - [ ] After `idle_free_secs` with no activity (including runs from the ComfyUI page), `/free` is called once. - [ ] ComfyUI is never exposed to the apps directly. ## Out of scope - Starting and stopping the ComfyUI process (see VRAM). - Batch enhancing many photos from the apps. - Client UI (sections added to FileViewer-React#52 and Apollo#55).

cameron added the needs-triage label 2026-10-30 13:58:40 +00:00

cameron referenced this issue

2026-10-30 13:58:55 +00:00

Photo versions: link AI restorations/edits to their original, one photo with a global display choice #191

cameron commented

2026-10-30 14:06:04 +00:00

Template note: the imageapi:prompt node's text input isn't always named text (CLIP Text Encode); edit-model encoders often call it prompt (e.g. Qwen Image Edit's TextEncodeQwenImageEdit). Fill whichever of text / prompt the node has, and reject the template at startup if it has neither.

Template note: the `imageapi:prompt` node's text input isn't always named `text` (CLIP Text Encode); edit-model encoders often call it `prompt` (e.g. Qwen Image Edit's TextEncodeQwenImageEdit). Fill whichever of `text` / `prompt` the node has, and reject the template at startup if it has neither.

cameron commented

2026-10-30 14:28:49 +00:00

Refinements from the first real template (Fibo2 restore)

Prompt input names: fill the first of text / prompt / value on the imageapi:prompt node (a String Multiline primitive uses value).
Prompt semantics are append, not replace: the template's prompt is a full base instruction, so the app's adjustment is appended (e.g. \n\nAdditional instructions: …). An empty adjustment leaves the template's prompt as is. A per-template prompt_mode = "replace" can be added if a template ever wants replace.
A missing reference photo must never fall back to the export's leftover path. When the request has no reference_path, remove the imageapi:reference node and drop every input that links to it (["<id>", n]). If ComfyUI then rejects the prompt (node_errors), report "this template needs a reference photo". GET /enhance/templates can expose requires_reference once that has been probed at startup.
Seeds: randomise inputs named seed or noise_seed, or ending in .seed (e.g. TextGenerate's sampling_mode.seed).
Path map example: the ComfyUI host uses a mapped drive, /media/nas/ ↔ Z:\. Mapped drives only exist in the logged-in session; UNC (\\nas\…) is needed if ComfyUI ever runs as a service.
Stray output nodes: templates may still contain a stock SaveImage; every API run then also writes into ComfyUI's output folder. Log a warning for that at startup instead of editing the graph.

## Refinements from the first real template (Fibo2 restore) - **Prompt input names:** fill the first of `text` / `prompt` / `value` on the `imageapi:prompt` node (a String Multiline primitive uses `value`). - **Prompt semantics are append, not replace:** the template's prompt is a full base instruction, so the app's adjustment is appended (e.g. `\n\nAdditional instructions: …`). An empty adjustment leaves the template's prompt as is. A per-template `prompt_mode = "replace"` can be added if a template ever wants replace. - **A missing reference photo must never fall back to the export's leftover path.** When the request has no `reference_path`, remove the `imageapi:reference` node and drop every input that links to it (`["<id>", n]`). If ComfyUI then rejects the prompt (`node_errors`), report "this template needs a reference photo". `GET /enhance/templates` can expose `requires_reference` once that has been probed at startup. - **Seeds:** randomise inputs named `seed` or `noise_seed`, or ending in `.seed` (e.g. TextGenerate's `sampling_mode.seed`). - **Path map example:** the ComfyUI host uses a mapped drive, `/media/nas/` ↔ `Z:\`. Mapped drives only exist in the logged-in session; UNC (`\\nas\…`) is needed if ComfyUI ever runs as a service. - **Stray output nodes:** templates may still contain a stock `SaveImage`; every API run then also writes into ComfyUI's output folder. Log a warning for that at startup instead of editing the graph.

cameron commented

2026-10-30 14:43:54 +00:00

Template outcome (Fibo 2.0)

Tested in ComfyUI:

Restore (single image) works well with a short imperative instruction in Fibo's trained edit form, with the prompt rewrite off. The fuller wording beat the bare Increase the clarity and resolution.:

Increase the clarity and resolution of the image, and refine it to obtain sharp, finely detailed, natural textures as if shot on a modern mirrorless camera, keeping the subject, pose, composition, colours and lighting the same. Remove any phone interface overlays.
Restore with reference doesn't work with Fibo. Multi-reference in Fibo 2.0 is built for compositing (try-on, product integration, appearance transfer), so it blends the two photos instead of using the second only for detail. No reference template for now. Keep the generic imageapi:reference support for future templates on other models.
Appending the adjustment for Fibo templates: Fibo is trained on short imperative edit instructions (wrapped into VGL edit_instruction), so append the app's adjustment as another sentence, not under an "Additional instructions:" heading. Make the join text configurable per template (default: a single space).

## Template outcome (Fibo 2.0) Tested in ComfyUI: - **Restore (single image) works well** with a short imperative instruction in Fibo's trained edit form, with the prompt rewrite off. The fuller wording beat the bare `Increase the clarity and resolution.`: > Increase the clarity and resolution of the image, and refine it to obtain sharp, finely detailed, natural textures as if shot on a modern mirrorless camera, keeping the subject, pose, composition, colours and lighting the same. Remove any phone interface overlays. - **Restore with reference doesn't work with Fibo.** Multi-reference in Fibo 2.0 is built for compositing (try-on, product integration, appearance transfer), so it blends the two photos instead of using the second only for detail. No reference template for now. Keep the generic `imageapi:reference` support for future templates on other models. - **Appending the adjustment for Fibo templates:** Fibo is trained on short imperative edit instructions (wrapped into VGL `edit_instruction`), so append the app's adjustment as another sentence, not under an "Additional instructions:" heading. Make the join text configurable per template (default: a single space).

cameron commented

2026-10-30 15:13:32 +00:00

Dev vs Turbo over 8 varied archive photos

Tested on scanned prints, 200–480 px phone shots, a night portrait, indoor flash, a stadium wide shot and a backlit face, with the same seed per photo for both models.

Turbo (fibo2_edit_turbo_int8_convrot, 4 steps, CFG 1) takes 3.1–3.7 s; Dev (50 steps, CFG 5) takes 50–59 s. Quality is practically indistinguishable: same composition, same fixes, differences only in fine grain. Ship Turbo as the default "Restore" template. Dev isn't worth a separate template on this evidence.
Output is always about 1 MP, snapped to the model's size buckets (e.g. 200×120 → 1392×784, a slight aspect change; 805×1182 → 816×1200). For originals above ~1.2 MP, Restore lowers the resolution (2048×1536 → 1152×864). Options:
- GET /enhance/templates exposes an output-size hint, and the apps warn or grey out Enhance above ~1.5 MP;
- and/or the template appends an upscale-model step after VAE decode, so the output matches the original's size.
  The size hint is cheap and worth doing either way.
Strong gains on tiny, blurry and scanned photos (colour casts, grain, mush); small gains on decent 2–3 MP photos.
Re-lighting risk: on an under-exposed or backlit subject, the model re-lights the face instead of restoring it (invented skin and teeth detail), even though the prompt says to keep the lighting. The approval step covers this. Clients need to show the version clearly as "AI".
Versions can have a different size or aspect from the original. That matters for face-box overlays (Apollo#55 already hides them on mismatch) and for anything else that assumes the original's geometry.

## Dev vs Turbo over 8 varied archive photos Tested on scanned prints, 200–480 px phone shots, a night portrait, indoor flash, a stadium wide shot and a backlit face, with the same seed per photo for both models. - **Turbo (`fibo2_edit_turbo_int8_convrot`, 4 steps, CFG 1) takes 3.1–3.7 s; Dev (50 steps, CFG 5) takes 50–59 s.** Quality is practically indistinguishable: same composition, same fixes, differences only in fine grain. **Ship Turbo as the default "Restore" template.** Dev isn't worth a separate template on this evidence. - **Output is always about 1 MP**, snapped to the model's size buckets (e.g. 200×120 → 1392×784, a slight aspect change; 805×1182 → 816×1200). For originals above ~1.2 MP, Restore **lowers the resolution** (2048×1536 → 1152×864). Options: - `GET /enhance/templates` exposes an output-size hint, and the apps warn or grey out Enhance above ~1.5 MP; - and/or the template appends an upscale-model step after VAE decode, so the output matches the original's size. The size hint is cheap and worth doing either way. - **Strong gains** on tiny, blurry and scanned photos (colour casts, grain, mush); **small gains** on decent 2–3 MP photos. - **Re-lighting risk:** on an under-exposed or backlit subject, the model re-lights the face instead of restoring it (invented skin and teeth detail), even though the prompt says to keep the lighting. The approval step covers this. Clients need to show the version clearly as "AI". - Versions can have a different size or aspect from the original. That matters for face-box overlays (Apollo#55 already hides them on mismatch) and for anything else that assumes the original's geometry.

cameron commented

2026-10-30 18:50:27 +00:00