{ "cells": [ { "cell_type": "markdown", "id": "7a80b74380ba", "metadata": {}, "source": [ "# Cohort screenshot factory\n", "\n", "This notebook is the capstone of the example suite: it turns the\n", "Workbench into a scriptable visualisation service.\n", "\n", "We iterate over a small synthetic cohort and, for every case and every\n", "label, drive the Workbench to produce a labelled screenshot. The end\n", "result is a Markdown report you could ship as a per-cohort QA artifact.\n", "\n", "What this exercises:\n", "\n", "- Generating an in-memory cohort with `phantoms.make_cohort`\n", "- Applying a reusable layout document via `set_layout(Path(...))`\n", "- Computing per-label centroids and extents from a `MultiLabelSegmentation`\n", "- Driving the global crosshair (`wb.set_position`) and per-window cameras\n", " (`RenderWindow.set_camera(parallel_scale=...)`) from those measurements\n", "- Capturing editor-level screenshots and assembling a Markdown report\n", "\n", "**Prerequisites**\n", "\n", "- A running MITK Workbench at `http://localhost:8080`.\n", "- The **MxN multi-widget editor** open in the workbench before you\n", " run the notebook (*Window > New Editor > MxN*).\n", "- The optional `mitk` Python package (provides `mitk.mxn.layout`). The\n", " package ships with a local MITK build's Python wrapping; it is not\n", " available from public PyPI.\n", "- Install the library with the layout extra:\n", " `pip install -e \".[layout]\"` from the repo root.\n" ] }, { "cell_type": "markdown", "id": "1415179af5b4", "metadata": {}, "source": [ "## 1. Connect and import\n" ] }, { "cell_type": "code", "execution_count": 8, "id": "11b89a275615", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Connected to MITK Workbench REST API.\n" ] } ], "source": [ "import tempfile\n", "from pathlib import Path\n", "\n", "import numpy as np\n", "\n", "import mitk_workbench_remote as mw\n", "from mitk_workbench_remote.examples import phantoms\n", "\n", "wb = mw.connect(\"http://localhost:8080\")\n", "if not wb.ping():\n", " raise SystemExit(\"Workbench unreachable at http://localhost:8080. Start it and re-run.\")\n", "\n", "try:\n", " from mitk.mxn.layout import (\n", " LayoutWindow,\n", " MxNLayoutDocument,\n", " Split,\n", " ViewDirection,\n", " )\n", "except ImportError as exc:\n", " raise SystemExit(\n", " \"This notebook requires the optional 'mitk' package. \"\n", " \"Build it from a local MITK CMake build (Wrapping/Python) and install the wheel.\"\n", " ) from exc\n", "\n", "print(f\"Connected to {wb.info.name}.\")\n", "\n", "if not wb.mxn.get_info().active:\n", " raise SystemExit(\n", " \"The MxN multi-widget editor is not open in the workbench. \"\n", " \"Open it via 'Window > New Editor > MxN' and re-run.\"\n", " )\n" ] }, { "cell_type": "markdown", "id": "66b7ad10a7b7", "metadata": {}, "source": [ "## 2. Generate the cohort\n", "\n", "`phantoms.make_cohort` builds `n` synthetic cases in memory. Each case\n", "carries an MR-like image and a single-group `MultiLabelSegmentation`\n", "with a few labels (tumor, edema, ...). Tumor location and label count\n", "vary per case so the screenshots look meaningfully different.\n" ] }, { "cell_type": "code", "execution_count": 9, "id": "7632bbb1542a", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "case_001: shape=(32, 96, 96), labels=3\n", "case_002: shape=(32, 96, 96), labels=4\n", "case_003: shape=(32, 96, 96), labels=3\n" ] } ], "source": [ "cases = phantoms.make_cohort(n=3, seed=0, modality=\"mr\")\n", "for case in cases:\n", " n_labels = len(case.segmentation.labels)\n", " print(f\"{case.case_id}: shape={case.image.shape}, labels={n_labels}\")\n" ] }, { "cell_type": "markdown", "id": "40d300f71611", "metadata": {}, "source": [ "## 3. Define the layout once -- reuse it for every case\n", "\n", "The layout is a piece of *configuration*, not per-case state. We build\n", "one `MxNLayoutDocument` here and push it to the workbench in section 5\n", "once -- not per case.\n", "\n", "Two rows of three cells each:\n", "\n", "- **overview row** (`mxn__overview_*`): full image + segmentation,\n", " reinitialised per case so the camera frames the whole volume.\n", "- **zoomed row** (`mxn__zoom_*`): same three planes, but per-label\n", " cameras zoomed onto the label's extent.\n", "\n", "We deliberately keep the two `Split` rows as named locals\n", "(`overview_row`, `zoom_row`). They are the structural truth of the\n", "layout, and section 5 uses them as its iteration source -- so the\n", "cohort loop never reconstructs cell ids from a naming convention. The\n", "same DSL nodes you build the layout from are the ones you iterate.\n" ] }, { "cell_type": "code", "execution_count": 10, "id": "cc70a57066b3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Output directory: C:\\Users\\floca\\AppData\\Local\\Temp\\cohort_screenshots_lzihzd1b\n" ] } ], "source": [ "def make_row(role: str) -> Split:\n", " return Split.horizontal(\n", " LayoutWindow.create(\n", " id=f\"mxn__{role}_axial\",\n", " view_direction=ViewDirection.AXIAL,\n", " selection=role,\n", " name=f\"{role.title()} axial\",\n", " ),\n", " LayoutWindow.create(\n", " id=f\"mxn__{role}_coronal\",\n", " view_direction=ViewDirection.CORONAL,\n", " selection=role,\n", " name=f\"{role.title()} coronal\",\n", " ),\n", " LayoutWindow.create(\n", " id=f\"mxn__{role}_sagittal\",\n", " view_direction=ViewDirection.SAGITTAL,\n", " selection=role,\n", " name=f\"{role.title()} sagittal\",\n", " ),\n", " )\n", "\n", "# Keep the rows as named locals: they double as the iteration structure\n", "# in section 5, so the cohort loop never has to reconstruct cell ids\n", "# from a naming convention.\n", "overview_row = make_row(\"overview\")\n", "zoom_row = make_row(\"zoom\")\n", "\n", "cohort_layout = MxNLayoutDocument.create(\n", " root=Split.vertical(overview_row, zoom_row),\n", " name=\"cohort overview/zoom\",\n", ")\n", "\n", "# Tempdir for screenshots + the rendered report. The layout itself\n", "# stays in memory -- see section 7 for when you would persist it.\n", "work_dir = Path(tempfile.mkdtemp(prefix=\"cohort_screenshots_\"))\n", "print(f\"Output directory: {work_dir}\")\n" ] }, { "cell_type": "markdown", "id": "06390c6fa538", "metadata": {}, "source": [ "## 4. Per-label geometry helper\n", "\n", "For each label we need its world-space centroid (where to point the\n", "crosshair) and its world-space extent (how far to zoom in). This is the\n", "kind of small, reusable analytic block you would normally factor into\n", "your own utilities -- we keep it inline here because computing it *is*\n", "the lesson.\n", "\n", "The library follows the SimpleITK / pynrrd convention: numpy array axes\n", "are ordered ``(Z, Y, X)`` (slowest first), while ``spacing`` and the\n", "workbench's world coordinates are in ``(X, Y, Z)`` order. To convert a\n", "voxel index ``(k, j, i)`` to a world position ``(x, y, z)`` for the\n", "identity-direction case:\n", "\n", "```\n", "world_x = i * sx + origin_x\n", "world_y = j * sy + origin_y\n", "world_z = k * sz + origin_z\n", "```\n", "\n", "i.e. the index components are reversed before scaling. Real data with a\n", "non-identity direction matrix needs ``world = direction @ index_xyz *\n", "spacing + origin``.\n" ] }, { "cell_type": "code", "execution_count": 11, "id": "d3dbfd5ba946", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "case_001 / tumor: centroid=[60.02173913 34.30434783 23.91304348], extent=[4. 6. 2.5]\n" ] } ], "source": [ "def label_geometry(\n", " seg: mw.MultiLabelSegmentation,\n", " label_value: int,\n", ") -> tuple[np.ndarray, np.ndarray]:\n", " \"\"\"Return (world_centroid, world_extent) for a label.\n", "\n", " Both are in workbench world-coordinate order ``(x, y, z)``.\n", " ``world_extent`` is the bounding-box edge length in world units\n", " along each axis.\n", " \"\"\"\n", " arr = seg.get_group_image(0).array # numpy axes (Z, Y, X)\n", " mask = arr == label_value\n", " if not mask.any():\n", " raise ValueError(f\"label value {label_value} has no voxels\")\n", " # coords[0] = k (Z), coords[1] = j (Y), coords[2] = i (X)\n", " coords = np.array(np.where(mask), dtype=np.float64)\n", " spacing = np.asarray(seg.spacing, dtype=np.float64) # (sx, sy, sz)\n", " origin = np.asarray(seg.origin, dtype=np.float64) # (ox, oy, oz)\n", " centroid_idx = coords.mean(axis=1) # (k, j, i)\n", " extent_idx = coords.max(axis=1) - coords.min(axis=1) # (extent_k, extent_j, extent_i)\n", " # Reverse the index components so they pair (i, j, k) with (sx, sy, sz).\n", " centroid_world = centroid_idx[::-1] * spacing + origin\n", " extent_world = extent_idx[::-1] * spacing\n", " return centroid_world, extent_world\n", "\n", "# Quick self-check on the first case.\n", "first = cases[0]\n", "label0 = first.segmentation.labels[0]\n", "c, e = label_geometry(first.segmentation, label0.value)\n", "print(f\"{first.case_id} / {label0.name}: centroid={c}, extent={e}\")\n" ] }, { "cell_type": "markdown", "id": "263749c2cbe4", "metadata": {}, "source": [ "## 5. The cohort loop\n", "\n", "The script:\n", "\n", "1. Pushes the in-memory `cohort_layout` to the workbench once\n", " (`mxn.set_layout(cohort_layout)`).\n", "2. For every case: shows the image and segmentation, reinitialises the\n", " overview row to frame the whole volume, then iterates the labels.\n", "3. For every label: moves the global crosshair to the label centroid,\n", " tightens the cameras of the zoom row to the label's extent, captures\n", " an editor-level screenshot, and saves it next to the report.\n", "4. Removes the case's nodes before the next iteration so subsequent\n", " cases start from a clean DataStorage state. The editor's own\n", " geometry-helper nodes (planes, widget tree) are left untouched.\n", "\n", "The per-cell loops iterate `overview_row.windows()` /\n", "`zoom_row.windows()` -- the cached `Split` objects from section 3.\n", "That is the cleanest pattern when the layout is authored *here*.\n", "For layouts loaded from disk that were authored elsewhere,\n", "see the note in section 7.\n" ] }, { "cell_type": "code", "execution_count": 12, "id": "6de201b499af", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Layout applied: ['mxn__overview_axial', 'mxn__overview_coronal', 'mxn__overview_sagittal', 'mxn__zoom_axial', 'mxn__zoom_coronal', 'mxn__zoom_sagittal']\n", "\n", "Wrote C:\\Users\\floca\\AppData\\Local\\Temp\\cohort_screenshots_lzihzd1b\\report.md\n", "Captured 10 screenshots.\n" ] } ], "source": [ "ZOOM_PADDING = 1.5 # parallel_scale multiplier on top of the label half-extent\n", "MIN_PARALLEL_SCALE = 5.0 # millimetres; floors the zoom for tiny labels\n", "\n", "mxn = wb.mxn\n", "mxn.set_layout(cohort_layout)\n", "print(f\"Layout applied: {[c.id for c in mxn.list_windows()]}\")\n", "\n", "report_lines: list[str] = [\"# Cohort screenshot report\\n\"]\n", "\n", "\n", "def _aim_camera_at(window, target: tuple[float, float, float], parallel_scale: float) -> None:\n", " \"\"\"Translate ``window``'s camera so the lesion sits in frame.\n", "\n", " Setting only the selected position drops the crosshair on the lesion\n", " but leaves the camera focused wherever it was -- so a zoomed\n", " parallel_scale ends up showing the wrong region. We preserve the\n", " camera's view direction and shift both ``focal_point`` and\n", " ``position`` to the new target.\n", " \"\"\"\n", " cam = window.get_camera()\n", " fields: dict[str, object] = {\n", " \"focal_point\": tuple(float(c) for c in target),\n", " \"parallel_scale\": float(parallel_scale),\n", " }\n", " if cam.position is not None and cam.focal_point is not None:\n", " view_offset = tuple(p - f for p, f in zip(cam.position, cam.focal_point))\n", " fields[\"position\"] = tuple(\n", " float(c + v) for c, v in zip(target, view_offset)\n", " )\n", " window.set_camera(**fields)\n", "\n", "\n", "for case in cases:\n", " img_node = wb.show(case.image, name=f\"{case.case_id}-image\")\n", " seg_node = wb.show(case.segmentation, name=f\"{case.case_id}-seg\")\n", " wb.reinit([img_node, seg_node])\n", "\n", " report_lines.append(f\"## {case.case_id}\\n\")\n", "\n", " for label in case.segmentation.labels:\n", " centroid, extent = label_geometry(case.segmentation, label.value)\n", " target = tuple(float(c) for c in centroid)\n", " # The MxN editor's per-cell crosshair is currently not coupled to\n", " # the global selected position, so wb.set_position(...) wouldn't\n", " # move the cells onto the label. Drive each cell directly --\n", " # iterating the cached row Splits instead of reconstructing cell\n", " # ids from a naming convention.\n", " for row in (overview_row, zoom_row):\n", " for win in row.windows():\n", " mxn[win.id].set_selected_position(target)\n", "\n", " # parallel_scale is the half-height of the orthographic view; pad\n", " # the dominant extent so the label sits comfortably in frame.\n", " scale = max(MIN_PARALLEL_SCALE, ZOOM_PADDING * 0.5 * float(extent.max()))\n", " # Aim each zoom-row camera at the lesion AND tighten parallel_scale.\n", " # Selected position alone moves the crosshair but leaves the camera\n", " # looking at the original centre, so the zoom would crop the wrong\n", " # region.\n", " for win in zoom_row.windows():\n", " _aim_camera_at(mxn[win.id], target, scale)\n", "\n", " # Force a redraw so the camera-driven changes settle before the\n", " # snapshot; otherwise the screenshot races the renderer.\n", " wb.update()\n", " png = mxn.screenshot()\n", " fname = work_dir / f\"{case.case_id}_{label.name}.png\"\n", " fname.write_bytes(png)\n", " report_lines.append(\n", " f\"### {label.name} (value={label.value})\\n\"\n", " f\"- centroid (world, mm): \"\n", " f\"{centroid[0]:.1f}, {centroid[1]:.1f}, {centroid[2]:.1f}\\n\"\n", " f\"- extent (world, mm): \"\n", " f\"{extent[0]:.1f}, {extent[1]:.1f}, {extent[2]:.1f}\\n\"\n", " f\"\\n![{case.case_id} {label.name}]({fname.name})\\n\"\n", " )\n", "\n", " img_node.remove()\n", " seg_node.remove()\n", "\n", "report_path = work_dir / \"report.md\"\n", "report_path.write_text(\"\\n\".join(report_lines), encoding=\"utf-8\")\n", "print(f\"\\nWrote {report_path}\")\n", "print(f\"Captured {sum(1 for f in work_dir.glob('*.png'))} screenshots.\")\n" ] }, { "cell_type": "markdown", "id": "b0ecd386b47a", "metadata": {}, "source": [ "## 6. Render the report inline\n", "\n", "The on-disk `report.md` keeps relative image paths so it stays portable\n", "as a per-cohort QA artifact. Jupyter's Markdown renderer, however,\n", "won't follow file paths into the tempdir. We post-process the same\n", "report and inline each screenshot as a base64 data URI just for the\n", "inline preview." ] }, { "cell_type": "code", "execution_count": 13, "id": "72bf60a68e29", "metadata": {}, "outputs": [ { "data": { "text/plain": "", "text/markdown": "# Cohort screenshot report\n\n## case_001\n\n### tumor (value=1)\n- centroid (world, mm): 60.0, 34.3, 23.9\n- extent (world, mm): 4.0, 6.0, 2.5\n\n\"case_001\n\n### edema (value=2)\n- centroid (world, mm): 35.7, 40.6, 14.1\n- extent (world, mm): 6.0, 5.0, 2.5\n\n\"case_001\n\n### necrosis (value=3)\n- centroid (world, mm): 56.3, 52.4, 44.2\n- extent (world, mm): 5.0, 6.0, 5.0\n\n\"case_001\n\n## case_002\n\n### tumor (value=1)\n- centroid (world, mm): 60.6, 53.5, 28.2\n- extent (world, mm): 3.0, 5.0, 5.0\n\n\"case_002\n\n### edema (value=2)\n- centroid (world, mm): 34.9, 52.8, 21.9\n- extent (world, mm): 6.0, 5.0, 5.0\n\n\"case_002\n\n### necrosis (value=3)\n- centroid (world, mm): 33.0, 57.5, 25.5\n- extent (world, mm): 6.0, 8.0, 5.0\n\n\"case_002\n\n### lesion (value=4)\n- centroid (world, mm): 43.3, 59.9, 27.0\n- extent (world, mm): 6.0, 5.0, 5.0\n\n\"case_002\n\n## case_003\n\n### tumor (value=1)\n- centroid (world, mm): 34.3, 50.3, 31.2\n- extent (world, mm): 7.0, 7.0, 5.0\n\n\"case_003\n\n### edema (value=2)\n- centroid (world, mm): 45.1, 36.5, 45.4\n- extent (world, mm): 5.0, 3.0, 5.0\n\n\"case_003\n\n### necrosis (value=3)\n- centroid (world, mm): 59.2, 43.9, 25.0\n- extent (world, mm): 6.0, 6.0, 5.0\n\n\"case_003" }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import base64\n", "\n", "from IPython.display import Markdown\n", "\n", "# Pixel width for the inline preview. A percentage value would be\n", "# ignored: Jupyter's markdown output area has no explicit width, so\n", "# percentage widths resolve against an undefined containing block and\n", "# the browser falls back to the image's intrinsic pixel size.\n", "INLINE_IMG_WIDTH_PX = 200\n", "\n", "\n", "def _embed_pngs(md: str, base_dir: Path, width_px: int) -> str:\n", " \"\"\"Inline every relative PNG reference as a thumbnail data URI.\n", "\n", " Jupyter's Markdown renderer won't follow file paths under a tempdir,\n", " and a plain ``![alt](data:...)`` reference is shown at native size.\n", " We rewrite each image line to a raw-HTML ```` with an explicit\n", " pixel width so the report stays compact in the notebook. The\n", " on-disk ``report.md`` (which still references relative PNG paths at\n", " full resolution) is left untouched.\n", " \"\"\"\n", " out: list[str] = []\n", " for line in md.splitlines():\n", " if line.startswith(\"![\") and \"](\" in line and line.rstrip().endswith(\".png)\"):\n", " alt_part, _, rest = line.partition(\"](\")\n", " alt = alt_part[2:] # drop leading '!['\n", " png_name = rest[:-1] # drop trailing ')'\n", " png_path = base_dir / png_name\n", " if png_path.is_file():\n", " b64 = base64.b64encode(png_path.read_bytes()).decode(\"ascii\")\n", " out.append(\n", " f'\"{alt}\"'\n", " )\n", " continue\n", " out.append(line)\n", " return \"\\n\".join(out)\n", "\n", "\n", "inline_md = _embed_pngs(\n", " report_path.read_text(encoding=\"utf-8\"), work_dir, INLINE_IMG_WIDTH_PX\n", ")\n", "Markdown(inline_md)\n" ] }, { "cell_type": "markdown", "id": "c94a77c46053", "metadata": {}, "source": [ "## 7. From synthetic to real\n", "\n", "What you would change to point this at real DICOM cases:\n", "\n", "- Replace `make_cohort(...)` with iteration over a directory: each\n", " case becomes the path to an image plus a segmentation NRRD\n", " (`wb.show(path)` / `node.set_data(path)` accept file paths).\n", "- Switch the transport to `file-reference` mode for large data on\n", " localhost -- saves the full HTTP round-trip on every case.\n", "- Move `wb.show` calls under a transaction-style helper that batches\n", " rendering updates (`wb.update()`) to skip per-call refresh flicker.\n", "- For very large cohorts, run multiple workbench instances in\n", " parallel via `mw.launch(...)` and dispatch cases with a job queue.\n", "\n", "The shape of the script -- *layout once, loop over cases, loop over\n", "labels, screenshot, report* -- stays the same.\n", "\n", "### Persisting the layout\n", "\n", "This notebook keeps `cohort_layout` in memory and pushes it straight\n", "to the workbench. For a production cohort report script you'd usually\n", "serialise it once and commit it next to the script:\n", "\n", "```python\n", "layout_path.write_text(json.dumps(cohort_layout.to_json(), indent=2),\n", " encoding=\"utf-8\")\n", "# ... and load with:\n", "mxn.set_layout(layout_path) # accepts MxNLayoutDocument | dict | str | Path\n", "```\n", "\n", "The on-disk JSON is reviewable in PRs and lets non-Python tools edit\n", "the layout. It also decouples the \"what does the report look like\"\n", "decision from the \"what does the script do\" decision.\n", "\n", "### Iterating over windows when the layout was authored elsewhere\n", "\n", "Section 5 iterates `overview_row` / `zoom_row` because we built those\n", "`Split`s ourselves. When the layout comes from a preset, a file you\n", "did not author, or `mxn.get_layout()`, you don't have those locals --\n", "use `MxNWindowSelector` instead. It filters by *structural* invariants\n", "(group binding, view direction, ids, display names) rather than name\n", "conventions:\n", "\n", "```python\n", "from mitk.mxn.layout import MxNWindowSelector, ViewDirection\n", "\n", "doc = mxn.get_layout() # cache once, outside the loop\n", "zoom_ids = MxNWindowSelector(doc).where(group=\"zoom\").ids()\n", "axial_ids = MxNWindowSelector(doc).by_view(ViewDirection.AXIAL).ids()\n", "\n", "for wid in zoom_ids:\n", " mxn[wid].set_selected_position(target)\n", "```\n", "\n", "The selector returns DSL `LayoutWindow` data classes (or just their\n", "ids); bridge to live cells with `mxn[id]`. Filtering on `group=` is\n", "the most robust choice -- it survives id renames, because the binding\n", "is structural, not lexical.\n" ] }, { "cell_type": "code", "execution_count": 14, "id": "dea8a8277440", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Output directory: C:\\Users\\floca\\AppData\\Local\\Temp\\cohort_screenshots_lzihzd1b\n" ] } ], "source": [ "print(f\"Output directory: {work_dir}\")\n" ] }, { "cell_type": "code", "execution_count": 14, "id": "f99aed97", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.1" } }, "nbformat": 4, "nbformat_minor": 5 }