From Pixels to Polygons
A survey of deep learning approaches for medical image-to-mesh reconstruction
From Pixels to Polygons
A visual survey of deep learning approaches that reconstruct anatomical meshes directly from medical images, bridging imaging, geometric modelling, and simulation-ready digital anatomy.
Drag to compare benchmark landscapes
A Neuralangelo-style comparison block for this project page: move the slider between the cortical and cardiac meta-analysis figures.
Project video
A short overview of the survey. The embedded player keeps the browser's native controls enabled so visitors can drag the progress bar, adjust volume, and switch to fullscreen.
Overview video. The video is configured to attempt autoplay on page load and keeps the native controls enabled for seeking.
Why image-to-mesh reconstruction matters
In computational medicine and in-silico trials, image-to-mesh reconstruction is the bridge between medical imaging and numerical simulation. The paper emphasizes a shift from segmentation-heavy pipelines toward models that directly predict geometrically meaningful mesh representations from CT, MR, ultrasound, projections, or sparse point observations.
Patient-specific meshes are more than a visualization layer. They are the structural backbone for downstream tasks such as hemodynamic simulation, structural mechanics, virtual device testing, motion analysis, and digital twin generation. When mesh quality is poor, simulation quality suffers too.
That is why the survey treats reconstruction as a full pipeline problem: input modality, representation, topology, regularization, and evaluation all matter. The most compelling methods are not only accurate against ground truth, but also stable, anatomically plausible, and ready for computation.
In-silico trial pipeline. Medical images feed reconstruction, generation, and simulation stages. This figure captures why anatomical meshing matters beyond segmentation: it is the geometric substrate for downstream evidence generation.
Survey map
The paper provides a unified taxonomy that connects methodology, input modality, output representation, and evaluation strategy. It also makes the field easier to navigate by splitting it into four families and twelve subcategories.
Taxonomy of the field. Template models, statistical shape models, generative models, and implicit models are connected to input type, feature representation, output form, and evaluation design.
Template models
Start from an initial mesh and learn how to deform it into the target anatomy.
- Strong control over topology
- Good fit for stable anatomical priors
- Conditioned deformation and registration variants
Statistical shape models
Use low-dimensional shape priors to constrain reconstruction around plausible anatomical variation.
- Linear and non-linear latent spaces
- Robustness to noisy data
- Especially useful when strong population priors exist
Generative models
Synthesize shapes directly from data distributions rather than relying on a fixed template.
- VAE, GAN, interpolation, and diffusion families
- Adapt well to diverse or pathological shapes
- Useful when completion or synthesis matters
Implicit models
Represent anatomy through continuous functions such as SDFs, occupancy fields, Neural ODEs, or NeRF-style densities.
- High-resolution and continuous surfaces
- Flexible topology
- Strong overall robustness in the meta-analysis
Representative formulations
Rather than showing every method in the paper, the page highlights a compact set of schematics that capture how the main families are typically formulated.
Conditioned deformation. CNN features are transferred to a graph network that deforms a template mesh while preserving a known topological scaffold.
Template-based registration. A learned deformation field aligns a template representation with the target anatomy, connecting image registration and mesh generation.
Linear SSM. A PCA/SVD block predicts coefficients in a low-dimensional statistical basis, turning image evidence into a structured shape prior.
Non-Linear SSM. A deep encoder predicts coefficients in a low-dimensional statistical basis, turning image evidence into a structured shape prior.
Completion-based generative model. Sparse contours or points are densified into complete point clouds before meshing, making generative completion central to reconstruction.
Implicit flow / Neural ODE. Continuous deformation dynamics provide smooth trajectories and strong geometric regularity for complex anatomical surfaces.
What the survey finds
The meta-analysis compares method categories on cardiac and cortical benchmarks. The key message is not that one family always wins, but that different families dominate under different anatomical and geometric constraints.
Relative trend across the reviewed studies
Implicit models > Generative models > Statistical shape models > Template models
The paper frames this as a relative trend rather than an absolute rule. Anatomy, modality, topology, supervision, and simulation requirements can still change which model family is best for a specific task.
Why the ranking is not the whole story
- Template: controlled topology and simulation-friendly priors
- SSM: strong regularization from population anatomy
- Generative: flexibility for diverse and pathological shapes
- Implicit: continuous, high-resolution, topology-flexible surfaces
Metrics, datasets, and evidence base
One strength of the survey is that it does not stop at method taxonomy. It also systematizes the losses, evaluation metrics, and public datasets that shape experimental practice in this field.
PRISMA-guided review process. The survey follows a systematic review structure to define inclusion, screening, and analysis.
Metrics span geometry, topology, and function
- Shape similarity: Chamfer, Hausdorff, ASSD, MSD, Dice, IoU, normal consistency
- Regularization: smoothness, curvature, self-intersection, topology, edge quality
- Functional utility: ejection fraction, disease prediction, CFD velocity, kinetic energy
- Efficiency: inference time and simulation readiness
Datasets span many organs and modalities
The paper curates datasets across cardiac imaging, brain MRI, thoracic and abdominal CT, skull reconstruction, endoscopy, microscopy, fetal ultrasound, and more. That breadth is valuable because it connects reconstruction design choices to anatomy and clinical use.
Open challenges and future directions
The field is moving toward higher-fidelity, more topology-aware, and more simulation-ready reconstruction. The paper frames the future around a few central tensions: topology, geometric fidelity, multi-modality, and computational usability.
Challenge map. Topological requirements, geometric accuracy, and multi-modality integration determine which reconstruction strategy is most appropriate for a given medical task.
Mesh validity still matters
Simulation-ready models require connectivity, manifoldness, and low self-intersection, especially for anatomy where downstream solvers are sensitive to geometric artifacts.
Fine detail is clinically meaningful
Thin walls, folds, bifurcations, and device-relevant structures push models toward higher-resolution and more continuous representations.
Multi-modal and dynamic data are rising
CT, MR, ultrasound, sparse views, and time-varying sequences increasingly need to be integrated into a single reconstruction framework.
Citation
Show BibTeX
@article{lin2026pixels,
title = {From Pixels to Polygons: A Survey of Deep Learning Approaches for Medical Image-to-Mesh Reconstruction},
author = {Lin, Fengming and Zakeri, Arezoo and Xue, Yidan and MacRaild, Michael and Dou, Haoran and Zhou, Zherui and Zou, Ziwei and Sarrami-Foroushani, Ali and Duan, Jinming and Frangi, Alejandro F.},
journal = {Medical Image Analysis},
year = {2026}
}