This paper addresses the challenge of novel view synthe-sis for a human performer from a very sparse set of cameraviews. Some recent works have shown that learning implicitneural representations of 3D scenes achieves remarkableview synthesis quality given dense input views. However,the representation learning will be ill-posed if the views arehighly sparse. To solve this ill-posed problem, our key ideais to integrate observations over video frames. To this end,we propose Neural Body, a new human body representationwhich assumes that the learned neural representations atdifferent frames share the same set of latent codes anchoredto a deformable mesh, so that the observations acrossframes can be naturally integrated. The deformable meshalso provides geometric guidance for the network to learn3D representations more efficiently. Experiments on a newlycollected multi-view dataset show that our approach out-performs prior works by a large margin in terms of the viewsynthesis quality. We also demonstrate the capability of ourapproach to reconstruct a moving person from a monocularvideo on the People-Snapshot dataset. The code and datasetwill be available at https://zju3dv.github.io/neuralbody/. Read More