Method for alignment of low-quality noisy depth map to the high-resolution colour image

A method, apparatus or program for, obtaining an image frame 800 and a corresponding depth map 801 of a scene, determining a minimum and maximum depth of the scene, determining a number of depth layers 804 for the depth map, projecting pixels of the image frame to the depth layers, determining cost values 828 for the projected pixels, filtering the cost values 824, and obtaining a depth value of a pixel using the filtered cost values 834. The embodiment relates to the production of a high resolution depth image of a scene captured with a high resolution colour camera and an associated low resolution time of flight depth sensor. Preferably the image is colour. Cost values may be calculated using depth value confidence information. Other independent claims do not require the min or max depth to be determined before the number of layers is calculated, but specify that the image pixels are projected to the depth layers by back projecting according to an estimated projective transform. Yet more independent claims specify that two images of a scene are obtained along with information on the two differing viewpoints. Pixels are projected and costs values calculated and filtered to derive a virtual image from a virtual camera position.