Work Plan

i3DPost is planned as a 36-month work programme, which is structured into 8 Work Packages.

  • WP1 Management
  • WP2 Functional requirements and scenarios of use
  • WP3 Multiple-source On-set Data Capture
  • WP4 3D Feature Analysis and Understanding
  • WP5 Structured 3D Content
  • WP6 Intelligent tools for content manipulation
  • WP7 User Evaluation in Experimental Production
  • WP8 Dissemination

The 36-month work plan is organised into three Phases. These provide control points related to Deliverables and major Milestones, the end of each phase resulting in a Milestone.

The Phases are:

  • Phase 1, Proof of Concepts, from start M1 to end M12
  • Phase 2, Initial Prototypes, from M13 to M24
  • Phase 3, Advanced Prototypes, from M25 to M36.F

The research is based on and directed by WP2 Functional Requirements and Scenarios of Use. This package embodies task analyses requirement definitions driven by the media industry partners and validated by discussions with companies in the postproduction and games sectors representing the principal future customer base. Since the requirements themselves will be challenged and refined by the research outcomes and early prototypes, WP2 contains iterative tasks that will be repeated in Phase 2 in light of the results of the initial proof of concept work.

WP3, WP4 and WP5 are the core research activities. In WP3 Multiple-source On-set Data Capture we will validate two forms of multi-camera system and generate test sequences of 3D data. For the first, in WP3T1, we will use existing state-of-the-art HD digital film camera research facilities at UNIS; the second stage in WP3T2 will develop a portable capture method to allow existing multiple HD digital film cameras to be used on set. The capture of experimental datasets at the outset will facilitate subsequent research into the processing of captured content by animators to allow reuse and editing whilst aiming to render with the quality of source video. WP3 will take the initial requirements and scenarios as its starting point and run to the end of Phase 2.

WP4 3D Feature Analysis and Understanding and WP5 Structured 3D Content cover a series of major research tasks. WP4T1 investigates the extraction of point clouds from the 3D data streams, the analysis of motion data, video reconstruction and matting for post-production; WP4T2 covers the semantic labelling of 3-D point clouds. The essential idea in WP4 is to exploit higher-level scene knowledge to i) improve motion estimation and hence object matting, and ii) to improve information access to the media stream as a whole. Higher-level scene knowledge is embodied in the temporally evolving point clouds representing high confidence points in the 3D scene. The semantic aspect of WP4 therefore allows the association of point clouds with particular objects e.g. people, hence enabling the media stream to be accessed by ?object? for manipulation later on. Motion estimation (the mapping of each pixel in a given image to another in some other image) will be improved by exploiting geometrical constraints again derived from these 3D point clouds. Hence one major aspect of WP4 is the reliable extraction of this 3D point cloud information and that is the starting exercise in WP4.

WP5 explores ways of structuring the captured data hence creating models of sets, actors and faces. This further facilitates semantic labelling of these models. It is through this structured data that scenes and performances captured in the real world (at film quality) can be manipulated and animated independently in postproduction. The work is divided into the organisation of the unstructured 3D point data from WP4 into structured and temporally consistent mesh models of the 3D scenes (WP5T1), characters (WP5T2) and faces (WP5T3), taking advantage of 3D assisted motion estimation. WP5T4 will investigate semantic descriptions of face, character, and character action for metadata tagging and storage, and the association of data and metadata. Semantic descriptions will not only facilitate search and retrieval from archives but will additionally help in a number of lower-level tasks like segmentation of the scene elements, motion estimation and creation of structured scene content. We will also address the problem of relighting actor and scene representations for reuse in different contexts. While the general case re-lighting problem is likely to remain unsolved for the foreseeable future, the ability to identify and extract lighting data from similar takes under different conditions should lead to an increasing number of useful instances. WP4 and WP5 will not only use the data captured in WP3, vut also build on previous research by the partners: this, and the theoretical work required, allow WP4 and WP5 to start at the outset of the project and run through until the end of Phase 2 at M30.

WP6 Intelligent tools for content manipulation will investigate practical solutions for the use of 3D semantically tagged data and models, and their integration with the professional media processes for postproduction and games development. Here we will create software tools for 3D metadata generation and management, software plug-ins for compositing the structured models and performances into typical production sequences, and workflows for handling 3D data streams from multiple sources. There are separate tasks to address Workstation applications (WP6T1), dataflow applications capable of managing the multiple streams created by 3D capture (WP6T2), Open-FX plug-ins to industry-standard software tools, enhanced with 3D technologies (WP6T3) and a software library to support the development of compositor toolsets. The software tools developed in WP6 will progressively incorporate the successive stages of the software prototypes developed as part of the research in WP3, 4 and 5, translating the most promising algorithms into robust industrially-useful code. This work starts at the close of Phase 1 and runs throughout Phases 2 and 3.

The Deliverables from WP6 will provide the prototypes for the professional end-user environments in Games and Postproduction, which are evaluated in Experimental Production under WP7. As soon as sufficient early prototypes are available, halfway through Phase 2, the user partners will be in a position to start work on the WP7 User Evaluation in Experimental Production of film and game sequences, testing the emerging tools and processes with real data. The rapid involvement of users (who have high level technology skills and in-house R&D facilities) is designed to steer and focus the prototyping process. The work in WP7 will centre on Field Trials in the domains of digital cinema postproduction (WP7T1 and games development (WP7T2), with formal evaluation procedures.
The results of the R&D are fed through to the R&D, postproduction and games communities by the dissemination activities in WP8, and to the software development community by the sub-task for Contribution to Standards. Contributions to the development of data formats and standards will be encapsulated in proposals for extensions to the OpenFX (OFX) standard for Open Source plug-ins for visual effects to cover 3D data. (WP8T2).