|
|
The research is designed to provide the industry, at the end of forty-two months, with calibrated tools for cross-platform management of the highest quality cinematic data in an increasingly automated, self-adjusting manner based on a coherent new workflow model proposed which is optimised for Digital Cinema. The following table sets out, in summary form, the relations between the State of the Art, the Research Activities, and the principal Innovations expected to result over the forty-two-month lifetime of the Project.
There is an emphasis throughout on technologies that offer the potential for the real-time processes, whether for capture or synthesis, postproduction processing, encoding/decoding et cetera. Given the very large data sets currently involved in ‘film quality’ digital movies, many of the procedures require very large computation resources and bandwidth. IP-RACINE looks at flexible ways of increasing the cost-effectiveness of processing power (by using the range of GPU, CPU and re-configurable FPGA technology now becoming available) and at new data representations that may reduce computational requirements.
While the work of IP-RACINE is targeted at the requirements of digital cinema, as the ‘top quality’ end of the digital entertainment technology chain, several of the RTD activities will facilitate future cross-media and cross-platform authoring. The focus on a coherent metadata and format chain, to facilitate adaptive and contextualised delivery is the most obvious example. The integration of adaptive sound and sound synthesis is a significant extension in the audio domain (usually the Cinderella of multimedia). In the longer term, the development of means of describing and processing ‘film objects’ at a higher semantic level (possibly based on vision description) raises the possibility of being able to cut an paste characters, sets, plots or behaviours from (say) a movie to a game, thus creating “The Ben Hur Experience” or creating new programmes from unexpected combinations of classic film characters.
The IST’s concern to foster efficiency and sustainability by promoting generic software, computing systems and content development tools is reflected in IP-RACINE’s stress on the development of workflow models, standardised formats and common metadata schemas which can be applied by all system and service providers.
The following table summarises the relationship between IP-RACINE research areas, the current state-of-the-art and the innovations expected to result from IP-RACINE.
|
Area of Activity
|
State of the Art
|
Expected Innovation
|
|
Workflow and data handling, storage, and processing through each of the creative stages in the movie making process with integrated metadata concepts
|
Digital Cinema process fragmented, rudimentary and highly skills-based, operating at or beyond the limits of technology
|
Standards profile for Digital Intermediate encompassing all the requirements of the Digital Cinema workflow
|
|
Lack of standardised file formats, data handling and processing methods
|
Tools and structures for colour space representation and conversion to support the standardised workflow at all stages
|
|
Stand-alone, incompatible platforms with different processing requirements & results
|
Workflow Support Service making it possible to monitor and control production of digital Cinema content keeping track of versions and storage locations as well as supporting cross media production
|
|
Inefficient file-based systems requiring massive data transfer.
|
Metadata structures, encoding and transport mechanism based on existing or emerging standards
|
|
Lack of metadata-based workflow.
|
Tools for content and metadata formats conversion covering AA F, MXF, MPEG-7, and SMPTE
|
|
Optimal methods of capturing, storing and pre-processing data from digital cameras
|
Electronic cameras are optimised for CRT viewing with HDTV cameras capturing at b bit/pixel spatially subsampled in YCrCb component video and compressed.
|
Represent various stages in the process as a set of the raw data from the camera combined with a processing description (rendering intent metadata), for example as 3-D-LUT
|
|
Data capture formats tied to log format derived from film characteristics and film-based frame structures that hinder post-processing
|
Linear light representation of the scene
|
|
Cameras do not allow facilitate speed recording (equivalent to over- / under-cranking)
|
Remove barriers in the way of using electronic acquisition. A few known areas are the absence of over-/undercranking, speed ramps, acquisition at a high frame rate, image spatial resolution
|
|
No standardised metadata workflow
|
Record Metadata relevant to the workflow in a standardised, structured way
|
|
Camera tethered to cumbersome field recorder
|
Camcording with in-camera recording of sequences of at least 10 minutes stored in a device of less than 600 cm3 volume featuring a power consumption of less than 10W and capable of keeping track with the growing frame formats (2K, 4K) and frame rates
|
|
3-D Imaging for Virtual Film Studios
|
Virtual studio technology has not been fully accepted, as the results look too ‘computer generated’ and it is necessary to increase the visual impression of the reality of environments.
|
Removing the ‘fixed camera’ requirement for high quality 3-D ray tracing. Virtual studios using IBR to achieve 3 degree of freedom movement.
|
|
Pre-calculation enables a real time 3-D ray traced virtual studio in SD TV resolution for fixed viewpoints
|
Increase in sensor accuracy, calculation power and performance to increase virtual studio resolution progressively from SD to Film resolution.
|
|
Shaders, group dynamics and particle-based techniques are extending the range of real-time graphics in video games
|
Support for camera movement, improved sensing, real-time data rates; parametrical real time group dynamics and atmospheric effects at low cost, integrated with a film-resolution virtual studio
|
|
High-level data descriptions necessary to define and repurpose aspects of the data with real-time processing devices
|
Image-Based Rendering techniques theoretically allow visualisation from novel points-of-view but practical implementation is limited to video resolution. Video- Based Rendering is a theoretical possibility.
|
Video-based rendering based on multiple video streams rather than images. Manipulation of Video-Based Representations (VBR) to yield animated sequences not in the original sequences
|
|
Real-time multi-view video interpolation is still in its infancy.
|
Multi-view interpolation of high-resolution video streams in real time
|
|
Digital Intermediate processing is pixel based and increasingly cumbersome as resolutions increase
|
Images treated as intensity surfaces represented and manipulated as mathematical functions to give resolution-independent processing and rendering at any resolution for any platform.
|
|
Postproduction operations based on recolourisation of image objects requiring image segmentation and tracking using pixel-based motion vectors.
|
Topographic approaches to image matching based on 3D volumes, manipulated in a continuous timeflow independent of film frames.
|
|
Processing devices are based on manipulating very large numbers of pixels, using CPUs, GPUs, FPGA hardware accelerators in various configurations.
|
Novel programmable device, based on accelerated PC architectures, to manipulate & render novel data formats in real-time.
|
|
Audio content is a mix of natural and synthetic material. Natural material includes multiple sound sources such as different voices or voices plus background material.
|
Object-based audio scene representation allowing transparent processing of natural and synthetic audio with sound morphing, individual voice extraction & transformation etc. Use of high-level descriptions to improve transformations.
|
|
Indexing and annotation of image sequences is a largely manual technique
|
Optic flow techniques to prove concept of automated indexation. Prototype space-variant techniques and multimodal methods for indexing specific D-Cinema material
|
|
More cost effective and efficient playout and projection systems, which are also capable of providing novel and compelling forms of entertainment including high quality 3-D projection systems
|
Micro-displays with 2048 horizontal pixels are commercially available for projectors. JVC & NTT have demonstrated a 3840x2160 display device. Hollywood studios have expressed interest in 4k-resolution projection
|
Prototype projector platform with >2K resolution, high-bandwidth signal interface and signal integrity, and improved light sources, capable of commercialisation.
|
|
3-D projection for large audiences is best supported using low-cost passive polarised glasses, but poor image brightness due to light loss.
|
New architectures giving much brighter 3D projection from single and dual projector systems.
|
|
Digital Cinema Distribution master is immature and seldom used.
|
Improved DCDM with multilingual versioning and subtitling, automated versioning for different formats and improved compression.
|
|
Cinema set-up and projection results are highly dependent on operator skills
|
Much greater automation based on metadata-driven server and projector technology with colour-gamut remapping and automated on-site maintenance.
|
|
Digital projectors costs are very much higher than film projectors.
|
Construction costs of digital projectors to be reduced to half present levels, with increased resolution & brightness.
|
|
Digital servers use external decoders in a separate media bloc
|
Reduce costs by integration between, server, media bloc & projector.
|
|
Audio projection is based on Dolby with 5.1 or 7.1 channels, with known defects I listening area, localisation and acoustics.
|
Audio system based on ‘acoustic quadraphony’ that recreate the original acoustic ambience.
|
|
Digital cinematography is chosen for economic reasons and lacks a ‘film look’ sought by directors. Format conversion generates image degradation.
|
Digital systems recreating ‘film look’. Digital format conversion (de-interlacing, frame-rate change, resolution change) without loss of image quality, based on a variational approach using Partial Differential equations.
|
Turning to the wider industrial and social policy concerns of the IST, IP-RACINE addresses the concern to ensure European leadership in the generic and applied technologies at the heart of the knowledge economy and to increase innovation and competitiveness in European businesses and industry. IP-RACINE is designed to reinforce the position of Europe’s film industry, its related service providers and manufacturers, in the face of growing competition from the Hollywood-based international movie industry. In a rare reversal of the usual role, in the dawn of the digital cinema process, Europe has been ahead in the provision of digital technologies and services to the US-dominated content creation (movie-making) and distribution process. This, however, is a volatile situation: as Digital Cinema moves from the experimental stage and becomes industrialised, European providers are in increasing danger of being squeezed by the industrial demands of Hollywood and need greater innovation to remain competitive.
In keeping with the aim of building a value chain across a sector, IP-RACINE is structured to include the providers of technology components, platforms, to applications and services, bringing together different types of constituencies from the IST user and supply industry, academic research labs, large and small companies. SMEs play a vital role in the development of new visions in IST and transforming them into business assets: two of IP-RACINE’s eight manufacturer partners are SMEs. The project also recognises the socio-economic dimension, with provision for research into the needs of both professional and general public users, and will build an international advisory group to ensure the involvement of the entire spectrum of industry professionals.
The IST thematic priority hopes to contribute to realising European policies for dynamic knowledge economy capable of sustainable growth, with more and better jobs. The film and audiovisual sector is widely recognised as one of the prime areas, with digital technologies playing a major effect. In 2002 the International Labour Organisation stated: For some occupational groups, particularly those engaged in providing creative content, the multimedia revolution promises tremendous growth in opportunities for work as distribution channels multiply. Employment in the production of films and audiovisual products in 1995 stood at over 850,000 people in Europe, compared to only 630,000 in 1985. The American motion picture industry employed more than 610,000 workers in mid-1999, compared with 221,000 in 1985, and now employs considerably more people than the aerospace industry. Some of that growth can be attributed to technology-related work in fields such as computer-generated digital production, visual special effects technologies, and systems and network management. At least one observer believes that by the year 2010, films, multimedia and television will be the single largest employer in Europe…. Many future jobs will be based on technology which is today in its infancy; these jobs will call for undreamed-of skills.
IP-RACINE will help strengthen the European Research Area by training a new generation of researchers to create future technologies, and foster new user skills through the development of model syllabuses and professional training courses in collaboration between Universities, technology providers and users.
Another aspect is the delivery of a far more satisfying experience to cinemagoers in remote areas and small countries. The linguistic ‘agility’ of the digital data flow and increased programming flexibility will make it possible to serve cultural and linguistic minorities.
|
