This comprehensive and state-of-the art approach to video processing gives engineers and students a comprehensive introduction and includes full coverage of key applications: wireless video, video networks, video indexing and retrieval and use of video in speech processing. Containing all the essential methods in video processing alongside the latest standards, it is a complete resource for the professional engineer, researcher and graduate student.
- Numerous conceptual and numerical examples
- All the latest standards are thoroughly covered: MPEG-1, MPEG-2, MPEG-4, H. 264 and AVC
- Coverage of the latest techniques in video security
"Like its sister volume "The Essential Guide to Image Processing," Professor Bovik's Essential Guide to Video Processing provides a timely and comprehensive survey, with contributions from leading researchers in the area. Highly recommended for everyone with an interest in this fascinating and fast-moving field." -Prof. Bernd Girod, Stanford University, USA
- Edited by a leading person in the field who created the IEEE International Conference on Image Processing, with contributions from experts in their fields
- Numerous conceptual and numerical examples
- All the latest standards are thoroughly covered: MPEG-1, MPEG-2, MPEG-4, H. 264 and AVC
- Coverage of the latest techniques in video security
Inhaltsverzeichnis
1;Front Cover;1 2;The Essential Guide to Video Processing;4 3;Copyright Page;5 4;Table of Contents;6 5;Preface;18 6;About the Author;20 7;Chapter 1. Introduction to Digital Video Processing;22 7.1;1.1 Sampled Video;24 7.2;1.2 Video Transmission;27 7.3;1.3 Objectives of this Guide;28 7.4;1.4 Organization of the Guide;29 8;Chapter 2. Video Sampling and Interpolation;32 8.1;2.1 Introduction;32 8.2;2.2 Spatiotemporal Sampling Structures;33 8.3;2.3 Sampling and Reconstruction of Continuous Time-Varying Imagery;37 8.4;2.4 Sampling Structure Conversion;41 8.4.1;2.4.1 Frame-rate Conversion;43 8.4.2;2.4.2 Spatiotemporal Sampling Structure Conversion;47 8.5;2.5 Conclusion;49 8.6;References;49 8.7;Further Information;50 9;Chapter 3. Motion Detection and Estimation;52 9.1;3.1 Introduction;52 9.2;3.2 Notation and Preliminaries;53 9.2.1;3.2.1 Binary Hypothesis Testing;53 9.2.2;3.2.2 Markov Random Fields;54 9.2.3;3.2.3 MAP Estimation;55 9.2.4;3.2.4 Variational Formulations;55 9.3;3.3 Motion Detection;56 9.3.1;3.3.1 Hypothesis Testing with Fixed Threshold;57 9.3.2;3.3.2 Hypothesis Testing with Adaptive Threshold;59 9.3.3;3.3.3 MAP MRF Formulation;62 9.3.4;3.3.4 MAP Variational Formulation;63 9.3.5;3.3.5 Experimental Comparison of Motion Detection Methods;64 9.4;3.4 Motion Estimation;65 9.4.1;3.4.1 Motion Models;66 9.4.2;3.4.2 Estimation Criteria;72 9.4.3;3.4.3 Search Strategies;75 9.5;3.5 Practical Motion Estimation Algorithms;77 9.5.1;3.5.1 Global Motion Estimation;77 9.5.2;3.5.2 Block Matching;80 9.5.3;3.5.3 Phase Correlation;82 9.5.4;3.5.4 Optical Flow via Regularization;83 9.5.5;3.5.5 MAP Estimation of Dense Motion;84 9.5.6;3.5.6 Experimental Comparison of Motion Estimation Methods;85 9.6;3.6 Perspectives;86 9.7;3.7 Acknowledgments;87 9.8;References;87 10;Chapter 4. Video Enhancement and Restoration;90 10.1;4.1 Introduction;90 10.2;4.2 Spatiotemporal Noise Filtering;93 10.2.1;4.2.1 Linear Filters;93 10.2.2;4.2.2 Order-Statistic Filters;97 10.2.3;4.2.3 Multiresolution Filters;10
0 10.3;4.3 Coding Artifact Reduction;103 10.3.1;4.3.1 Artifact Reduction in the Spatial Domain;104 10.3.2;4.3.2 Artifact Reduction in the Frequency Domain;104 10.4;4.4 Blotch Detection and Removal;105 10.4.1;4.4.1 Blotch Detection;106 10.4.2;4.4.2 Motion Vector Repair and Interpolating Corrupted Intensities;109 10.4.3;4.4.3 Video Inpainting;112 10.4.4;4.4.4 Restoration in Conditions of Difficult Object Motion;113 10.5;4.5 Vinegar Syndrome Removal;115 10.6;4.6 Intensity Flicker Correction;118 10.6.1;4.6.1 Flicker Parameter Estimation;119 10.6.2;4.6.2 Estimation on Sequences with Motion;120 10.7;4.7 Kinescope Moiré Removal;122 10.8;4.8 Scratch Removal;124 10.9;4.9 Conclusions;125 10.10;Acknowledgements;126 10.11;References;126 11;Chapter 5. Video Stabilization and Mosaicing;130 11.1;5.1 Introduction;130 11.1.1;5.1.1 Video Stabilization;131 11.1.2;5.1.2 Outline;131 11.2;5.2 Biological Motivation: Insect Navigation;132 11.2.1;5.2.1 Centering Behavior and Collision Avoidance;132 11.2.2;5.2.2 Control of Flight Speed and Stabilization;133 11.2.3;5.2.3 Measuring Distance by Integrating Optical Flow;133 11.3;5.3 Camera Model and Image Motion Model;134 11.3.1;5.3.1 Camera Model;134 11.3.2;5.3.2 Effect of Camera Motion;135 11.3.3;5.3.3 Image Features;136 11.3.4;5.3.4 Structure from Motion;137 11.3.5;5.3.5 Feature based Algorithms;138 11.4;5.4 Flow-Based Approaches;139 11.4.1;5.4.1 Global Flow Models;139 11.4.2;5.4.2 Flow-Based Algorithm;140 11.5;5.5 Stabilization and Mosaicing;143 11.5.1;5.5.1 Video Mosaicing;145 11.6;5.6 Stabilization and Mosaicing with Additional Information;148 11.6.1;5.6.1 VIVID Metadata;148 11.6.2;5.6.2 Stabilization with Metadata;149 11.6.3;5.6.3 Inertial Measurements;151 11.6.4;5.6.4 Stabilization with Inertial Measurements;151 11.6.5;5.6.5 Motion Segmentation;155 11.7;5.7 Motion Super-resolution;155 11.8;5.8 Three-dimensional Stabilization;158 11.9;5.9 Summary;158 11.10;Acknowledgements;159 11.11;References;159 12;Chapter 6. Video Segmentation;162 12.1
;6.1 Introduction;162 12.2;6.2 Scene Change Detection;163 12.3;6.3 Spatiotemporal Change Detection;165 12.3.1;6.3.1 Spatial Change Detection Using Two Frames;165 12.3.2;6.3.2 Temporal Integration;166 12.3.3;6.3.3 Combination with Spatial Segmentation;167 12.4;6.4 Motion Segmentation;167 12.4.1;6.4.1 Dominant Motion Segmentation;168 12.4.2;6.4.2 Multiple Motion Segmentation;171 12.5;6.5 Simultaneous Motion Estimation and Segmentation;181 12.5.1;6.5.1 Motion-Field Model and MAP Framework;182 12.5.2;6.5.2 Two-Step Iteration Algorithm;183 12.6;6.6 Semantic Video Object Segmentation;185 12.6.1;6.6.1 Chroma-Keying;185 12.6.2;6.6.2 Semiautomatic Segmentation;185 12.7;6.7 Examples;186 12.8;6.8 Performance Evaluation of Video Segmentation;191 12.9;Acknowledgments;191 12.10;References;191 13;Chapter 7. Motion Tracking in Video;196 13.1;7.1 Introduction;196 13.2;7.2 Rigid Object Tracking;201 13.2.1;7.2.1 2D Rigid Object Tracking;201 13.2.2;7.2.2 3D Rigid Object Tracking;226 13.3;7.3 Articulated Object Tracking;230 13.3.1;7.3.1 3D Articulated Object Tracking;230 13.3.2;7.3.2 2D Articulated Object Tracking;240 13.4;References;241 14;Chapter 8. Basic Transform Video Coding;252 14.1;8.1 Introduction to Video Compression;253 14.2;8.2 Video Compression Application Requirements;258 14.3;8.3 Digital Video Signals and Formats;262 14.3.1;8.3.1 Sampling of Analog Video Signals;262 14.3.2;8.3.2 Digital Video Formats;264 14.4;8.4 Video Compression Techniques;266 14.4.1;8.4.1 Entropy and Predictive Coding;267 14.4.2;8.4.2 Block Transform CodingThe DCT;269 14.4.3;8.4.3 Quantization;271 14.4.4;8.4.4 MC and Estimation;274 14.5;8.5 Transform Coding: Introduction to the Video Encoding Standards;277 14.5.1;8.5.1 Transform Coding Standard Example: The H.261 Video Encoder;279 14.6;8.6 Closing Remarks;286 14.7;References;286 15;Chapter 9. MPEG-1 and MPEG-2 Video Standards;288 15.1;9.1 MPEG-1 Video Coding Standard;288 15.1.1;9.1.1 Introduction;288 15.1.2;9.1.2 MPEG-1 Video Coding versus H.261;289 15.
1.3;9.1.3 MPEG-1 Video Structure;291 15.1.4;9.1.4 Summary of the Major Differences between MPEG-1 Video and H.261;295 15.1.5;9.1.5 Simulation Model;296 15.1.6;9.1.6 MPEG-1 Video Bit-Stream Structures;297 15.1.7;9.1.7 Summary;298 15.2;9.2 MPEG-2 Video Coding Standard;298 15.2.1;9.2.1 Introduction;298 15.2.2;9.2.2 MPEG-2 Profiles and Levels;300 15.2.3;9.2.3 MPEG-2 Video Input Resolutions and Formats;301 15.2.4;9.2.4 MPEG-2 Video Coding Standard Compared to MPEG-1;302 15.2.5;9.2.5 Scalable Coding;307 15.2.6;9.2.6 Data Partitioning;309 15.2.7;9.2.7 Other Tools for Error Resilience;310 15.2.8;9.2.8 Test Model;310 15.2.9;9.2.9 MPEG-2 Video and System Bit-Stream Structures;311 15.2.10;9.2.10 Summary;312 15.3;References;313 16;Chapter 10. MPEG-4 Visual and H.264/AVC: Standards for Modern Digital Video;316 16.1;10.1 Introduction;316 16.2;10.2 Terminology;317 16.3;10.3 MPEG-4 Part 2;319 16.3.1;10.3.1 Object-based Representation;319 16.3.2;10.3.2 Video Object Coding;320 16.3.3;10.3.3 Mesh Object Coding;325 16.3.4;10.3.4 Model-based Coding;326 16.3.5;10.3.5 Still Texture Coding;328 16.3.6;10.3.6 Scalability;328 16.3.7;10.3.7 Error Resilience;329 16.3.8;10.3.8 MPEG-4 Part 2 Profiles;329 16.4;10.4 MPEG-4 Part 10: H.264/AVC;331 16.4.1;10.4.1 H.264/AVC Video Coding Layer: Technical Overview;331 16.4.2;10.4.2 Profiles;343 16.5;10.5 MPEG-4 Compression Performance;344 16.5.1;10.5.1 MPEG-4 Part 2;344 16.5.2;10.5.2 MPEG-4 Part 10: H.264/AVC;346 16.6;10.6 MPEG-4 Video Applications;347 16.7;10.7 Conclusions and Outlook;348 16.8;Acknowledgment;349 16.9;References;349 17;Chapter 11. Interframe Subband/Wavelet Scalable Video Coding;352 17.1;11.1 Introduction;352 17.2;11.2 Motion Estimation and Compensation for MCTF;354 17.2.1;11.2.1 Connected and Unconnected Blocks;356 17.2.2;11.2.2 Using Chroma for Motion Estimation;358 17.2.3;11.2.3 Improving the Haar MCTF;359 17.3;11.3 New Haar MCTF;363 17.3.1;11.3.1 Overlapped Block Motion Compensation;365 17.3.2;11.3.2 Scalable Motion Vector Coding;366
17.4;11.4 EZBC Coder;369 17.4.1;11.4.1 Coding Process;370 17.4.2;11.4.2 Context Modeling;372 17.4.3;11.4.3 Scalability;373 17.4.4;11.4.4 Packetization;374 17.4.5;11.4.5 Frequency Roll-Off;374 17.5;11.5 Extension to LeGall and Tabatabai 5/3 Filtering;376 17.6;11.6 Objective and Visual Comparisons;378 17.6.1;11.6.1 Some Visual Results;378 17.7;11.7 Multiple Adaptations;381 17.8;11.8 Related Coders;382 17.9;11.9 Conclusions;384 17.10;References;384 18;Chapter 12. Digital Video Transcoding;388 18.1;12.1 Introduction;388 18.2;12.2 Video Transcoding for Bit Rate Reduction;390 18.2.1;12.2.1 Transcoding of Intracoded Frame;391 18.2.2;12.2.2 Transcoding of Intercoded Frame;393 18.2.3;12.2.3 Fast Video-Transcoding Architectures;393 18.2.4;12.2.4 DCT Domain IMC;396 18.3;12.3 Heterogeneous Video Transcoding;398 18.3.1;12.3.1 MV Estimation for Spatial Resolution Reduction;400 18.3.2;12.3.2 MV Estimation for Temporal Resolution Reduction;401 18.3.3;12.3.3 Spatial Resolution Reduction;402 18.3.4;12.3.4 Macro-block-Coding Type Decision;404 18.4;12.4 Bit Rate Control in Video Transcoding;404 18.5;12.5 Error-Resilient Video Transcoding;405 18.6;12.6 Concluding Remarks;406 18.7;References;407 19;Chapter 13. Embedded Video Codecs;410 19.1;13.1 Introduction;410 19.2;13.2 Block-Based Video Coding;412 19.3;13.3 Embedded Video Codec Requirements and Constraints;414 19.4;13.4 Embedded Video Codec Design Flow;418 19.4.1;13.4.1 Understanding the Chip Architecture;418 19.4.2;13.4.2 Understanding the Codec Algorithms;421 19.4.3;13.4.3 Modularity and APIs Definitions;422 19.4.4;13.4.4 Reference Codec Software Development in Golden C;425 19.4.5;13.4.5 Platform-Specific Development and Porting;426 19.4.6;13.4.6 Kernel Optimization and Integration;427 19.4.7;13.4.7 Concurrent Processing;428 19.4.8;13.4.8 Overall Optimization;429 19.4.9;13.4.9 Stress and Conformance Testing;431 19.5;13.5 New Trends;432 19.6;13.6 Summary;434 19.7;References;435 20;Chapter 14. Video Quality Assessment;438 20.1;14.1 In
troduction;438 20.2;14.2 HVS Modeling Based Methods;440 20.3;14.3 Feature Based Methods;443 20.3.1;14.3.1 VQM;443 20.4;14.4 Motion Modeling Based Methods;446 20.5;14.5 Performance;452 20.6;14.6 Conclusions;454 20.7;References;455 21;Chapter 15. A Unified Framework for Video Indexing, Summarization, Browsing, and Retrieval;458 21.1;15.1 Introduction;458 21.1.1;15.1.1 Content Categories;459 21.1.2;15.1.2 Storage and Compression;460 21.1.3;15.1.3 Terminology;461 21.2;15.2 Image and Video Features;463 21.2.1;15.2.1 Statistical Features;463 21.2.2;15.2.2 Compressed-Domain Features;471 21.2.3;15.2.3 Content-Based Features;473 21.3;15.3 Video Analysis;480 21.3.1;15.3.1 Shot Boundary Detection;480 21.3.2;15.3.2 Key-Frame Extraction;480 21.3.3;15.3.3 Play/Break Segmentation;481 21.3.4;15.3.4 Audio Marker Detection;481 21.3.5;15.3.5 Video Marker Detection;481 21.4;15.4 Video Representation;481 21.4.1;15.4.1 Video Representation for Scripted Content;481 21.4.2;15.4.2 Video Representation for Unscripted Content;483 21.5;15.5 Video Browsing;484 21.5.1;15.5.1 Video Browsing Using ToC-Based Summary;484 21.5.2;15.5.2 Video Browsing Using Highlights-Based Summary;484 21.6;15.6 Video Retrieval;484 21.6.1;15.6.1 Feature-Based Retrieval (Statistical and Compressed);485 21.6.2;15.6.2 Content-Based Retrieval;485 21.6.3;15.6.3 Relevance Feedback;486 21.6.4;15.6.4 Query-Concept Learner;487 21.6.5;15.6.5 Efficient Annotation through Active Learning;487 21.6.6;15.6.6 Considerations in Multimedia Databases;488 21.7;15.7 A Unified Framework for Indexing, Summarization, Browsing, and Retrieval;489 21.8;15.8 Conclusions and Promising Research Directions;490 21.9;Acknowledgment;491 21.10;References;491 22;Chapter 16. Video Communication Networks;494 22.1;16.1 Introduction;495 22.2;16.2 Video Compression Standards;496 22.2.1;16.2.1 Introduction;496 22.2.2;16.2.2 Overview;497 22.2.3;16.2.3 MPEG-2 Video Compression Standard;499 22.2.4;16.2.4 MPEG-2 Systems Standard;500 22.3;16.3 Video Communication
Networks;506 22.3.1;16.3.1 Introduction;506 22.3.2;16.3.2 Hybrid Fiber-Coax Networks;507 22.3.3;16.3.3 Digital Subscriber Loop;508 22.3.4;16.3.4 Wireless Networks;509 22.3.5;16.3.5 Fiber Optics;512 22.3.6;16.3.6 Integrated Services Digital Network;513 22.3.7;16.3.7 ATM Networks;513 22.4;16.4 Internet Protocol Networks;520 22.4.1;16.4.1 Introduction;520 22.4.2;16.4.2 Multicast Backbone;523 22.4.3;16.4.3 Real-Time Transport Protocol;524 22.4.4;16.4.4 Real-Time Transport Control Protocol;532 22.4.5;16.4.5 Real-Time Transport Streaming Protocol;538 22.4.6;16.4.6 H.323;539 22.4.7;16.4.7 Session Initiation Protocol;541 22.4.8;16.4.8 Integrated ServicesResource Reservation Protocol;542 22.4.9;16.4.9 Differentiated ServicesDiffServ;544 22.5;16.5 Summary;546 22.6;References;546 23;Chapter 17. Video Security and Protection;548 23.1;17.1 Introduction;548 23.2;17.2 Video Encryption;548 23.2.1;17.2.1 Candidate Domains for Encrypting Multimedia;550 23.2.2;17.2.2 Building Blocks for Media Encryption;551 23.2.3;17.2.3 Security Evaluation of Media Encryption;557 23.2.4;17.2.4 Video Encryption System Design;560 23.3;17.3 Video Authentication;566 23.3.1;17.3.1 Background;566 23.3.2;17.3.2 Content Level Authentication;567 23.3.3;17.3.3 Stream Level Authentication;572 23.4;17.4 Video Fingerprinting for Traitor Tracing;574 23.4.1;17.4.1 The Background;575 23.4.2;17.4.2 Coded Fingerprinting;577 23.4.3;17.4.3 Experimental Results of Video Fingerprinting;584 23.4.4;17.4.4 Intravideo Collusion;587 23.5;References;587 24;Chapter 18. Wireless Video Streaming;592 24.1;18.1 Introduction;592 24.2;18.2 On Joint Source-Channel Coding;596 24.2.1;18.2.1 Rate-Distortion Theory;596 24.2.2;18.2.2 Operational Rate-Distortion Theory;597 24.2.3;18.2.3 Practical Constraints in Video Communications;598 24.2.4;18.2.4 Illustration;599 24.3;18.3 Video Compression and Transmission;600 24.3.1;18.3.1 Video Transmission System;600 24.3.2;18.3.2 Video Compression Basics;602 24.3.3;18.3.3 Channel Models;605 24.3.4;18
.3.4 End-to-End Distortion;606 24.3.5;18.3.5 Error Resilient Source Coding;608 24.4;18.4 Channel Coding;610 24.4.1;18.4.1 Forward Error Correction;611 24.4.2;18.4.2 Retransmission;613 24.5;18.5 Joint Source-Channel Coding;615 24.5.1;18.5.1 Problem Formulation;615 24.5.2;18.5.2 Internet Video Transmission;617 24.5.3;18.5.3 Wireless Video Transmission;620 24.6;18.6 Distributed Multimedia Communications;625 24.6.1;18.6.1 Video Streaming over Multiuser Networks;625 24.6.2;18.6.2 Mobile TV Standards;629 24.6.3;18.6.3 Peer-to-Peer Internet Video Broadcasting;631 24.6.4;18.6.4 Video Streaming over Multihop Wireless Networks;632 24.7;18.7 Discussion;633 24.8;References;634 25;Chapter 19. Video Surveillance;640 25.1;19.1 Introduction;640 25.2;19.2 Categorizing Applications, Target Scenes, and Video Analytics;641 25.2.1;19.2.1 Video Surveillance Applications;641 25.2.2;19.2.2 Video Surveillance Target Scenes;645 25.2.3;19.2.3 Video Analytics for Video Surveillance;647 25.3;19.3 Review of Video Analytic Algorithms;649 25.3.1;19.3.1 Motion and Change Detection;650 25.3.2;19.3.2 Object Detection;656 25.3.3;19.3.3 Object Tracking;658 25.3.4;19.3.4 Behavioral Analysis Tools;665 25.3.5;19.3.5 Gait Recognition;666 25.3.6;19.3.6 Face Recognition;669 25.4;19.4 Conclusion;669 25.5;References;670 26;Chapter 20. Face Recognition from Video;674 26.1;20.1 Introduction;674 26.2;20.2 Properties and Literature Review;676 26.2.1;20.2.1 Set of Observations;676 26.2.2;20.2.2 Temporal Continuity/Dynamics;682 26.2.3;20.2.3 3D Model;684 26.3;20.3 A General Framework of Probabilistic Identity Characterization;686 26.3.1;20.3.1 Recognition Setting and Issues;689 26.4;20.4 Instances of Probabilistic Identity Characterization;691 26.4.1;20.4.1 FR from a Group of Still Images;691 26.4.2;20.4.2 FR from a Video Sequence;696 26.5;20.5 A System Identification Approach;703 26.5.1;20.5.1 The ARMA Model;703 26.5.2;20.5.2 Framework for Recognition;704 26.5.3;20.5.3 Experiments, Results, and Discussion;705 26.6;
20.6 Conclusions;706 26.7;References;706 27;Chapter 21. Audiovisual Speech Processing;710 27.1;21.1 Introduction;710 27.2;21.2 Analysis of Visual Signals;712 27.2.1;21.2.1 Face Detection, Mouth, and Lip Tracking;713 27.2.2;21.2.2 Visual Features;715 27.2.3;21.2.3 Two Visual Feature Extraction Systems;719 27.3;21.3 Audiovisual Information Fusion;721 27.3.1;21.3.1 Speech Classes in Audiovisual Integration;721 27.3.2;21.3.2 Classifiers in Speech Applications;723 27.3.3;21.3.3 Feature and Classifier Fusion;725 27.4;21.4 Audiovisual Automatic Speech Recognition;728 27.4.1;21.4.1 Bimodal Corpora for ASR;729 27.4.2;21.4.2 Experimental Results;730 27.5;21.5 Audiovisual Speech Synthesis;733 27.5.1;21.5.1 Coarticulation Modeling;734 27.5.2;21.5.2 Facial Animation;735 27.5.3;21.5.3 Visual Text-to-Speech;738 27.5.4;21.5.4 Speech-to-Video Synthesis;739 27.5.5;21.5.5 Visual Speech Synthesis Evaluation;742 27.6;21.6 Audiovisual Speaker Recognition;744 27.7;21.7 Summary and Discussion;750 27.8;References;752 28;Index;760