WebCam Test

Starting with a rigorous evaluation methodology, VCX-WebCam 2023 evaluates camera performance in controlled conditions that replicate the kind of stressful environments devices are used in for video conferencing. Tests are built on up-to-date evaluation methods that work in many of the same ways modern image processing algorithms do. In addition to basic image quality concerns, tests are focused on common pain points reported by users and industry experts throughout the development process. With a special emphasis on face-present and face-moving scenes, auto-exposure, white balance, noise and spatial frequency performance, the VCX score provides a comprehensive measurement of camera imaging performance. A high VCX score means a high level of image quality.

In the test environment

The test environment, designed to reproduce common conditions for video conferencing and other types of video communication, uses a variety of light sources with varying brightness levels. The webcam studio chart is centered within the field of view to support various test procedures. This studio chart includes mannequins of different skin tones and a head-turn device which mimics a human turning their face away from the camera. Each test is repeated for light skin tone, dark skin tone, and no face present.

Evaluating camera performance in repeatable laboratory conditions that replicate many of the same challenges experienced by consumers “in the wild” allows for thorough, repeatable testing anywhere in the world. Cameras are tested in static and dynamic conditions with changing lights, movement within the scene, and challenging backdrops.

Testing is aimed at the “out of the box” experience, meaning that it is performed on devices that are set to factory default video mode. Devices are tested in “as-shipped” condition and in the orientation (landscape or portrait) intended for their use. Camera settings are set to default values during device preparation, with only resolution settings being changed during image capture. While some devices allow manual adjustment, of ISO, exposure, white balance, and other parameters, these settings remain unchanged during the test.

As the target for VCX-WebCam 2023 is video, images are captured as a video by a dedicated test application or the default video capture app for that platform. No special processing is allowed. After capture, videos are processed through a neutral approach that separates individual frames after the video has stabilized. These individual frames are then analyzed for the various KPI’s within the specification.

Tests are performed under a range of lighting conditions:

Bright Light

Bright Light - In bright light conditions, cameras are most likely to be used outside or near a window where bright outside light may be prevalent. Light sources are tested at 350 and 500lux.

o D50 – Many light sources in commercial building are designed to replicate this “sunny day” light, but at light levels that are reasonable for indoors.

o D65 – Similar to D50, but designed to replicate more of a “cloudy day” light.

Mid Light

Mid Light - Representing the more common “daytime” lighting seen in homes and businesses, we test these light sources at 80 and 250 lux.

o A – a warmer, yellowish color that commonly refers to incandescent lights. Today there are many other lighting technologies that replicate this same behavior.

o Warm White LED – A little cooler than A, but still “warm” in appearance. This is the common “white” lamp seen in many homes around the world.

o Cool White LED – a more blue color, this light source replicates commonly seen LED lamps used in homes and businesses around the world.

o D50 – Many light sources in commercial building are designed to replicate this “sunny day” light, but at light levels that are reasonable for indoors.

o D65 – Similar to D50, but designed to replicate more of a “cloudy day” light.

Low Light

Low Light – Chosen specifically to represent the most common light sources in homes and other low-light situations, we test these light sources at 10 and 20 lux.

o A - this is a warmer, yellowish color that commonly refers to incandescent lights. Today there are many other lighting technologies that replicate this same behavior.

o Cool White LED – on the cooler, bluer side, this light source replicates commonly seen LED lamps used in homes and businesses around the world.

Test Metrics

Video image quality is about more than just megapixels. Camera sensors, optics, software, hardware, and image processing algorithms (such as sharpness, noise reduction and color as well as dynamic behaviors like exposure and white balance) all have a key part to play. Test metrics are carefully chosen to evaluate basic image quality and go beyond to identify poor image quality behaviors that are most annoying to the end users.

The specification behind the VCX-WebCam 2023 benchmark consists of 24 tests across 6 areas of focus. Focus areas were selected based on pain points collected from member tuning and customer teams throughout the PC ecosystem. Special attention was paid to areas that have been difficult to test objectively, until now, such the convergence process for auto exposure and white balance as well as challenging AWB scenes such as memory color and moving backdrops.

The 6 key areas of focus for VCX-WebCam 2023 are:

Metrics:

Visual Noise – a way of measuring image noise similarly to the way the human eye does.
Contrast, Dynamic Range and Exposure – These three areas are often linked. How bright your image is, the range of what is “bright” and “dark” and how well details in dark areas are reproduced are all key areas that separate better from lesser designs.
Spatial Frequency Response – a way of measuring not just how “sharp” an image is, but how well image detail is preserved and carefully enhanced without adding unpleasant artifacts.
Color Reproduction – How well a camera reproduces colors, not just in a test chart but in different faces across many different test scenarios.
Image Uniformity – Sometimes called “shading”, a measure of how consistent brightness and color accuracy are across the entire image.
Framerate – How fast your camera responds to the real scene often depends solely on how often frames are captured. Thresholds for framerate in low and mid-light conditions focus on the places problems happen the most.

Of particular interest is the special attention paid to skin tone response throughout the VCX-WebCam 2023 test suite. VCX partnered with a prop house to develop custom, realistic mannequin heads modeled on real people in multiple skin tones. Nearly every test is repeated with these heads to challenge camera system behavior. VCX research has demonstrated there is a notable difference in how many camera systems respond to different skin colors.

We hope that by embedding this into the very core of the test philosophy, VCX-WebCam can help deliver a better experience for everyone.