Test Environment

The VCX test measurement currently updated to the 2020 version. It has now included a selfie camera test, also rear camera test has been extended to cover various capturing environment as well as video. The image quality test of the rear camera consists of several parts: Sunny, Indoor, Night, Flashlights conditions, Zoom, Video, Selfie(each under 3 condition: sunny, indoor, night). Even more VCX-Forum members consider that taking pictures under low light conditions is challenging, hence it additionally tested under 6 different faint lightroom as an ‘Extended Low Light condition’. Still, you can find the camera performance test in the updated version.

Test condition

Variables are controlled and kept constant throughout the test. Any changes in a variable during an experiment or in between tests would invalidate the correlation of dependent variables to the independent variable, thus skewing the results.

Ideally one image contains all the information that we need for the various image quality aspects, e.g. dynamic range and sharpness.

Factory default settings are selected and the default camera app with automatic exposure control is used. The analysis is performed based on the final JPG image data using the camera internal image processing pipeline.

Image Quality

Metrics:

  • Resolution
  • Texture Loss
  • Sharpness
  • Colors
  • Visual Noise
  • Dynamic Range
  • Shading
  • Distortion

All measurements are obtained under controlled lighting situations. We obtain the metrics with exactly the same exposure setting.

The image quality evaluation based on the TE42 LL test chart is performed under different lighting conditions:

Bright - This condition is used as the Reference. It is performed with a brightness of 2000lux Day light D55.
Mid - Which results in an illumination level of 250lux Cool LED. This light condition reflects a normal indoor light situation without direct sunlight.
Low – A low light situation is the most challenging situation for a camera. In this case, we reduce the illumination to 10lux, warm LED.

Extended Low light – Additionally images captured under decreasing illumination. A reference at bright (see above) and then the images with the illumination of 10lux, 5 lux, 2.5 lux, 1 lux, 0.75 lux

Flash – If a situation is too dark, mobile phones mainly use LEDs to illuminate the scene. As a phone is rarely used in cases where there is absolutely no light, the flash is activated for this measurement while the scene is still illuminated with 10 lux (Night).
Zoom(4x) – To zoom onto a smaller object is a very common action user of mobile phone cameras performs. To evaluate the image quality of a zoomed image, we capture an image of the TE42 LL with 4x zoom at the mentioned conditions bright, mid and low. If the device provides an optical zoom, we use the optical zoom first and add digital zoom, if needed, to achieve a 4x zoom. For devices that only offer digital zoom (the standard case of the majority of today’s mobile phones), we use a 4x digital zoom. Devices that achieve less than 4x zoom will receive 0 points for this measurement condition. To perform the measurement, the distance between the chart and the camera is increased according to the zoom factor and the entire chart is captured.

Video The image quality evaluation of video is based on the TE42 LL test chart, an extracted image of a frame from the 10 seconds of the video is used for the calculation. The extracted frames for analysis are extracted using ffmpeg after at least 5s in the video file. This is also performed under three different measurement conditions: Sunny, Indoor, Night

Selfie The image quality evaluation based on the TE42 LL A460 test chart is performed under three different measurement conditions: Sunny, Indoor, Night

Timing Measurement

Metrics:

  • Shutter release time lag measurement images
  • Shooting time lag measurement images
  • Frame rate measurement images (Burst Mode compression)
  • AF failure rate measurement images

The metrics in the part “Response Time” are measured using the Image Engineering timing kit. The start of the exposure can still be determined, as it is the position of the first LED that is of interest.

 

Motion Control

STEVE is an abbreviation of “STabilization EValuation Equipment”. It is a mechanical apparatus that can shake the device under test in a defined way. That way we can simulate the motion introduced due to human tremor while capturing an image. The device simulates a two-axis motion (pitch and yaw). The camera under test has to reproduce a test target showing slanted edges. The chart is the TE261 of Image Engineering inside the AF-Box (also used for timing measurement) and the evaluation is done as described in ISO 20954.

Images for performance test are taken under two different light conditions: 800 Lux (Bright), 40 Lux (Low)

The analysis is made using software that has been shown to comply with the IQ-Analyzer and the VCX own tools. All the data which is extracted from the measurements will be computed without additional subjective analysis. Totally over 300 test metrics are engaged to calculate the final VCX score. The proportion of each metric is determined by the VCX-Forum members when generating a VCX version.

 

The process is updated annually with new versions. the professional and detailed information can be found on the white paper. If you have any further questions, please contact the VCX coordinator.