Score System
What is VCX score —
The VCX score is a numeric indicator of the camera value of mobile phones and products and is based on an objective assessment of equipment.
The range is designed in such a way that a value of 100 means that the device yields the best possible result in every metric that is achievable with today’s camera technology.
How is it designed?
VCX-Forum adopts various scoring methods for each metric. The VCX Score is fundamentally designed to keep subjective influences as well as human errors as low as possible. It does not contain any visual assessment or other subjective factors.
The only subjective component is the weighting, that is, the decision of which metric is more important for the overall performance compared to others. This weighting has been accurately determined by a group of experts and it is identical for every device, so the comparison between devices is fixed and not influenced by individual opinions.
Selection of parameters, LGC and HGC, formulae, and weighting are based on physical facts, surveys, and long-term experiences. In version 2020, VCX test measurement consists of more than 300 test metrics. This may be updated or be excluded for every new version, in order to reflect the trend of the user experience.
The entire analysis is only based on the captured images of test charts and scenes under defined lighting conditions and the analysis algorithms are applied to these images. For each of the metrics used, a single score is calculated via bespoke algorithms/formulae developed specifically for VCX Forum by the members, derived from use-case studies. Based on a fixed algorithm, the score is calculated using the numerical results. The total score is a weighted sum of the individual scores.
Why is it objective?
The testing is accomplished by an independent test lab, the evaluation is automated and 100% objective. It does not contain any visual assessment or other subjective components. All devices are tested under the same conditions under multiple illumination levels and use cases. All images are analyzed without any human interaction. The same parameters, LGC and HGC, formulae, and weighting are applied to all devices in exactly the same way. The measured results are converted into the VCX score in exactly the same way. To prevent manufacturers from cheating, the detailed scoring process is limited to members who have to sign the association code with ethical standards.
To be improved
The weighting of the different aspects of the image quality is the result of a case study on how mobile phones are used as well as internal research. It correlates well with the outcome of other independent studies. The transformation of metrics into scores is performed under the definition of a theoretical worst and theoretical best value. The scaling is performed in different ways between the extreme points, depending on the metric itself.
For some metrics, the correlation between “metric” and “influence in image quality” is linear, so the score is a linear function of the metric. This would be in the case of a simple “the higher the better” or “the lower the better” assumption. For others, this assumption is not true. Some metrics require a different approach to the one previously mentioned because it would not reflect the perceived quality. Sharpening is a good example of this behavior. No sharpening is not beneficial for the image quality, as an image would appear flat. At the same time, too much sharpening very quickly results in an artificial and unpleasant. So, there is a “sweet spot” below or above which leads to a reduction in the score.
We regularly check the latest development in the camera industry and update every year the test procedure, the score generation process as well as the weighting of the scoring system with the continuous evolution of technology. e.g. it does not reflect the improvements in camera performance or when new technologies need to be included in the procedure.
Mission
Test procedures need to evolve and keep track of technological development. To make sure that VCX produces meaningful results in the future the standard department meets at least three times a year and discusses and implements new test procedures. On a year basis, VCX tries to update the whole procedure and take it to the next generation.
With its effort, VCX offers an objective consumer-oriented test procedure for mobile phone cameras that are widely accepted and used in the industry and known for its high-quality results.
What has been updated?
From Version 2019(1.5) to 2020
Tool: When VCX started as a small hobby project on a single PC, little did the team envisage that it would grow to be the industry standard that it is today. The weapon of choice at that time was Microsoft excel which donned the role of a prototyping and modeling tool with macros and formulae forming the core logic. Although this worked well in a limited set up up to a point, given the complexity and inherent limitations, VCX-Forum has moved to a more open, flexible, expandable, and a robust open-source platform with a multi-team collaborative approach where experts from around the globe can participate. This makes VCX development accessible on all popular operating systems "Python combines remarkable power with very clear syntax. It has interfaces to many system calls and libraries, as well as to various window systems, and is extensible in C or C++." Python being an open-source platform enables anyone who wants to contribute to the code of VCX to do so unencumbered by the burdens of licensing and unnecessary software fees. This might not seem to a big deal for the end customer, but strengthening the platform on which VCX is built on is a very important milestone. This platform is planned to be expanded further to facilitate the integration of back-end database, website automation, membership and credential management apart from being secure and state-of-the-art.
Image Quality: Updated metric to score with a new approach and formulae. Imaging technology and implementation in the smartphone arena have grown leaps and bounds from whence the VCX 1.5 was formulated. Customer’s expectation from their smartphone has also risen which meant a new approach to metrics and scoring. Over the last several years VCX-Forum has been the central hub where experts in the field have brought their experience back, participated in blind-tests, made recommendations, presented evidence and produced a new set to metrics and scoring that reflects current user expectation. More than 20 companies from all over the world participated in an exercise to finalize these metrics. There were no parameters in VCX V1.5 that were untouched; each and every one of them was relooked, reviewed and updated for Version 2020.
Improved-
Dynamic Range: Given the vast improvement in sensor and display technology in capturing and displaying dynamic range, along with user expectation the methods of measuring the ability of a smartphone to capture dynamic range had to be updated. VCX 2020 improved upon the V1.5 by employing a high contrast back-lit target instead of a reflective TE42 chart. This updated methodology closely matches the user expectation and brings forth a more realistic measurement.
Light performance: large pixels and innovative ways to capture light and reduce noise has given rise to a crop of top-shelf smartphones that capture images in low light that was hitherto the realm of large-bodied SLRs/Mirrorless cameras. The low-light tests and metrics incorporated in v1.5 were not only deemed deprecated to differentiate these top crop cameras but could not match with the evolving user expectation. In V2020, this has been addressed by a new gamut of extended low-light testing and appropriate metrics to reflect the same.
Video: In the recent past many have used the video feature on their phones to start a revolution. VCX v2020 takes a step forward in that direction to add tests and metrics. As of this version, a static keyframe is analyzed for imaging metrics that could give the user adequate guidance into the video performance of the phone. VCX-forum is committed to expanding this in future versions. (start the analysis of video performance in Version 2020 this is image quality from frames only)
Selfie: Use a single chart in size A460 (added image quality check of selfie camera based on the same target and light condition) in VCX v2020 the sophisticated tests and metrics have been carried over to the front-facing camera by introducing the same target and lighting conditions. This brings the score and rating on par and in harmony with the rear-facing camera which was necessary given the increase in priority of this feature since the last release, thanks to increasing popularity social media platforms like Instagram and others.
Motion: A key element of any picture-taking process is image stabilization when capturing the picture to this end VCX has built on the latest research by Apple research which is now part of ISO standard. This research goes in-depth on the kinesthetic of human hand movement and compensating for that as part of the image-stabilization apparatus in the smartphone. To make testing more effective additional metrics have been added which necessitated the addition of dead-leaves and Siemens-star modules to the test chart. A newly developed TE42-LL was employed apart from adding delta-Acutance Siemens and delta Acutance dead leaves as part of the metric ensample. Further Optimization was done by removing the Edge width parameter and harmonizing the lighting with other measurements (250 Lux + 10 Lux)
Timing: as listed below.
• Switched to the described procedure (minimizing AF-Failure rate with delay for focus)
• Removed “ShutterLag” and “ResponseTime-Startup” which are affected by the test person
Test score:
• Rear Camera
• Image Quality ( o Bright IQ / DR, o Mid IQ / DR, o Low IQ / DR, Flash, o Zoom - Bright, Mid, Low, o Low Light Performance)
• Handling (o Motion, o Timing)
• Video (o Bright, o Mid, o low)
• Selfie ( o bright, o Mid, o Low)
From Version 1.0 to 2019(1.5)
Changed weighting distribution of the main shooting conditions (bright, mid, low, flash, zoom).
Clear separation between the main shooting conditions by taking “pinch zoom” out of the VCX score.
Consistent weighting throughout the main shooting conditions in order to make the individual scores for each shooting condition more intuitive and comparable.
Adjustment of HGC and LGC and consistent HGC and LGC throughout the main shooting conditions, except for resolution and texture related parameters in the shooting condition “zoom”. The reason is that most devices currently only offer digital zoom and therefore HGC and LGC are set lower for zoom.
Revision of formulae, including introduction of formulae that lead to point deduction in case the result is way below user acceptance (resolution and texture related parameters only)
Revision of parameters that are used to create the score
Correction and adjustment of the whitepaper