Face Image Quality Assessment Toolkit (fiqat)
The intended purpose of this toolkit is to facilitate face image quality assessment or face recognition experiments with a simple API. While e.g. the face detection and image preprocessing parts could be used to help with the training of new models, this toolkit doesn’t provide any training-specific utilities.
Toolkit features include:
Face detection (
fiqat.main_api.detect_faces()
): Including face bounding box detection and facial landmark detection.Primary face estimation (
fiqat.main_api.estimate_primary_faces()
): When multiple faces are detected in an image, but only one is relevant.Drawing face detector output (
fiqat.draw.draw_face_detector_output()
): To manually examine the face detector output.
Face image preprocessing (
fiqat.main_api.preprocess_images()
): Cropping and landmark-based alignment of face images for further processing.Face image quality assessment (
fiqat.main_api.assess_quality()
): Usually a preprocessed image’s quality in terms of utility for face recognition.Face recognition feature extraction (
fiqat.main_api.extract_face_recognition_features()
): Extracting a feature vector from a preprocessed image.Comparison (
fiqat.main_api.compute_comparison_scores()
): Comparing feature vector pairs for face recognition.
“Error versus Discard Characteristic” (EDC,
fiqat.edc
): Including “partial Area Under Curve” (pAUC) computation. Used to evaluate quality assessment algorithms with respect to specific face recognition models, see e.g. “Considerations on the Evaluation of Biometric Quality Assessment Algorithms” for details.Basic JPEG XL support: Currently only via the djxl/cjxl command line tools from https://github.com/libjxl/libjxl/releases. Both for loading & saving, the latter e.g. to facilitate more efficient lossless storage of preprocessed images.
Storage helper utility for computed data (
fiqat.storage
, fully optional): Using SQLite and a simple custom binary Python object serialization format with included numpy array support (fiqat.serialize
).This is used in example code, but it can easily be replaced with a custom implementation or be fully removed in your own code.
Example code for quick experiment setups: See the Examples section.
Custom method support: Methods can easily be registered for all main processing functions, see the create_edc_plot.py example in the Examples section.
The setup isn’t completely automatic, but the default setup without download times should take less than ten minutes.
Download and default configuration file setup
Download this repository.
Download and extract the external dependency package from https://cloud.h-da.de/s/a7pHZqBHptHaHRc (password
7W4Ei1FjhlV
), which mainly consists of the model files for various included methods, see the External dependency package locations section. You could omit these dependencies if you don’t want to use these methods.In the repository, create a copy of fiqat_example.toml at local/fiqat.toml. If you want to store this config file at another location, see
fiqat.config.load_config_data()
.Edit the local/fiqat.toml by changing the
models = "/path/to/fiqat/dependencies/"
path to your local path for the external method file dependency directory.
Python setup using an Anaconda environment
First create a new Anaconda environment. The following example names it fiqae
for “Face Image Quality Assessment Environment”:
conda create -n fiqae python=3.9
conda activate fiqae
Then install the default Python requirements used by the toolkit’s included methods (run this in the fiqat
repository directory):
pip install -r requirements.txt
Note: The default requirements.txt will install dependencies for CPU method execution.
If you want to run certain methods on a GPU, you may have to modify the package installation.
You could also omit certain dependencies (e.g. the insightface
and onnxruntime
packages) if you don’t need the corresponding methods.
The requirements.txt contains dependencies for some of the toolkit’s examples as well.
That’s why these dependencies currently are specified by this separate requirements.txt file,
instead of all dependencies being specified as part of the fiqat
package setup (in pyproject.toml).
Finally install fiqat
as an “editable” mode package in the new environment (run this in the fiqat
repository directory):
pip install --editable .
The fiqat
package should now be available within the fiqae
environment:
import fiqat
# ...
Special fd/retinaface
setup step
One additional setup step is required if you want to use the fd/retinaface
face detector method:
Run make
in insightface-f89ecaaa54/detection/RetinaFace of the external dependency package, using your fiqae
Python environment.
Documentation
You can build the documentation by running make html
in docs/, using your previously created Python environment,
which should create HTML documentation files in docs/_build/html/.
This uses the Sphinx
Python package, which is specified in the requirements.txt.
Examples
These standalone example scripts can be found in the repository’s examples directory:
load_all_methods.py: This example tries to initialize all methods included in the toolkit. If you successfully installed all dependencies, then all methods should be shown as available.
check_methods.py: This example will run all available face detector, face image quality assessment, and face recognition feature extractor methods. It needs an example image as input for the test runs. With the default configuration, the output of each method will be tested for consistency across multiple runs.
face_detection.py: This example will run the
fd/scrfd
face detector for a given input image, and save a new output image with drawn face detector information (such as the facial landmarks).create_edc_plot.py: This example goes through all the steps necessary to create a simple “Error versus Discard Characteristic” (EDC) plot that compares mutliple face image quality assessment algorithms with respect to face recognition performance. I.e. this example uses all method types of the
fiqat.main_api
(but not every included method).Intermediate data is stored during computation using the
fiqat.storage.StorageSqlite
helper class, and the script can be aborted and continued using the already computed data.The example expects input images stored so that the images of each subject are contained in a directory with a subject-specific name (i.e. the images’ directory names are the subject identifiers). For instance the extracted LFW “All images as gzipped tar file (173MB, md5sum a17d05bd522c52d84eca14327a23d494)” can be used as input.
A simple custom example method for quality assessment is defined (
_custom_fiqa_method
), registered (fiqat.registry.register_method()
), and used (_assess_quality(storage, image_dicts, fiqa_configs)
) in the example.
Included methods
The toolkit currently includes these methods:
fiqat.types.MethodType.FACE_DETECTOR
:fd/dlib
: Separate face detection and facial landmark detection (shape_predictor_68_face_landmarks.dat.bz2) using dlib.Configuration option
resize_to: Optional[ImageSize]
: Described below.Configuration option
detect_faces: bool
: IfTrue
, detect faces (i.e. face bounding boxes). Otherwise the whole image area is assumed to depict one face.True
by default.Configuration option
detect_landmarks: bool
: IfTrue
, detect facial landmarks for the detected faces.True
by default.
fd/mtcnn
: MTCNN from https://github.com/deepinsight/insightface/tree/60bb5829b1d76bfcec7930ce61c41dde26413279/deploy (of the “multi-task CNN” approach from “Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks”).Configuration option
resize_to: Optional[ImageSize]
: Described below.None
by default.Configuration option
device_config: DeviceConfig
: Described below.
fd/retinaface
: RetinaFace from https://github.com/deepinsight/insightface/tree/f89ecaaa547f12127165fc5b5aefca6d979b228a/detection/RetinaFace using the “RetinaFace-R50” model.Configuration option
resize_to: Optional[ImageSize]
: Described below.ImageSize(250, 250)
by default.Configuration option
device_config: DeviceConfig
: Described below.Configuration option
batch_size: int
: Described below.
fd/scrfd
: SCRFD from the insightface Python package.Configuration option
resize_to: Optional[ImageSize]
: Described below.ImageSize(160, 160)
by default.Configuration option
device_config: DeviceConfig
: Described below.Configuration option
model_name: str
: Which SCRFD model should be used, see https://github.com/deepinsight/insightface/tree/48282789fa2e440868b971a4b72fbec7fbc3c049/python-package#model-zoo.'buffalo_l'
by default.
fiqat.types.MethodType.PRIMARY_FACE_ESTIMATOR
:pfe/sccpfe
: “Size- and Center- and Confidence-based Primary Face Estimation”, which selects the primary face based on the size of the face ROI (fiqat.types.DetectedFace.roi
), the position of the ROI relative to the image center, and based on the face detector’s confidence values.Configuration option
use_roi: bool
: IfTrue
, theroi
data of thefiqat.types.FaceDetectorOutput.detected_faces
will be used for the estimation. The firstroi
estimation score factor for each candidate face is the minimum of the ROI’s width and height. The second factor is only computed ifinput_image_size
information is available in thefiqat.FaceDetectorOutput
, and favors ROIs that are closer to the image center. This second factor is meant to help with cases where multiple face ROIs with similar sizes andconfidence
values are detected.True
by default.Configuration option
use_landmarks: bool
: IfTrue
, the bounding box for thelandmarks
of thefiqat.types.FaceDetectorOutput.detected_faces
will be used as ROI information, with score factor computation as described foruse_roi
. If bothuse_roi
anduse_landmarks
isTrue
, thenroi
data will be used whenever available, andlandmarks
-based ROIs are used as fallback.True
by default.Configuration option
use_confidence: bool
: IfTrue
, the storedconfidence
values are used as an estimation score factor, normalized relative to the maximum value among thefiqat.types.FaceDetectorOutput.detected_faces
. If eitheruse_roi
oruse_landmarks
isTrue
as well, all factors are combined by multiplication.True
by default.
fiqat.types.MethodType.PREPROCESSOR
:prep/crop
: Simple preprocessing method that crops the image to thefiqat.types.DetectedFace.roi
, then resizes the cropped region to the output size (if specified).Configuration option
image_size: Optional[ImageSize]
: The size of the output image. If this isNone
, the cropped region will not be resized.None
by default.
prep/simt
: “Similarity transformation” face image preprocessing/alignment. It crops and aligns the facial image to five facial landmarks, two for the eyes, one of the tip of the nose, and two for the mouth corners, as produced e.g. byfd/retinaface
. This approach has been used e.g. in “ArcFace: Additive Angular Margin Loss for Deep Face Recognition”, “CosFace: Large Margin Cosine Loss for Deep Face Recognition”, “SphereFace: Deep Hypersphere Embedding for Face Recognition”.Configuration option
image_size: Optional[ImageSize]
: The size of the output image. If this isNone
, thefiqat.types.DetectedFace.roi
width/height will be used.None
by default.
fiqat.types.MethodType.FACE_IMAGE_QUALITY_ASSESSMENT_ALGORITHM
:fiqa/crfiqa
: CR-FIQA from https://github.com/fdbtrs/CR-FIQA using the “CR-FIQA(S)” or “CR-FIQA(L)” model.Configuration option
device_config: DeviceConfig
: Described below.Configuration option
batch_size: int
: Described below.Configuration option
model_type: str
: Specifies the model that should be used, which must be either'CR-FIQA(S)'
or'CR-FIQA(L)'
. This must be set explicitly.The model image input size is 112x112. Images are resized via
cv2.resize(image, (112, 112), interpolation=cv2.INTER_LINEAR)
.
fiqa/faceqnet
: FaceQnet from https://github.com/javier-hernandezo/FaceQnet using the “FaceQnet v0” or “FaceQnet v1” model.Configuration option
device_config: DeviceConfig
: Described below.Configuration option
batch_size: int
: Described below.Configuration option
model_type: str
: Specifies the model that should be used, which must be either'FaceQnet-v0'
or'FaceQnet-v1'
. This must be set explicitly.The model image input size is 224x224. Images are resized via
cv2.resize(image, (224, 224), interpolation=cv2.INTER_LINEAR)
.
fiqa/magface
: MagFace for quality assessment from https://github.com/IrvingMeng/MagFace using the iResNet100-MS1MV2 model (283MBmagface_epoch_00025.pth
with sha256sumcfeba792dada6f1f30d1e118aff077d493dd95dd76c77c30f57f90fd0164ad58
).Configuration option
device_config: DeviceConfig
: Described below.Configuration option
batch_size: int
: Described below.Configuration option
return_features_and_quality_score: bool
: IfTrue
, the method will output dictionaries with a ‘features’ (fiqat.types.FeatureVector
) and a quality_score entry each, instead of only returning afiqat.types.QualityScore
per input. This is an “experimental” option, proper MagFace face recognition method support will be added to the toolkit in a future version.False
by default.The model image input size is 112x112. Images are resized via
cv2.resize(image, (112, 112), interpolation=cv2.INTER_LINEAR)
.
fiqat.types.MethodType.FACE_RECOGNITION_FEATURE_EXTRACTOR
:fr/arcface
: ArcFace from https://github.com/deepinsight/insightface using the LResNet100E-IR,ArcFace@ms1m-refine-v2 model.Configuration option
device_config: DeviceConfig
: Described below.Configuration option
batch_size: int
: Described below.The model image input size is 112x112. Images are resized via
cv2.resize(image, (112, 112), interpolation=cv2.INTER_LINEAR)
.
fiqat.types.MethodType.COMPARISON_SCORE_COMPUTATION
:csc/arcface
: Computesfiqat.types.SimilarityScore
output for features computed byfr/arcface
(i.e. the cosine score in the range [-1, +1]).
Configuration options can be passed as keyword arguments to the fiqat.main_api()
functions.
Common options are:
resize_to:
Optional[fiqat.types.ImageSize
], forfiqat.types.MethodType.FACE_DETECTOR
methods: If set, the input images are resized to this size usingcv2.resize(..., interpolation=cv2.INTER_LINEAR)
, prior to detection.device_config:
fiqat.types.DeviceConfig
: The method support both CPU and GPU execution. Note that you may need to install Python packages that differ from the requirements.txt to enable GPU support.DeviceConfig('cpu', 0)
by default.batch_size: int
: The method supports processing input in batches. Larger batch sizes may accelerate processing, but may also require more memory (especially for GPU execution).1
by default.
License information
This “Face Image Quality Assessment Toolkit (fiqat)” itself is released under the MIT License (see the LICENSE
file).
Please note that many of the included methods are based on external projects with their own licenses:
fd/dlib
:Dlib itself (http://dlib.net/license.html): Boost Software License - Version 1.0
The model used for facial landmark detection (shape_predictor_68_face_landmarks.dat.bz2) has additional restrictions according to http://dlib.net/face_landmark_detection.py.html:
# You can get the trained model file from: # http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2. # Note that the license for the iBUG 300-W dataset excludes commercial use. # So you should contact Imperial College London to find out if it's OK for # you to use this model file in a commercial product.
fd/mtcnn
&fr/arcface
(from the InsightFace GitHub repository README.md at commit 60bb5829b1d76bfcec7930ce61c41dde26413279), as well asfd/retinaface
(from the InsightFace GitHub repository README.md at commit f89ecaaa547f12127165fc5b5aefca6d979b228a):“The code of InsightFace is released under the MIT License. There is no limitation for both acadmic and commercial usage.
The training data containing the annotation (and the models trained with these data) are available for non-commercial research purposes only.”
fd/scrfd
(from the InsightFace Python Library GitHub repository README.md):“The code of InsightFace Python Library is released under the MIT License. There is no limitation for both academic and commercial usage.
The pretrained models we provided with this library are available for non-commercial research purposes only, including both auto-downloading models and manual-downloading models.”
fiqa/crfiqa
(from the CR-FIQA GitHub repository README.md):“This project is licensed under the terms of the Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license. Copyright (c) 2021 Fraunhofer Institute for Computer Graphics Research IGD Darmstadt”
fiqa/faceqnet
(according to the author Javier Hernandez-Ortega in GitHub):“FaceQnet is free of use for non-profit projects. If you wish to use it in your project, just please give us the proper recognition as authors, for example citing the FaceQnet ICB paper.”
fiqa/magface
(according to the GitHub repository): Apache License 2.0
Directly used Python packages that are available on https://pypi.org/:
In
pyproject.toml
:colorama (state 2023-12-19): BSD-3-Clause License
opencv-python (state 2023-12-19): MIT License, Apache License 2.0, and other licenses for various parts (see link)
Pillow (state 2023-12-19): “Historical Permission Notice and Disclaimer” (HPND) License
termcolor (state 2023-12-19): MIT License
toml (state 2023-12-19): MIT License
tqdm (state 2023-12-19): MIT License, Mozilla Public License 2.0 (MPL 2.0)
In
requirements.txt
:dlib (state 2023-12-19): Boost Software License
insightface (state 2023-12-19): MIT License, with restrictions regarding pretrained models (see link and quotes in the method-specific list above)
mxnet-native (state 2023-12-19): Apache License 2.0
numpy (state 2023-12-19): BSD-3-Clause License and various for bundled parts (see link)
onnxruntime (state 2023-12-19): MIT License
plotly (state 2023-12-19): MIT License
scikit-image (state 2023-12-19): Various (see link)
scikit-learn (state 2023-12-19): BSD-3-Clause License
Sphinx (state 2023-12-19): Various (see link)
sphinx-rtd-theme (state 2023-12-19): MIT License
tensorflow-cpu (state 2023-12-19): Apache License 2.0
torch (state 2023-12-19): BSD-style license (see link)
torchvision (state 2023-12-19): BSD-3-Clause License
Known issues and limitations
The toolkit is currently patching a few deprecated numpy Python type aliases, in
fiqat.patch
, due to the dependencies of some of the included methods. But this probably should not be an issue for any experiment code.As noted above, the default requirements.txt only installs dependencies to support CPU execution of the included methods. The dependencies need to be manually adjusted if you want to run methods with GPU support on a GPU.
Included methods may print internal information as they run.
The documentation currently does not provide recommendations on which methods may be preferable.
External dependency package locations
For the included methods, the relevant files are located as follows within the external dependency package:
fd/dlib
: dlib/shape_predictor_68_face_landmarks.datfd/mtcnn
: insightface-60bb5829b1/deployfd/retinaface
: insightface-f89ecaaa54/detection/RetinaFace & insightface-f89ecaaa54/models/retinaface-R50fd/scrfd
: insightface/models/buffalo_lfiqa/crfiqa
: crfiqafiqa/faceqnet
: faceqnetfiqa/magface
: MagFacefr/arcface
: insightface-60bb5829b1/models