The Defense Systems Information Analysis Center (DSIAC) was asked to identify datasets available for use in programs for the Defense Advanced Research Projects Agency.  DSIAC consolidated information from a previous Technical Inquiry and open-source infrared (IR) imagery datasets to address the inquiry.  IR datasets included Forward Looking IR Cameras Automated Driver Assistance Systems IR Imagery dataset, South China University of Technology Far-Infrared Pedestrian dataset, Night Vision and Electronics Sensors Directorate Automated Target Recognition database, and Eglin Signature Data Center (ESDC) datasets.  The datasets contain annotated IR and visual imagery with associated metadata of commercial vehicles and personnel that can be used for artificial intelligence and machine language training as well as other purposes.  DSIAC also provided contact information for ESDC, examples of an ESDC data listing, and the ESDC Data Request form.

 

---

1.0  Introduction

The Defense Systems Information Analysis Center searched open-source documents to identify infrared (IR) imagery datasets/databases that can be used within the United States (U.S.) Department of Defense (DoD).  The results were compiled in the following report.

 

---

2.0 Government Databases and Datasets

2.1. Eglin Air Force Base 96^th Test Wing Eglin Signature Data Center (ESDC)

The ESDC is one of the larger U.S. DoD repositories of electro-optical/infrared (EO/IR) and radio frequency (RF) signature data for domestic and foreign tactical and strategic vehicles.  This repository also includes some aircraft, watercraft, and personnel imagery.  Most of the data are collected with commercial imaging EO/IR and radar systems designed for collection of test data, not actual vehicle operational sensors [1].

Some of the data in the systems are at the Controlled Unclassified Information (CUI) level, but much of the information is classified at confidential or higher levels.  Vehicle data include nonrunning, or cold, stationary vehicles and running, or hot, stationary and exercised vehicles.  Data are collected with multiple, co-bore sighted imagers/sensors from various aspects, such as altitudes and azimuths, over diurnal time periods for various backgrounds, environmental conditions, solar loading conditions, configurations, etc.  Typically, there are associated documentary (videos/stills), meteorological, and ground truth data available.  The data are calibrated, and sometimes the associated uncalibrated data are available.  An example dataset could include mid-wave infrared (MWIR) data on ground vehicle systems and personnel for ranges limited between 1 and 5 kilometers (km).  Significantly more data are available that have been collected from mobile and fixed tower systems, ground systems, and airborne systems at shorter and longer ranges, as well as other bandwidths [1].

To obtain any of the data, an ESDC Data Request form must be completed.  Contractors and industry partners must have it endorsed by their U.S. Government sponsor and then submit the form to 782TS.RNWI.ESDC@us.af.mil. ESDC contacts can be found in Table 1.

 

Table 1:  ESDC Contacts

Name	Contact Information
Jacques M. Perry	96 TW, 782 TS/RNWI 303 N 7th St., Bldg. 22 Eglin AFB, Florida 32542 (850) 882-3336, DSN 872-3336 NIPR: Jacques.Perry@us.af.mil SIPR: Jacques.M.Perry.civ@mail.smil.mil
Leah W. Christmas	Leah W. Christmas 96 TW, 782 TS/RNWI 303 N 7th St., Bldg. 22 Eglin AFB, Florida 32542 (850) 883-2295, DSN: 872-2295 NIPR: leah.christmas.1@us.af.mil SIPR: leah.w.christmas.civ@mail.smil.mil

 

2.2 Automated Target Recognition (ATR) Algorithm Development Image Database

DSIAC distributes the ATR database package, which contains data collected by the U.S. Army Night Vision and Electronics Sensors Directorate (NVESD) intended to support the ATR algorithm community.  It includes a collection of MWIR imagery for people, foreign military vehicles, and civilian vehicles at a variety of ranges and aspect angles.  This database provides a broad set of IR and visible imagery along with ground truth data for ATR algorithm development and training [2].

 

---

3.0 Open-Source Databases and Datasets

The following open-source databases contain annotated IR and visual imagery with associated metadata of commercial vehicles and personnel. The data are widely used for artificial intelligence and machine learning training.

3.1 Forward-Looking Infrared (FLIR) Automated Driver Assistance Systems (ADAS) Dataset

The FLIR dataset for ADAS training has over 10,000 annotated thermal images

of vehicles and personnel during both day and night conditions. The dataset allows developers to begin developing and training convolutional neural networks, allowing the automotive community to create the next generation of ADAS and driverless vehicle systems using cost-effective thermal cameras from FLIR [3].  The imagery dataset specifications can be found in Table 2 [4]. Figure 1 shows images from the FLIR ADAS and driverless vehicle algorithm training.

 

Table 2:  FLIR ADAS Dataset Specifications (Source: https://www.flir.com/oem/adas/adas-dataset-form/)

Content	Synched annotated thermal imagery and non-annotated RGB imagery for reference. Camera centerlines approximately 2 inches apart and collimated to minimize parallax.
Images	>14K total images with >10K from short video segments and random image samples, plus >4K BONUS images from a 140 second video.
Image Capture Refresh Rate	Recorded at 30Hz. Dataset sequences sampled at 2 frames/sec or 1 frame/second. Video annotations were performed at 30 frames/sec recording.
Frame Annotation Label Totals	10,228 total frames and 9,214 frames with bounding boxes. 1.       Person (28,151) 2.       Car (46,692) 3.       Bicycle (4,457) 4.       Dog (240) 5.       Other Vehicle (2,228)
Video Annotation Label Totals	4,224 total frames and 4,183 frames with bounding boxes. 1.       Person (21,965) 2.       Car (14,013) 3.       Bicycle (1,205) 4.       Dog (0) 5.       Other Vehicle (540)
Driving Conditions	Day (60%) and night (40%) driving on Santa Barbara, CA area streets and highways during November to May with clear to overcast weather.
Capture Camera Specifications	IR Tau2 640×512, 13mm f/1.0 (HFOV 45°, VFOV 37°) FLIR BlackFly (BFS-U3-51S5C-C) 1280×1024, Computar 4-8mm f/1.4-16 megapixel lens (FOV set to match Tau2)
Dataset File Format	1.       Thermal – 14-bit TIFF (no AGC) 2.       Thermal 8-bit JPEG (AGC applied) w/o bounding boxes embedded in images 3.       Thermal 8-bit JPEG (AGC applied) with bounding boxes embedded in images for viewing purposes 4.       RGB – 8-bit JPEG 5.       Annotations: JSON (MSCOCO format)
Sample Results	mAP scores were obtained for People (0.794), Bicycle (0.580), and Car (0.856) categories. http://cocodataset.org/#detection-eval was used for accuracy assessment criteria.
FLIR ADK Training and Development Settings	Use the FLIR ADK with default settings to begin data collection

 

 

Figure 1:  FLIR Thermal Dataset for ADAS and Driverless Vehicle Algorithm Training (Source:  https://www.flir.com/oem/adas/adas-dataset-form/ [5]).

 

3.2 South China University of Technology (SCUT) Far-Infrared (FIR) Pedestrian Dataset

The SCUT FIR Pedestrian Dataset was recently published on GitHub and is available to the public for download online.  It is a large, annotated, FIR pedestrian detection dataset consisting of 11 hours of image sequences at a rate of 25 hertz (Hz) collected from a vehicle driving through diverse traffic scenarios at a speed of less than 80 km per hour.  Image sequences were collected from 11 road sections under four different scenes in Guangzhou, China:  downtown, suburbs, expressway, and campus.  There are 211,011 annotated frames with 477,907 bounding boxes around 7,659 pedestrians [5].  Figure 2 shows imagery from this dataset.

 

Figure 2:  SCUT FIR Pedestrian Dataset Imagery (Source: https://github.com/SCUT-CV/SCUT_FIR_Pedestrian_Dataset).

 

 

---

References

[1] Eglin Signature Data Center.  Personal communication, 6 January 2020.

[2] Defense Systems Information Analysis Center. “ATR Algorithm Development Image Database.” https://www.dsiac.org/resources/research-materials/cds-dvds-databases-digital-files/atr-algorithm-development-image, 23 January 2020.

[3] Defense Systems Information Analysis Center. “FLIR Offers Free Thermal Data Set for Algorithm Training.” https://www.dsiac.org/resources/news/flir-offers-free-thermal-data-set-algorithm-training, 23 January 2020.

[4] FLIR Systems, Inc. “FREE FLIR Thermal Dataset for Algorithm Training.” https://www.flir.com/oem/adas/adas-dataset-form/, 3 February 2020.

[5] Armistead, S. Personal communication. Defense Systems Information Analysis Center, 13 January 2020.