Compact Composite Descriptors

Several approaches for traditional content-based image retrieval have been proposed in the literature, ranging from global to local features and deep learning-based approaches. Global features, such as color, texture, and shape, are calculated to form a feature vector representation of an entire image. Such an approach has a low computational cost due to the single-feature vector and is very effective for specific retrieval tasks.

Compact Composite Descriptors (CCDs) are a set of low-level features that can be used as global or local descriptors to describe various types of multimedia information. The localized version of the CCDs is known as SIMPLE Descriptors. The set contains descriptors for 3 different types of images.

• A group of 3 descriptors (CEDD, CoMo, and FCTH) combines color and texture information in order to describe the visual content of the real-world color images.
• The brightness and Texture descriptor (BTDH) describes the visual content of grayscale images (as well as radiology medical images).
• Spatial Color Distribution descriptor SpCD describes artificially generated images (computer graphics, color sketches, etc.).

One can download open source implementations in C#, Java, MATLAB, and CUDA of the CCDs from this page.

Please note that this page only details the descriptors’ Global form. Details about the localized version of these descriptors are given Here.

Compact Composite Descriptors for Content-Based Image Retrieval. Basics, Concepts, Tools ISBN-10: 363937391X, 2011

Book Description: This book covers the state of the art in image indexing and retrieval techniques, particularly in recent trends and applications. It presents the basic notions and tools of content-based image description and retrieval, covering all significant aspects of image preprocessing, features extraction, similarity matching, and evaluation methods. Particular emphasis is given to recent computational intelligence techniques for producing compact content-based descriptors comprising color, texture, and spatial distribution information. Early and late fusion techniques are also used to improve retrieval results from large, distributed inhomogenous databases. The book reports on the international standards and provides an updated presentation of the current retrieval systems. Numerous utilities and techniques are implemented in software, provided as supplementary material under an open-source license agreement. The book is particularly useful for postgraduate students and researchers in the field of image retrieval who want to quickly elaborate and test state-of-the-art techniques and possibly incorporate them into their development.

[Download the Preface]

• Color and Edge Directivity Descriptor (CEDD), Color Moments (CoMo) and Fuzzy Color and Texture Histogram (FCTH)
• Brightness and Texture Directionality Histogram (BTDH)
• Spatial Color Distribution Descriptor (SpCD)
• Web Based Demo 

1. Compact Composite Descriptors for Natural Color Images:

Color and Edge Directivity Descriptor (CEDD), Color Moments (CoMo), and Fuzzy Color and Texture Histogram (FCTH)

The structure of these descriptors consists of n texture areas. In particular, each texture area is separated into 24 sub-regions, with each sub-region describing a color. CEDD, CoMo, and FCTH use the same color information, resulting from 2 fuzzy systems that map the colors of the image in a 24-color custom palette.

To extract texture information, CEDD uses a fuzzy version of the five digital filters proposed by the MPEG-7 EHD, forming 6 texture areas. CoMo descriptor uses Hu moment to capture the texture information forming again, 6 texture areas. On the other hand, FCTH uses the high-frequency bands of the Haar wavelet Transform into a fuzzy system, to form 8 texture areas.

When an image block (rectangular part of the image) interacts with the system that extracts CEDD, CoMo and FCTH, this section of the image simultaneously go across 2 units. The first unit, the color unit, classifies the image block into one of the 24 shades used by the system. Let the classification be in the color m, m E [0,23]. The second unit, the texture unit, classifies this image section in the texture area a, a E[0,n]. The image block is classified in the bin a X 24 +m. The process is repeated for all the image blocks of the image. On the completion of the process, the histogram is normalized within the interval [0:1] and quantized for binary representation in a three bits per bin quantization.

The most important characteristic of CEDD, CoMo, and FCTH is the very good results (kindly refer to the following table for MAP values) that they bring up in various well-known benchmarking image databases. Please note that this table illustrates is the performance of global descriptors. Follow this link to see the performance of the localized version of the CCDs descriptors.

Another important characteristic of CEDD, CoMo, and FCTH is the small size needed for indexing images. Both CEDD and CoMo require less than 54 bytes per image, while FCTH requires 72 bytes per image.

[*] Important note: There is no significant difference between these two versions of the descriptors (experiments conducted on UCID and Wang database). Ver 1.01 fixes minor bugs and slightly improves the efficiency of the descriptor. If you choose to update the descriptor in your implementation, please note that you must re-index your data.

[**] For academic purposes only

1.1 C# Instructions (CEDD and FCTH): 

In order to use them, add the dll as references in your c# project and then call the classes as:

(Highlight) Searching in more than 1.000.000 images with the CEDD feature

1.2 MATLAB Instructions (CEDD and FCTH): 

The source code is simple and easy to handle by all users. The main function has the task of extracting the CEDD descriptor from a given image.

• Download the MATLAB implementation of CEDD [Ver 1.0] [FAPO] [updated:1/27/2014]

function [ DescriptorVector ] = CEDD(ImageRGB)

1.3 Joint Composite Descriptor:

Observing the CEDD and FCTH results in various queries, it is easy to ascertain that in some of the queries, better retrieval results are achieved by using CEDD, while in others by using FCTH. Given that the CEDD descriptor and the FCTH descriptor for each image are available in the index file that the retrieval system uses, it should be asked if the 2 descriptors could be combined to achieve better retrieval results. Based on the fact that the color information given by the 2 descriptors comes from the same fuzzy system, we can assume that joining the descriptors will rely on the combining of texture areas carried by each descriptor. Therefore, uniting the CEDD and FCTH will lead to a new descriptor comprising a combination of CEDD and FCTH texture areas. The combined descriptor is called Joint Composite Descriptor (JCD).

• Download JCD as C# Source Code [Ver 1.0] [GNU GPL]

In order to use the JCD Source Code, add the CEDD and FCTH dlls, as references in your c# project, and then call the classes as:

1.4 How to Compare the descriptors:

Finally, in order to calculate the distance between two descriptors, use the following method (Tanimoto Coefficient)

Download an example application, in which we describe the retrieval procedure. Source Code included (C# Visual Studio 2008 Solution) [GNU GPL]

1.5 Java Version of the Descriptors:

A JAVA implementation of the descriptors (CEDD, CoMo, FCTH and JCD) is available through the LIRE library. LIRE (Lucene Image Retrieval) is an open-source library for content-based image retrieval, which means you can search for images that look similar. Besides providing multiple common and state-of-the-art retrieval mechanisms, LIRE allows for easy use on multiple platforms. LIRE is actively used for research, teaching, and commercial applications. Due to its modular nature, it can be used on the process level (e.g. index images and search) as well as on the image feature level. Developers and researchers can easily extend and modify LIRE to adapt it to their needs.

1.6 Discussion on the rotation invariance of CoMo

1.7 CUDA-based implementation of CEDD:

In order to execute this application, the latest version on the nVIDIA CUDA driver must be installed. A message stating “There is no device supporting CUDA” will appear during execution if CUDA’s GPU is not enabled or the respective driver is not up-to-date. The CUDA drivers can be found here.

Make sure to place the “cudart32_42_9.dll” file together with the application in the same directory and execute the latter using the Windows Command Prompt. Executing the application without any arguments will extract the CEDD descriptors for all images (bmp format) that are contained in the same directory as the application. Including the folder’s full path as an argument in the Command Prompt allows the user to extract the descriptors of images stored in directories other than the one the application is in. Please note that the application only works with bmp formatted images.

• Download CUDA based implementation [Ver 0.7 (beta)] [GNU GPL]
• Download the source code of the CUDA™-based implementation of CEDD [CUDA_Only] [Hybrid] [GNU GPL]

Preliminary results

Average Indexing Time for 1000 Images

More details, comparisons and other staff will be added soon. Please note that CUDA based implementation produce a lossless representation of the descriptor.

2. CCD for Medical Radiology Images: Brightness and Texture Directionality Histogram (BTDH)

A fuzzy rule-based scalable composite descriptor (BTDH) is a compact composite descriptor that can be used to index and retrieve radiology medical images. This descriptor uses brightness and texture characteristics as well as the spatial distribution of these characteristics in one compact 1D vector. The most important characteristic of the proposed descriptor is that its size adapts according to the storage capabilities of the application that is using it. This characteristic renders the descriptor appropriate for use in large medical (or grayscale) image databases. To extract the proposed descriptor, a two-unit fuzzy system is used. To extract the brightness information, a fuzzy unit classifies the brightness values of the image’s pixels into L_{Bright} clusters. The cluster centers are calculated using the Gustafson Kessel Fuzzy Classifier. The texture information embodied in the proposed descriptor comes from the Directionality histogram. This feature is part of the well-known Tamura texture features. Fractal Scanning method through the Hilbert Curve or the Z-Grid method is used to capture the spatial distribution of brightness and texture information. Seven approaches used in our experiments in order to evaluate the performance of this descriptor:

The approach displaying the best correlation between size and result is approach E5. [Download the retrieval results in several databases].

• Download BTDH as .NET DLL [ Ver 1.01 (Updated Aprils 1st 2013)] [GNU GPL]
• Download BTDH as c# Source Code [ Ver 1.01 (Updated Aprils 1st 2013)] [GNU GPL]
• Download an example application, in which we describe the retrieval procedure [Ver 1.00] [Ver 1.01] [GNU GPL]

2.1 Instructions:

In order to employ BTDH in your solution, add the dll as a reference in your c# project and then call the class as:

Details regarding BTDH descriptor can be found at the following paper:

3. CCD for Artificially Generated Images: Spatial Color Distribution Descriptor (SpCD)

The descriptor is not available at this time.