EFFICIENCY OF HIERARCHIC AGGLOMERATIVE CLUSTERING USING THE ICL DISTRIBUTED ARRAY PROCESSOR

• Published in: Journal of documentation Vol. 45; no. 1; pp. 1 - 24
• Main Authors: RASMUSSEN, EDIE M., WILLETT, PETER
• Format: Journal Article
• Language: English
• Published: London MCB UP Ltd 1989; Aslib, etc
• Subjects: Clustering; Computerised information retrieval; Computerized information storage and retrieval; Computerized subject indexing; File organization; ICL Distributed Array Processor; Information storage and retrieval; Information work; Searching; Software; Subject indexing; Technical services
• ISSN: 0022-0418; 1758-7379
• DOI: 10.1108/eb026836

The implementation of hierarchic agglomerative methods of cluster anlaysis for large datasets is very demanding of computational resources when implemented on conventional computers. The ICL Distributed Array Processor (DAP) allows many of the scanning and matching operations required in clustering to be carried out in parallel. Experiments are described using the single linkage and Ward's hierarchical agglomerative clustering methods on both real and simulated datasets. Clustering runs on the DAP are compared with the most efficient algorithms currently available implemented on an IBM 3083 BX. The DAP is found to be 2.9-7.9 times as fast as the IBM, the exact degree of speed-up depending on the size of the dataset, the clustering method, and the serial clustering algorithm that is used. An analysis of the cycle times of the two machines is presented which suggests that further, very substantial speed-ups could be obtained from array processors of this type if they were to be based on more powerful processing elements.