Efficient Manipulation of Large Datasets on Heterogeneous Storage Systems

TitleEfficient Manipulation of Large Datasets on Heterogeneous Storage Systems
Publication TypeConference Papers
Year of Publication2002
AuthorsBeynon MD, Sussman A, Kurc T, Catalyurek U, Saltz J
Conference NameParallel and Distributed Processing Symposium, International
Date Published2002///
PublisherIEEE Computer Society
Conference LocationLos Alamitos, CA, USA
ISBN Number0-7695-1573-8
Keywordscomponent-based frameworks, data-intensive computing, load balancing

In this paper we are concerned with the efficient use of a collection of disk-based storage systems and computing platforms in a heterogeneous setting for retrieving and processing large scientific datasets. We demonstrate, in the context of a data-intensive visualization application, how heterogeneity affects performance and show a set of optimization techniques that can be used to improve performance in a component-based framework. In particular, we examine the application of parallelism via transparent copies of application components in the pipelined processing of data.