Abstract

Data mining is the exploration and analysis of large quantities of data to discover meaningful patterns and rules. Mining frequent itemsets plays an essential role in many data mining tasks, which attempts to find interesting associations or correlations among a large set of data items. Efficient discovery of frequent large itemsets and its dual problem of mining association rules are well studied and efficient solution techniques have been developed and deployed in data analysis and mining tools. When new transactions are added to the dataset, it is important to maintain such discovered patterns and rules without requiring processing the whole dataset and re-computing from scratch. In this research, we first focus on the maintenance problem and propose an in-memory technique to identify frequent large itemsets when the data set grows by addition of new transactions. The basic solution idea is to identify and use negative borders for maintenance. We then use this idea and develop a divide-and-conquer technique, based on partitioning , to compute frequent itemsets in large datasets, which do not fit into the main memory. Our experimental results show that the proposed techniques are efficient and scalable.