Abstract:With the fast development of network technology and the electric power information service, the expansion of network information will be lead to the phenomenon of massive web redundancy in the Internet and the power information network.it''s increates the complexity of data mining and rapid retrieval, which brings a great challenge to the performance of network equipment and storage equipment. Therefore, it is necessary to study massive Web deduplication.Web deduplication is a process which detected duplicate content pages from a given amount of data collection, and then removed from the copy of the collection. Which research of web deduplication based on URL filter has achieved great development,But it is no good solution to the problem of massive web pages filter.Based on web-based MD5 fingerprint deduplication algorithm, and using Counting Bloom filter algorithm, this essay proposed a algorithm for rapidly deduplication called IMP-CMFilter,which could improve the efficiency of mass web pages filter by reducing the frequent operation of I/O . On the fact that the IMP-CMFilter algorithm had higher performance.