问题描述

我正在使用 Python 中的 threading 和 Queue 模块编写一个简单的爬虫.我获取一个页面，检查链接并将它们放入队列中，当某个线程完成处理页面时，它会从队列中获取下一个.我正在为我已经访问过的页面使用一个数组来过滤我添加到队列中的链接，但是如果有多个线程并且它们在不同的页面上获得相同的链接，它们会将重复的链接放入队列中.那么如何确定某个 url 是否已经在队列中以避免再次将其放在那里呢?

I'm writing a simple crawler in Python using the threading and Queue modules. I fetch a page, check links and put them into a queue, when a certain thread has finished processing page, it grabs the next one from the queue. I'm using an array for the pages I've already visited to filter the links I add to the queue, but if there are more than one threads and they get the same links on different pages, they put duplicate links to the queue. So how can I find out whether some url is already in the queue to avoid putting it there again?

推荐答案

如果您不关心处理项目的顺序，我会尝试使用 Queue 的子类代码>设置内部:

If you don't care about the order in which items are processed, I'd try a subclass of Queue that uses set internally:

class SetQueue(Queue):

    def _init(self, maxsize):
        self.maxsize = maxsize
        self.queue = set()

    def _put(self, item):
        self.queue.add(item)

    def _get(self):
        return self.queue.pop()

正如 Paul McGuire 所指出的，这将允许在从待处理"集中删除且尚未添加到已处理"集中的重复项目之后添加.为了解决这个问题，您可以将这两个集合存储在 Queue 实例中，但是由于您使用较大的集合来检查项目是否已被处理，您也可以返回 queue 将正确排序请求.

As Paul McGuire pointed out, this would allow adding a duplicate item after it's been removed from the "to-be-processed" set and not yet added to the "processed" set. To solve this, you can store both sets in the Queue instance, but since you are using the larger set for checking if the item has been processed, you can just as well go back to queue which will order requests properly.

class SetQueue(Queue):

    def _init(self, maxsize):
        Queue._init(self, maxsize) 
        self.all_items = set()

    def _put(self, item):
        if item not in self.all_items:
            Queue._put(self, item) 
            self.all_items.add(item)

与单独使用一个集合相比，这样做的优点是 Queue 的方法是线程安全的，因此您不需要额外的锁定来检查另一个集合.

The advantage of this, as opposed to using a set separately, is that the Queue's methods are thread-safe, so that you don't need additional locking for checking the other set.

这篇关于如何检查任务是否已经在 python 队列中?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持跟版网！