您的位置:首页 > 脚本大全 > > 正文

python的pickle用法(Python multiprocess pool模块报错pickling error问题解决方法分析)

更多 时间:2021-11-05 14:46:52 类别:脚本大全 浏览量:1361

python的pickle用法

Python multiprocess pool模块报错pickling error问题解决方法分析

本文实例讲述了Python multiprocess pool模块报错pickling error问题解决方法。分享给大家供大家参考,具体如下:

问题

之前在调用class内的函数用multiprocessing模块的pool函数进行多线程处理的时候报了以下下错误信息:

PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

查了下官方文档发现python默认只能pickle以下的类型:

  • None, True, and False
  • integers, floating point numbers, complex numbers
  • strings, bytes, bytearrays
  • tuples, lists, sets, and dictionaries containing only picklable objects
  • functions defined at the top level of a module (using def, not lambda)
  • built-in functions defined at the top level of a module
  • classes that are defined at the top level of a module
  • instances of such classes whose dict or the result of calling getstate() is picklable (see section -
  • Pickling Class Instances for details).

函数只能pickle在顶层定义的函数,很明显的class内的函数无法被pickle因此会报错。

  • ?
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • import multiprocessing
  • def work():  # top-level 函数
  •   print "work!"
  • class Foo():
  •   def work(self): # 非top-level函数
  •     print "work"
  • pool1 = multiprocessing.Pool(processes=4)
  • foo = Foo()
  • pool1.apply_async(foo.work)
  • pool1.close()
  • pool1.join()
  • # 此时报错
  • pool2 = multiprocessing.Pool(processes=4)
  • pool2.apply_async(work)
  • pool2.close()
  • pool2.join()
  • # 此时工作正常
  • 解决方案

    调用pathos包下的multiprocessing模块代替原生的multiprocessing。pathos中multiprocessing是用dill包改写过的,dill包可以将几乎所有python的类型都serialize,因此都可以被pickle。或者也可以自己用dill写一个(有点重复造轮子之嫌啊)

    参考

    1. https://stackoverflow.com/questions/8804830/python-multiprocessing-picklingerror-cant-pickle-type-function
    2. https://docs.python.org/3/library/pickle.html#what-can-be-pickled-and-unpickled
    3. https://github.com/uqfoundation/pathos

    希望本文所述对大家Python程序设计有所帮助。

    原文链接:https://www.cnblogs.com/arkenstone/p/7901129.html

    您可能感兴趣