尽管分配了内存,多处理池仍无法正常工作

我处理了预处理,该预处理必须使用多重处理进行迭代。

下面的代码正常工作。

wt_adherence = partial(get_wt_adherence, wt=wt_logs, user_profile=user_profile)

n_cpus = multiprocessing.cpu_count()
pool = multiprocessing.Pool(10)
result_wt = pool.map(wt_adherence, accessCodes)         ; print('wt adherence finished')
pool.close()

当以下代码出现问题。

pool = multiprocessing.Pool(10)
result_meal = pool.map(meals_adherence, accessCodes)     ; print('meal adherence finished')
pool.close()

当第二个脚本运行时,并行进程在Linux内核中显示如下(带有顶部)。

尽管已为每个进程分配了内存(10.8%),但CPU无法正常工作。

21934 hoheon    20   0 38.431g 0.035t  29488 S 100.7 14.4  50:22.80 python3                                                                                        
22396 hoheon    20   0 29.399g 0.027t   3996 S   0.0 10.8   0:00.00 python3                                                                                        
22397 hoheon    20   0 29.399g 0.027t   3996 S   0.0 10.8   0:00.00 python3                                                                                        
22398 hoheon    20   0 29.399g 0.027t   3996 S   0.0 10.8   0:00.00 python3                                                                                        
22399 hoheon    20   0 29.399g 0.027t   3996 S   0.0 10.8   0:00.00 python3                                                                                        
22400 hoheon    20   0 29.399g 0.027t   3996 S   0.0 10.8   0:00.00 python3                                                                                        
22401 hoheon    20   0 29.399g 0.027t   3996 S   0.0 10.8   0:00.00 python3                                                                                        
22402 hoheon    20   0 29.399g 0.027t   3996 S   0.0 10.8   0:00.00 python3                                                                                        
22403 hoheon    20   0 29.399g 0.027t   3996 S   0.0 10.8   0:00.00 python3                                                                                        
22395 hoheon    20   0 29.399g 0.027t   3984 S   0.0 10.8   0:00.00 python3                                                                                        
22404 hoheon    20   0 29.399g 0.027t   3984 S   0.0 10.8   0:00.00 python3   

我怀疑使用大小为1GB的pandas数据框的名为“ meals_adherence”的函数。

<class 'pandas.core.frame.DataFrame'>
MultiIndex: 32691788 entries, ('111TN3AT', Timestamp('2018-06-27 00:00:00')) to ('ZZZG24EX', Timestamp('2018-09-24 00:00:00'))
Data columns (total 5 columns):
 #   Column       Dtype  
---  ------       -----  
 0   timeSlot     object 
 1   didSkipMeal  int8   
 2   foodType     object 
 3   foodName     object 
 4   calories     float16
dtypes: float16(1), int8(1), object(3)
memory usage: 966.7+ MB

但是,由于我的PC的内存容量为250GB和50CPU,因此内存和CPU进程处于空闲状态 甚至剩下50%的内存空间。

当我等待3个小时并再次检查脚本时,也会出现以下错误。

Traceback (most recent call last):
  File "3_feature_ext_multiprocessing.py", line 137, in <module>
    result_meal = pool.map(meals_adherence, accessCodes)     ; print('meal adherence finished')
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 424, in _handle_tasks
    put(task)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 393, in _send_bytes
    header = struct.pack("!i", n)
struct.error: 'i' format requires -2147483648 <= number <= 2147483647

为什么会出现此问题?

评论