带多处理,池和读取文件的Control-C处理

我刚刚开始使用Multiprocessing和Python,我需要一些帮助在程序中捕获Control-C。我正在制作的脚本将读取一个文件,然后在每一行上执行一些任务。在任何人对I / O和多处理的优点/缺点发表评论之前,我都知道:)这些任务使自己非常友好于多线程。

我有以下代码,并且从文档中,我希望它能正常工作,但是它没有捕获我的键盘异常!哎呀...请帮忙

如果可以,则在Win10上运行:

from multiprocessing import cpu_count
from multiprocessing.dummy import Pool as ThreadPool
import argparse
from time import sleep
import signal
import sys


def readfile(file):
  with open(file, 'r') as file:
    data = file.readlines()
    file.close()
  return data

def work(line):
  while(True):
    try:
      print(f"\rgoing to do some work on {line}")
      countdown(5)
    except (KeyboardInterrupt, SystemExit):
      print("Exiting...")
      break

def countdown(time=30):
  sleep(time)

def parseArgs(args):
  if args.verbose:
    verbose = True
    print("[+] Verbosity turned on")
  else:
    verbose = False
  if args.threads:
    threads = args.threads
  else:
    threads = cpu_count()
  print(f'[+] Using {threads} threads')
  return threads, verbose, args.file


if __name__ == '__main__':
  parser = argparse.ArgumentParser()
  parser.add_argument("-f", "--file", required = True, help="Insert the flie you plan on parsing")
  parser.add_argument("-t", "--threads", help="Number of threads, by default will use all available processors")
  parser.add_argument("-v", "--verbose", help="increase output verbosity",
                       action="store_true")
  threads, verbose, filename = parseArgs(parser.parse_args())
  #read the entire file and store it in a variable:
  data = readfile(filename)
  #Init the data pool
  pool = ThreadPool(threads) # Number of threads going to use
  try:
    pool.map(work,data) # This launches the workers at the function to do work
  except KeyboardInterrupt:
    print("Exiting...")
  finally:
    pool.close()
    pool.join()
评论
  • 给爷站住
    给爷站住 回复

    At the time when you use Control-C the program probably is at pool.join() waiting for all threads to be finished. The pool.map function just starts all the processes but does not block. This means that at the time the KeyboardInterrupt occurs it is not catched because the program is not inside the try-block.

    我不太确定这里的最佳做法,但我会尝试:

    try: 
        pool.map(work, data) # This launches the workers at the function to do work 
        pool.close() 
        pool.join()
    except KeyboardInterrupt: 
        print("Exiting...")