Multi-process logging error: PermissionError: [WinError 32] Another program is using this file, the process cannot access

Reproduce the problem

2019-12-23 13:28:24,415 - INFO: Ongoing D:\Anaconda_program\hangtian_spider\hangkong/data/EI/01.txt file<RECORD 216>Record reading
--- Logging error ---
Traceback (most recent call last):
  File "D:\Anaconda3\lib\logging\handlers.py", line 72, in emit
    self.doRollover()
  File "D:\Anaconda3\lib\logging\handlers.py", line 173, in doRollover
    self.rotate(self.baseFilename, dfn)
  File "D:\Anaconda3\lib\logging\handlers.py", line 113, in rotate
    os.rename(source, dest)
PermissionError: [WinError 32]  This file is being used by another program and cannot be accessed by the process.: 'D:\\Anaconda_program\\hangtian_spider\\hangkong\\log\\ei_txt_file.log' -> 'D:\\Anaconda_program\\hangtian_spider\\hangkong\\log\\ei_txt_file.log.1'
Call stack:
  File "D:\Anaconda3\lib\threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "D:\Anaconda3\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "D:\Anaconda3\lib\threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "D:\Anaconda_program\hangtian_spider\hangkong\web\app\data\EI_Data.py", line 228, in get_ei_txt_file
    read_txt(root_dir+file)
  File "D:\Anaconda_program\hangtian_spider\hangkong\web\app\data\EI_Data.py", line 260, in read_txt
    ei_data.add_ei_data(re_dict)
  File "D:\Anaconda_program\hangtian_spider\hangkong\web\app\data\EI_Data.py", line 126, in add_ei_data
    logger.info("The key fields screened in this record are: record_dict.keys(): {}".format(', '.join(record_dict.keys())))
Message: 'The key fields screened in this record are: record_dict.keys(): ei_an, ei_ti, ei_au, ei_af, ei_ca, ei_st, ei_ast, ei_vl, ei_is, ei_isd, ei_py, ei_pg, ei_la, ei_sn, ei_sn, ei_ , ei_ab, ei_nr, ei_mh, ei_ctt, ei_uctt, ei_cfc, ei_di, ei_db, ei_cit'
Arguments: ()
2019-12-23 13:28:24,467 - INFO:  The key fields screened in this record are: record_dict.keys(): ei_an, ei_ti, ei_au, ei_af, ei_ca, ei_st, ei_ast, ei_vl, ei_is, ei_isd, ei_py, ei_pg, ei_la, ei_sn, ei_cn, ei_dt, ei_pu, ei_ab, ei_nr, ei_mh, ei_ctt, ei_uctt, ei_cfc, ei_di, ei_db, ei_cit
--- Logging error ---

Solution

from logging import getLogger, INFO
from concurrent_log_handler import ConcurrentRotatingFileHandler
import os

log = getLogger()
# Use an absolute path to prevent file rotation trouble.
logfile = os.path.abspath("mylogfile.log")
# Rotate log after reaching 512K, keep 5 old copies.
rotateHandler = ConcurrentRotatingFileHandler(logfile, "a", 512*1024, 5)
log.addHandler(rotateHandler)
log.setLevel(INFO)

log.info("Here is a very exciting log message, just for you")

reference

https://github.com/Preston-Landers/concurrent-log-handler
https://blog.csdn.net/chongtong/article/details/80831782