场景模型>> pycommand>> 返回
项目作者： bodo-hugo-barwich

项目描述 ：
Python Package for Multiprocessing
高级语言： Python
项目地址: git://github.com/bodo-hugo-barwich/pycommand.git
创建时间： 2020-08-25T16:56:47Z
项目社区：https://github.com/bodo-hugo-barwich/pycommand
开源协议：
下载


Automated Tests Build Status

Command

Command - Python Package for Multiprocessing

Provides Classes to launch Child Processes asynchronously.\
The Object Oriented Design allows to create Groups of Child Processes and Child Process Pools to launch several child processes in an organized manner.

Features

Some important Features are:

  • Low Dependencies (uses only Python Core Packages)\
    Low Dependency Usage leads to:
    1. * Very High Compatibility (only Python 3 is required)
    2. * Easy Installation
    • Small Memory Footprint (Simple Structure Design leads to low Memory Usage)
    • Fast Startup (very few additional Libraries to load)
  • Asynchronous Launch
  • Reads Big Outputs
  • Execution Timeout
  • Configurable Read Interval
  • Captures possible System Errors at Launch Time like “file not found” Errors
  • Streamlined Error Handling while still providing the Outputs

Motivation

This Module was conceived out of the need to launch multiple tasks simultaneously while still keeping each Log and Error Messages and Exit Codes separately. \
As it is documented in Python Documentation - Thread-based parallelism
and Python - Global Interpreter Lock
processor intensive tasks cannot run in Python threads and are advised to be executed in multiple processes. \
The Python implementation was derived from a prototype I developed at:
Multi Process Manager\
The Object Oriented Design permits the implementation of the Command Pattern / Manager-Worker Pattern.\
Providing a similar functionality as the subprocess.run() Function it can serve as a Procedural Replacement for this function without the need of special error handling of possible Exceptions. \
This implementation aimes especially for Low Dependencies and Easy Installation.

Example Use Case

The Power of this Library is best shown by an Example Use Case as seen in the test_CommandGroupRun() Test:\
Having 3 Jobs at hand of 2 seconds, 3 seconds and 1 second running them sequencially would take aproximately 6 seconds.\
But using the CommandGroup Class it takes effectively only 3 seconds to complete.\
And still each Job can be evaluated separately by their own Results keeping Log Message separate from Error Messages and viewing them in their context.

  1. setUp - go ...
  2. setUp - Test Directory: '/home/runner/work/pycommand/pycommand/tests/'
  3. setUp - Test Module: 'commandgrouptests.py'
  5. test_CommandGroupRun - go ...
  6. Command Group Execution Start - Time Now: '1619689908.3838296' s
  7. Command Group Execution End - Time Now: '1619689911.4230719' s
  8. Command Group Execution finished in '3039.2422676086426' ms
  9. Command Group Execution Time '3 / 3' s
  10. Command Group ERROR CODE: '0'
  11. Command Group STDOUT:
  12. '2021-04-29 09:51:48 : Sub Process No. '0' - 'command-script:2s': Launching ...
  13. 2021-04-29 09:51:48 : Sub Process No. '0' - 'command-script:2s': Launch OK - PID (1656)
  14. 2021-04-29 09:51:48 : Sub Process No. '1' - 'command-script:3s': Launching ...
  15. 2021-04-29 09:51:48 : Sub Process No. '1' - 'command-script:3s': Launch OK - PID (1657)
  16. 2021-04-29 09:51:48 : Sub Process No. '2' - 'command-script:1s': Launching ...
  17. 2021-04-29 09:51:48 : Sub Process No. '2' - 'command-script:1s': Launch OK - PID (1658)
  18. 2021-04-29 09:51:49 : Sub Process (1658) 'command-script:1s': finished with [0]
  19. 2021-04-29 09:51:50 : Sub Process (1656) 'command-script:2s': finished with [0]
  20. 2021-04-29 09:51:51 : Sub Process (1657) 'command-script:3s': finished with [0]
  21. '
  22. Command Group STDERR:
  23. ''
  24. Command (1656) 'command-script:2s':
  25. ERROR CODE: '0'
  26. EXIT CODE: '0'
  27. STDOUT:
  28. 'Start - Time Now: '1619689908.4170215'
  29. Number of arguments: 2 arguments.
  30. Argument List: ['/home/runner/work/pycommand/pycommand/tests/command_script.py', '2']
  31. test script absolute path: '/home/runner/work/pycommand/pycommand/tests/command_script.py'
  32. script 'command_script.py' START 0
  33. script 'command_script.py' PAUSE '2' ...
  34. script 'command_script.py' END 1
  35. End - Time Now: '1619689910.4191885'
  36. script 'command_script.py' done in '2002.166986465454' ms
  37. script 'command_script.py' EXIT '0'
  38. '
  39. STDERR:
  40. 'script 'command_script.py' START 0 ERROR
  41. script 'command_script.py' END 1 ERROR
  42. '
  43. Command (1657) 'command-script:3s':
  44. ERROR CODE: '0'
  45. EXIT CODE: '0'
  46. STDOUT:
  47. 'Start - Time Now: '1619689908.418119'
  48. Number of arguments: 2 arguments.
  49. Argument List: ['/home/runner/work/pycommand/pycommand/tests/command_script.py', '3']
  50. test script absolute path: '/home/runner/work/pycommand/pycommand/tests/command_script.py'
  51. script 'command_script.py' START 0
  52. script 'command_script.py' PAUSE '3' ...
  53. script 'command_script.py' END 1
  54. End - Time Now: '1619689911.4193742'
  55. script 'command_script.py' done in '3001.2552738189697' ms
  56. script 'command_script.py' EXIT '0'
  57. '
  58. STDERR:
  59. 'script 'command_script.py' START 0 ERROR
  60. script 'command_script.py' END 1 ERROR
  61. '
  62. Command (1658) 'command-script:1s':
  63. ERROR CODE: '0'
  64. EXIT CODE: '0'
  65. STDOUT:
  66. 'Start - Time Now: '1619689908.4242425'
  67. Number of arguments: 2 arguments.
  68. Argument List: ['/home/runner/work/pycommand/pycommand/tests/command_script.py', '1']
  69. test script absolute path: '/home/runner/work/pycommand/pycommand/tests/command_script.py'
  70. script 'command_script.py' START 0
  71. script 'command_script.py' PAUSE '1' ...
  72. script 'command_script.py' END 1
  73. End - Time Now: '1619689909.425416'
  74. script 'command_script.py' done in '1001.1734962463379' ms
  75. script 'command_script.py' EXIT '0'
  76. '
  77. STDERR:
  78. 'script 'command_script.py' START 0 ERROR
  79. script 'command_script.py' END 1 ERROR
  80. '

Usage

runCommand() Function

The runCommand() Function is easy to use and straight forward.\
It is best seen in the pytest test_RunCommand() Test:\

  1. from libcommand import runCommand
  4. sdirectory = os.getcwd() + '/'
  5. smodule = ''
  6. stestscript = 'command_script.py'
  7. itestpause = 3
  8. iteststatus = 4
  10. spath = os.path.abspath(__file__);
  12. print("test script absolute path: '{}'".format(spath))
  14. slashpos = spath.rfind('/', 0)
  16. if slashpos != -1 :
  17.   sdirectory = spath[0 : slashpos + 1]
  18.   smodule = spath[slashpos + 1 : len(spath)]
  19. else :
  20.   smodule = spath
  22. print("Test Directory: '{}'".format(sdirectory))
  23. print("Test Module: '{}'".format(smodule))
  26. def test_RunCommand():
  27.   print("{} - go ...".format(sys._getframe().f_code.co_name))
  29.   stestscript = 'command_script.py'
  30.   itestpause = 3
  32.   arrrs = runCommand("{}{} {} {}".format(sdirectory, stestscript, itestpause, iteststatus))
  34.   print("EXIT CODE: '{}'".format(arrrs[2]));
  36.   assert arrrs[0] != '', "STDOUT was not captured."
  38.   print("STDOUT: '{}'".format(arrrs[0]));
  40.   assert arrrs[1] != '', "STDERR was not captured."
  42.   print("STDERR: '{}'".format(arrrs[1]));
  44.   print("")

The Output shows how STDOUT, STDERR and EXIT Code are cleanly separated.\
This will produce the Output:

  1. setUp - go ...
  2. setUp - Test Directory: '/path/to/pycommand/tests'
  3. setUp - Test Module: './commandtests.py'
  5. test_RunCommand - go ...
  6. EXIT CODE: '4'
  7. STDOUT: 'Start - Time Now: '1617575015.5021873'
  8. Number of arguments: 3 arguments.
  9. Argument List: ['/path/to/pycommand/tests/test_script.py', '3', '4']
  10. test script absolute path: '/path/to/pycommand/tests/test_script.py'
  11. script 'test_script.py' START 0
  12. script 'test_script.py' PAUSE '3' ...
  13. script 'test_script.py' END 1
  14. End - Time Now: '1617575018.503503'
  15. script 'test_script.py' done in '3001.3158321380615' ms
  16. script 'test_script.py' EXIT '4'
  17. '
  18. STDERR: 'script 'test_script.py' START 0 ERROR
  19. script 'test_script.py' END 1 ERROR
  20. '