To operate the terminal from python (subprocess)

Use the subprocess summarizes were studied how to operate the terminal from python.

### The subprocess

This package for operating the terminal from python. Fact that the terminal can be operated is not things like do a shell script can be one way. It is dangerous and convenient too.

Since have been developed for many years call, check_call, but there seems to be various modules such as check_output, when you normally run using the subprocess.run (), recommended to use the subprocess.Popen () when running asynchronously It seems to be. Since Popen is a module for parallel processing, it has become to be able or to kill or waiting for the end of the child process.

The official document
https://docs.python.org/ja/3.6/library/subprocess.html#using-the-subprocess-module

This time do not do only immediately end processing use the subprocess.run ().

Windows 10

python 3.6

Ubuntu 18.04

python 3.6

### To run the terminal command

Linux_terminal
ls


Windows_ command prompt
dir


The above command such as you can do the following in subprocess.run ().

In the case of Linux
import subprocess
subprocess.run('ls')


In the case of Windows
import subprocess
subprocess.run('dir', shell=True)


If the terminal on the display comes out command, you receive as a return value of the character string display. Since the character column can also things like taking out only a necessary part in the normal distribution.

In the case of Linux
import subprocess
result = subprocess.run('ls')
print(result)


In the case of Windows
import subprocess
result = subprocess.run('dir', shell=True)
print(result)


### To run giving arguments to the terminal command

If you want to give the argument, but it was space-separated in the terminal will pass in the list in the subprocess.

python
result = subprocess.run(['ls', '-l'])
print(result)


If there is more than one argument only list is getting longer.

python
result = subprocess.run(['ls', '-l', '-a'])
print(result)


By the way, it can also be performed in only one. When it or not or there is an argument is easy likely to do better here of how to write.

python
result = subprocess.run(['ls'])
print(result)


### To run the shell script (batch file)

Since the terminal command anything can be executed, it will pass or arguments can also perform naturally shell script. The following have been carried out in two arguments.

In the case of Linux
result = subprocess.run(['hoge.sh', 'value1', 'value2'])
print(result)


In the case of Win
result = subprocess.run(['hoge.bat', 'value1', 'value2'])
print(result)


### To run python code

The fact that terminal command can be executed, you can also run python if python installation. The following is running, passing two arguments to hoge.py.

python
subprocess.run(['python', 'hoge.py', 'value1', 'value2'])


### To run python code by specifying the conda environment

Since the python interpreter executable file, specify the PATH the environment that are running now if you run python You can also use the python of another conda environment. For example, E in a Windows environment:/Anaconda3 tf of conda environment that was installed in the/is E: since the/Anaconda3/envs/tf/python.exe, you can perform the following writing.

In the case of Windows
subprocess.run(['E:/Anaconda3/envs/tf/python.exe', 'hoge.py'])


In the case of Linux (write-once planned)

In addition, to examine whether the python interpreter where you are is easy if where/which sure you conda activate in the terminal.

In the case of Windows
conda activate tf
where tf


In the case of Linux
conda activate tf
which tf


### To check the gpu memory remaining

GPU usage to parse the string received if nvidia-smi can use environment were also some people that are doing the other I thought Dekirukana acquisition, and.
https://qiita.com/tomotaka_ito/items/1da001c98b46ecf28ec7

python
import subprocess
import json

DEFAULT_ATTRIBUTES = (
'index',
'uuid',
'name',
'timestamp',
'memory.total',
'memory.free',
'memory.used',
'utilization.gpu',
'utilization.memory'
)

def get_gpu_info(nvidia_smi_path='nvidia-smi', keys=DEFAULT_ATTRIBUTES, no_units=True):
nu_opt = '' if not no_units else ',nounits'
cmd = '%s --query-gpu=%s --format=csv,noheader%s' % (nvidia_smi_path, ','.join(keys), nu_opt)
output = subprocess.check_output(cmd, shell=True)
lines = output.decode().split('\n')
lines = [ line.strip() for line in lines if line.strip() != '' ]
return [ { k: v for k, v in zip(keys, line.split(', ')) } for line in lines ]

if __name__ == "__main__":
import pprint
pprint.pprint(get_gpu_info())


We have look like a shell arts.

### Open a new terminal to run the command

It's Linux in the gnome-terminal command, you can open a new terminal or command prompt and use the start command that it is Windows.

In the case of Linux
gnome-terminal ls -l


In the case of Windows
start dir/w


The above command can be run from a python so can be written as follows.

In the case of Linux
subprocess.run(['gnome-terminal', 'ls', '-l'])


In the case of Windows
subprocess.run(['start', 'dir', '/w'], shell=True)


People here if a long period of time-consuming process might be easy to monitor.

When you shell = True to run using the shell. Shell of the embedded command (Windows of dir Toka Toka start) will use in the shell = True because not be called only from the shell. But it means that she can do anything is the fact that to run in the shell will become a cause of vulnerability. It is not necessary to run a shell in the case, such as to run the file Leave the shell = False. The default is also OK if the shell = False So omitted.

https://docs.python.org/ja/3.6/library/subprocess.html#security-considerations

Also, if you shell = True will change the way given the argument. In the case of Linux, but gives the shell = False In the argument in the list, is set to give the argument that it is shell = True of space separated by a string. So caution is needed as you or I to unintended operation inadvertently got ignores the second and subsequent If you give in the list. For some reason you can not notice because the error is also not me out. In the case of Windows, shell = I will do the same for so space-delimited can be used and to True, we are able to receive normally the list even shell = True. Because to come also different stdio of behavior requires a variety of attention.

There is described in official documents
https://docs.python.org/ja/3.6/library/subprocess.html#popen-constructor

### Conclusion

• subprocess Seriously convenient
• Beware the vulnerability
• Let's try

### Modify History

20190903: shell = True has been fixed had been or not attached to the place where or attached to place unnecessary. shell = True we added the description of the.