Plumbum: Shell Combinators

Addresses only the local part of https://github.com/tomerfiliba/plumbum/issues/481 with minimal changes and no new dependencies. I haven't worked with the remote parts of plumbum, sorry.

We need to make a release fairly soon, while Travis is still able to run Python 2.6. I'd like to make a 1.6.8 release, and then drop Python 2.6 and non-setuptools builds from master. This would start on the 1.7 work; any critical fixes or easy patches could be backported to a 1.6 branch if needed, but 1.7 would not support Python 2.6. This should allow some cleanup as well. If you are using Python 2.6, which people on RHEL6 based systems might be, plumbum would need to be restricted to <1.7.

Todo:

• [ ] Check translations
• [ ] Fill out changelog
• [ ] Evaluate old PRs

Hi,

I'm using plumbum 1.4.1 to call reflector on Arch Linux. I'm not sure what the problem is, but somewhere along the way, way too much string/space escaping happens.

Here is code to reproduce the problem:

#!/usr/bin/env python3
from itertools import chain
from plumbum.cmd import reflector, sudo

def update_mirrorlist(countries=['United States', 'Canada']):
    countries = list(chain(*[('--country', country) for country in countries]))
    print(sudo[reflector['--save', '/etc/pacman.d/mirrorlist', '--sort', 'score', '--latest', '30', countries]])
    sudo(reflector['--save', '/etc/pacman.d/mirrorlist', '--sort', 'score', '--latest', '30', countries])

if __name__ == '__main__':
    update_mirrorlist()

This yields the output

/usr/bin/sudo /usr/bin/reflector --save /etc/pacman.d/mirrorlist --sort score --latest 30 --country 'United States' --country Canada

which looks perfect. If I run that printed command exactly, I get the desired result. However, that is not what gets run by the final line in update_mirrorlist. The generated mirrorlist does not include anything from the United States, and the comments report that the command actually run was

reflector --save /etc/pacman.d/mirrorlist --sort score --latest 30 --country ''"'"'United States'"'"'' --country Canada

I'm currently experiencing issues setting up remotes towards RHEL5 & 6 remotes. I'm getting a stack trace with EOFError (see below). plumbum version 1.4.0. Python 3.2.3 & 3.3.4

Connecting to remote Mac OS X 10.9.1 and Ubuntu 12.04.4 LTS works fine.

debug1: Local version string SSH-2.0-OpenSSH_6.5

RHEL 5 OpenSSH

debug1: Remote protocol version 2.0, remote software version OpenSSH_4.3
debug1: match: OpenSSH_4.3 pat OpenSSH_4* compat 0x00000000

RHEL 6 OpenSSH

debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000000

OS X 10.9.1 OpenSSH

debug1: Remote protocol version 2.0, remote software version OpenSSH_6.2
debug1: match: OpenSSH_6.2 pat OpenSSH* compat 0x04000000

Ubuntu 12.04 OpenSSH

debug1: Remote protocol version 2.0, remote software version OpenSSH_5.9p1 Debian-5ubuntu1.1
debug1: match: OpenSSH_5.9p1 Debian-5ubuntu1.1 pat OpenSSH_5* compat 0x0c000000

Traceback

Traceback (most recent call last):
  File "./plum-e.py", line 15, in <module>
    with SshMachine(ssh_host, user=ssh_user, ssh_command=ssh, scp_command=scp, keyfile=ssh_key_path) as remote:
  File "/usr/lib/python3.2/site-packages/plumbum/machines/ssh_machine.py", line 101, in __init__
    BaseRemoteMachine.__init__(self, encoding = encoding, connect_timeout = connect_timeout)
  File "/usr/lib/python3.2/site-packages/plumbum/machines/remote.py", line 124, in __init__
    self._session = self.session()
  File "/usr/lib/python3.2/site-packages/plumbum/machines/ssh_machine.py", line 145, in session
    self.encoding, isatty, self.connect_timeout)
  File "/usr/lib/python3.2/site-packages/plumbum/machines/session.py", line 141, in __init__
    self.run("")
  File "/usr/lib/python3.2/site-packages/plumbum/machines/session.py", line 224, in run
    return run_proc(self.popen(cmd), retcode)
  File "/usr/lib/python3.2/site-packages/plumbum/commands/processes.py", line 186, in run_proc
    stdout, stderr = proc.communicate()
  File "/usr/lib/python3.2/site-packages/plumbum/machines/session.py", line 92, in communicate
    line = pipe.readline()
  File "/usr/lib/python3.2/site-packages/plumbum/machines/session.py", line 42, in readline
    raise EOFError()
EOFError

I love the ability to create pipelines and to do file redirections in Plumbum with the bash-like syntax, but I'm not a big fan of the & FG/& BG constructs for actually running the pipelines, because I feel that they're an unintuitive syntax. I feel that there should be an explicit method call to run these commands, and indeed my IDE treats these statements as doing nothing:

Is there a nice way to say "run this pipeline, with these redirections", without using these & constructs? I've looked at run() and popen(), but run() seems to return the entire stdout, which worries me because I'm doing large data processing and I don't want python to ever have to see this data. popen() is also more complex than I need, since you have to call communicate, and when you do so it still returns stdout, which I don't want.

This fixes #365. I can't actually find when this syntax became illegal, but this replacement works both in Python 2.7 and 3.6 whereas the original doesn't.

As I wrote in #365, this patch unbreaks installs under versions of pip that have a hard dependency on the py->pyc step succeeding.

This PR fixes #335 - It adds gettext localization to all strings that are used in Application output. Also as a test I've written russian localization. Any other language can be easily added. This should be tested on py2.x and on Windows.

Hi, when using some sudo commands I found out that they were failing where they shouldn't have. E.g.:

from plumbum.cmd import sudo, yum

sudo(yum['install']['rubygem(rails)'])

This fails, because rubygem(rails) gets overquoted by formulate (which calls shquote). This ends up formulated like this:

['/usr/bin/sudo', '/usr/bin/yum', 'install', "'rubygem(rails)'"]

So yum in fact gets "'rubygem(rails)'" on commandline (because Popen puts the outer quotes there.

I think that the problem can be solved by adding '(' and ')' to _safechars, which seems ok to me, but I'm not sure of other consequences of this. Any other options how to solve this?

Thanks.

Fix handling of env-vars passed to plumbum BoundEnvCommands: - don't use the non-threadsafe and session-dependent .env() context manager - sync popen support for 'env' param in all machine impls: local/ssh/paramiko - add .with_cwd() to complement .with_env()

Previous behavior looks like this

Error: Argument of output expected to be <plumbum.cli.switches.Predicate object at 0x10ce91c10>, not 'requirements.txt':
        ValueError('output is a file, should be a directory',)

now looks like this

    Error: Argument of output expected to be 'MakeDirectory', not 'requirements.txt':
        ValueError('output is a file, should be a directory',)

when checking a Predicate.

Hi Tomer, what would be a good way to implement this command

diff <(sed 's/pattern//g' file1) <(sed 's/pattern//g' file2)

where pattern, file1 and file2 would change from one execution to another?

recursive globbing not work like pathlib. Is this intented?

if cwd is /home/yakkle/tmp has subdirectory foo/bar, and has file sample.zip in foo/bar :

>>> local.cwd // "**/*.zip"
[]
>>> 

will show empty list.

if base path is /home/yakkle/tmp/foo:

>>> local.cwd["foo"] // "**/*.zip"
[<LocalPath /home/yakkle/tmp/foo/bar/sample.zip>]
>>> 

will show found result. but it only 1 depth search. not recursive.

python docs says :

The “**” pattern means “this directory and all subdirectories, recursively”. In other words, it enables recursive globbing:

https://docs.python.org/3.9/library/pathlib.html?highlight=mkdir#pathlib.Path.glob

I can't get ssh with key authentication to work. The keyfile attribute seems to be ignored and the program keeps asking for user password. There's even no difference when I specify invalid path to the key file. There's no quick way to examine the ssh command being used as plumbum doesn't have debug facilities (https://github.com/tomerfiliba/plumbum/issues/474).

The key file is accessible and can be read in the same Python script.

The code is basically with SshMachine('[email protected]', keyfile='./ssh/keyfile') as remote_box:

Environment: Debian 11 Python 3.9.2 OpenSSH 8.4p1 Plumbum 1.7.2

for example:

print(plumbum.local['echo'](b'test'))

should result in

test

but instead it results in:

b'test'

byte objects are not even supported in command names:

plumbum.local[b'echo']

results in a type error

TypeError: a bytes-like object is required, not 'str'

in plumbum/machines/local.py:243

Hi, thanks for the great library. I have a question.

As far as I understand, when you run commands via local object, they read environ from frozen local.env object. Is there a way to change that behavior and to make them use os.environ instead?

The issue I have - I have other python code I don't have quite a control of that changes os.environ, so it would be more convenient if local would use os.environ instead of saved local.env.

Thank you.

Hi, thanks for the great library. I have a question - is it possible to set PR_SET_PDEATHSIG on Linux for all running commands?

When I use subprocess myself, I do it like:

SIGHUP = 1
PR_SET_PDEATHSIG = 1
libc = ctypes.CDLL("/lib64/libc.so.6")


def set_pdeathsig_cb():
    ppid = os.getpid()

    def cb():
        res = libc.prctl(PR_SET_PDEATHSIG, SIGHUP)
        if res != 0:
            print(f"Fail to set PR_SET_PDEATHSIG. Return value: {res}", file=sys.stderr)
            sys.exit(1)
        if os.getppid() != ppid:
            print("Parent process has already died", file=sys.stderr)
            sys.exit(1)

    return cb

with subprocess.Popen(cmd, preexec_fn=set_pdeathsig_cb()) as p:
    ...

Thank you.