I’m trying to use the fromfile-prefix-chars feature of argparse in Python to load all my command line arguments from a file, but it keeps complaining that I haven’t specified some argument.
The code:
import argparse def go(): parser = argparse.ArgumentParser(fromfile_prefix_chars='@') parser.add_argument("--option1") parser.add_argument("--option2", type=int, required=True) args = parser.parse_args() if __name__ == "__main__": go()
The argument file:
--option1 foo --option2 1234
The command line and output:
$ python testargparse.py @testargs usage: testargparse.py [-h] [--option1 OPTION1] --option2 OPTION2 testargparse.py: error: argument --option2 is required
You can see that I’m providing the required argument in the file, but argparse isn’t seeing it.
From the documentation:
Arguments read from a file must by default be one per line … and are treated as if they were in the same place as the original file referencing argument on the command line. So in the example above, the expression [‘-f’, ‘foo’, ‘@args.txt’] is considered equivalent to the expression [‘-f’, ‘foo’, ‘-f’, ‘bar’].
In the example:
fp.write('-f\nbar')
So the file contains:
-f bar
In other words, each of the file lines corresponds to one ‘word’ (blank separated) in the command line. --option1=foo
is one word. --option1 foo
is interpreted just as though it was quoted in the command line,eg. prog.py '--option1 foo' '--option2 1234'
The https://docs.python.org/dev/library/argparse.html#argparse.ArgumentParser.convert_arg_line_to_args has a custom function that will split lines on spaces. Experiment with that if you want to stick with the argument file.
import argparse with open('args.txt', 'w') as fp: fp.write('--option1 foo\n--option2 1234') # error # but works with modifed 'convert...' #fp.write('--option1=foo\n--option2=1234') # works #fp.write('--option1\nfoo\n--option2\n1234') # works def convert_arg_line_to_args(arg_line): for arg in arg_line.split(): if not arg.strip(): continue yield arg """ default line converter: def convert_arg_line_to_args(self, arg_line): return [arg_line] """ def go(): parser = argparse.ArgumentParser(fromfile_prefix_chars='@') parser.convert_arg_line_to_args = convert_arg_line_to_args parser.add_argument("--option1") parser.add_argument("--option2", type=int, required=True) args = parser.parse_args(['@args.txt']) print args if __name__ == "__main__": go()
@toes
suggested using shlex
to parse the file. shlex
has a nice feature in that it strips of unnecessary quotes.
shlex
can be used to split individual lines of the file.
def sh_split(arg_line): for arg in shlex.split(arg_line): yield arg parser.convert_arg_line_to_args = sh_split
Or it can replace the whole @file read method (_read_args_from_files
)- this should function the same as @toes
answer, except that the @file
string can be anywhere in the commandline (or even be repeated).
def at_read_fn(arg_strings): # expand arguments referencing files new_arg_strings = [] for arg_string in arg_strings: if not arg_string or not arg_string.startswith('@'): new_arg_strings.append(arg_string) else: with open(arg_string[1:]) as args_file: arg_strings = shlex.split(args_file.read()) new_arg_strings.extend(arg_strings) return new_arg_strings parser._read_args_from_files = at_read_fn
Obviously a cleaner production version would modify these methods in an ArgumentParser
subclass.
The problem is that, when specified in a file, each argument must have an ‘=’ between it and the option name. While argparse is somewhat more flexible on that format when run from the command line (where space or = is ok), when run from the file it must have an ‘=’.
So, a working argument file would be:
--option1=foo --option2=1234
Something else to be aware of, be sure you don’t have any extra whitespace at the end of the lines or that whitespace will get included with the option when argparse reads the file.
I think there’s a better answer to this: use shlex.
if sys.argv[1].startswith('@'): args = parser.parse_args( shlex.split(open(sys.argv[1][1:]).read()) ) else: args = parser.parse_args()
This allows you to specify args in a file in a more natural way e.g., it allows using spaces or equals sign to specify your args on a single line as in:
arg1 arg2 --opt1 'foo' --opt2='bar'
shlex.split splits this as you would expect:
['arg1', 'arg2', '--opt1', 'foo', '--opt2=bar']
The only thing this method doesn’t have is that it expects the @file.txt to be the first argument.
Try to this way
# encoding: utf-8 import imp import argparse class LoadConfigAction(argparse._StoreAction): NIL = object() def __init__(self, option_strings, dest, **kwargs): super(self.__class__, self).__init__(option_strings, dest) self.help = "Load configuration from file" def __call__(self, parser, namespace, values, option_string=None): super(LoadConfigAction, self).__call__(parser, namespace, values, option_string) config = imp.load_source('config', values) for key in (set(map(lambda x: x.dest, parser._actions)) & set(dir(config))): setattr(namespace, key, getattr(config, key))
Usage
parser.add_argument("-C", "--config", action=LoadConfigAction, default=None) parser.add_argument("-H", "--host")
Example config (real is python file)
# Config example /etc/my.conf import os # Parameter definition host = os.getenv("HOST", "127.0.0.1")