f90nml - A Fortran namelist parser, generator, and editor

Overview

f90nml - A Fortran namelist parser, generator, and editor

A Python module and command line tool for parsing Fortran namelist files

https://travis-ci.org/marshallward/f90nml.svg?branch=master https://ci.appveyor.com/api/projects/status/bcugyoqxiyyvemy8?svg=true https://coveralls.io/repos/marshallward/f90nml/badge.svg?branch=master

Documentation

The complete documentation for f90nml is available from Read The Docs.

http://f90nml.readthedocs.org/en/latest/

About f90nml

f90nml is a Python module and command line tool that provides a simple interface for the reading, writing, and modifying Fortran namelist files.

A namelist file is parsed and converted into an Namelist object, which behaves like a standard Python dict. Values are converted from Fortran data types to equivalent primitive Python types.

The command line tool f90nml can be used to modify individual values inside of a shell environment. It can also be used to convert the data between namelists and other configuration formats. JSON and YAML formats are currently supported.

Quick usage guide

To read a namelist file sample.nml which contains the following namelists:

&config_nml
   input = 'wind.nc'
   steps = 864
   layout = 8, 16
   visc = 1.0e-4
   use_biharmonic = .false.
/

we would use the following script:

import f90nml
nml = f90nml.read('sample.nml')

which would would point nml to the following dict:

nml = {
    'config_nml': {
        'input': 'wind.nc',
        'steps': 864,
        'layout': [8, 16],
        'visc': 0.0001,
        'use_biharmonic': False
    }
}

File objects can also be used as inputs:

with open('sample.nml') as nml_file:
    nml = f90nml.read(nml_file)

To modify one of the values, say steps, and save the output, just manipulate the nml contents and write to disk using the write function:

nml['config_nml']['steps'] = 432
nml.write('new_sample.nml')

Namelists can also be saved to file objects:

with open('target.nml') as nml_file:
   nml.write(nml_file)

To modify a namelist but preserve its comments and formatting, create a namelist patch and apply it to a target file using the patch function:

patch_nml = {'config_nml': {'visc': 1e-6}}
f90nml.patch('sample.nml', patch_nml, 'new_sample.nml')

Command line interface

A command line tool is provided to manipulate namelist files within the shell:

$ f90nml config.nml -g config_nml -v steps=432
&config_nml
   input = 'wind.nc'
   steps = 432
   layout = 8, 16
   visc = 1.0e-4
   use_biharmonic = .false.
/

See the documentation for details.

Installation

f90nml is available on PyPI and can be installed via pip:

$ pip install f90nml

The latest version of f90nml can be installed from source:

$ git clone https://github.com/marshallward/f90nml.git
$ cd f90nml
$ pip install .

Package distribution

f90nml is not distributed through any official packaging tools, but it is available on Arch Linux via the AUR:

$ git clone https://aur.archlinux.org/python-f90nml.git
$ cd python-f90nml
$ makepkg -sri

Volunteers are welcome to submit and maintain f90nml on other distributions.

Local install

Users without install privileges can append the --user flag to pip from the top f90nml directory:

$ pip install --user .

If pip is not available, then setup.py can still be used:

$ python setup.py install --user

When using setup.py locally, some users have reported that --prefix= may need to be appended to the command:

$ python setup.py install --user --prefix=

YAML support

The command line tool offers support for conversion between namelists and YAML formatted output. If PyYAML is already installed, then no other steps are required. To require YAML support, install the yaml extras package:

$ pip install f90nml[yaml]

To install as a user:

$ pip install --user .[yaml]

Contributing to f90nml

Users are welcome to submit bug reports, feature requests, and code contributions to this project through GitHub. More information is available in the Contributing guidelines.

Comments
  • Add file arguments

    Add file arguments

    Some API questions are how to handle force= and the default output name for patching; I decided to ignore force for files. For the latter issue, I raise ValueError when nml_fname is a file since there is no sane default.

    (personally, if it was my library, I would rip out the default name functionality entirely, and make it so that when there is no output file or path, it will still build and return the patched Namelist object without writing a patched namelist file. But that's just me)

    Addresses #25 (although there is no string API here, at least yet)

    opened by ExpHP 25
  • Order of repeated blocks not preserved when mixed with other named blocks

    Order of repeated blocks not preserved when mixed with other named blocks

    File file contains

    &block00
    i=1
    k=2
    &end
    &block01
    m=0
    n=1
    &end
    &block00
    j=3
    z=4
    &end
    

    The statement f90nml.write(f90nml.read(file),outfile) will result in outfile containing

    &block00
    i=1
    k=2
    /
    &block00
    j=3
    z=4
    /
    &block01
    m=0
    n=1
    /
    

    where order of namelist blocks has not been preserved.

    opened by jpf-x 24
  • Strings starting with + are parsed as a list

    Strings starting with + are parsed as a list

    I have a file formatted this way

    &iostuff
    output_folder = +out
    /
    

    When parsed, ['iostuff']['output_folder'] equals ['+', 'out']. I guess this is done so that if the variable out is defined somewhere else, this list is replaced by the corresponding numerical value. But if it isn't defined, the string '+out' should be returned instead. Maybe a simple solution would be having a way to tell f90nml that some variables are strings and should be considered as such.

    opened by amorison 24
  • Namelist with repeated groups doesn't convert to string

    Namelist with repeated groups doesn't convert to string

    Trying to convert a namelist with repeated groups to a string seems to fail. If I do the following:

    >>> import f90nml
    >>> nml = f90nml.reads("""
    ... &foo
    ...  a = 1
    ... /
    ... &foo
    ...  b = 2
    ... /
    ... """)
    >>> str(nml)
    "Namelist([('foo', [Namelist([('a', 1)]), Namelist([('b', 2)])])])"
    

    I was expecting str(nml) to return "&foo\n a = 1\n/\n&foo\n b = 2\n/\n", as is the case when the group names do not repeat. Interestingly, if I use nml.write('out.nml') the the output file is formatted correctly.

    opened by chunderbunny 21
  • Write bug in v0.20

    Write bug in v0.20

    I'm getting an odd bug when trying the write the following namelist:

     &a
     b%c_d_E(1)%ID = 1,
     b%c_d_E(2)%ID = 2,
     /
    

    Code is:

        import f90nml
        data = f90nml.read('test.nml')
        with open('test_RESAVED.nml','w') as f:
            data.write(f, force=True)
    

    Error message is:

    Traceback (most recent call last):
      File "test.py", line 309, in <module>
        data.write(f, force=True)
      File "C:\Anaconda3\lib\site-packages\f90nml\namelist.py", line 232, in write
        self.write_nmlgrp(grp_name, grp_vars, nml_file)
      File "C:\Anaconda3\lib\site-packages\f90nml\namelist.py", line 253, in write_nmlgrp
        for v_str in self.var_strings(v_name, v_val, v_start=v_start):
      File "C:\Anaconda3\lib\site-packages\f90nml\namelist.py", line 302, in var_strings
        v_strs = self.var_strings(v_title, f_vals, v_start=v_start_new)
      File "C:\Anaconda3\lib\site-packages\f90nml\namelist.py", line 313, in var_strings
        i_s = v_start[::-1][len(v_idx)]
    TypeError: 'NoneType' object is not subscriptable
    

    Oddly, when I change c_d_E to c_d_e, it works fine.

    I just updated to the latest release using pip. I've been able to read variables like this before, so something has changed.

    opened by jacobwilliams 20
  • Add a dependency on the yaml module

    Add a dependency on the yaml module

    Trying to use f90nml after installing on macOS via pip3 install f90nml with the -f yaml flag results in:

    f90nml: error: YAML module could not be found.
    

    I think you should just add an explicit dependency on pyyaml. Or, better yet, if there's a way to add a recommended (installed by default) dependency on pyyaml that someone could opt out of. (I'm not familiar enough with python packaging to know for certain if this is possible one way or the other.)

    opened by zbeekman 19
  • Change to accept NumPy arrays and similar objects

    Change to accept NumPy arrays and similar objects

    Hi there! We've been using f90nml in Pyrokinetics to read/write input files to plasma physics codes, and we ran into an issue where NumPy 0D arrays would propagate unnoticed into a Namelist and cause an exception to be thrown on calling Namelist.write. I've added a small change to Namelist._f90repr to detect 0D arrays and convert them to their equivalent primitive type. Nothing changes if NumPy is not installed.

    Let me know if you'd like me to add further test cases, or if you'd prefer to avoid including these sorts of edge cases in your library.

    opened by LiamPattinson 18
  • Optimizing performance

    Optimizing performance

    I was wondering if there were possibilities for optimizing performance for very large namelist files. I'm in the process of doing some benchmarks for some large files (say around 1500 lines, with multiple namelists and pretty much all variable types). I can only get around 1.7 calls of f90nml.read() per second. I assume that it's the parsing and/or creating of the structures that is the bottleneck (actually reading in the lines should only take a fraction of a second), but I'm going to investigate further.

    One thing I was wondering if parsing of different namelists in the same file could take place in parallel? I don't know if such a thing is possible or not, but if it is that might be something to explore. Maybe this is something I can try and contribute to, rather than just reporting bugs and asking for features!

    enhancement 
    opened by jacobwilliams 17
  • repetition of null value bug

    repetition of null value bug

    Hi Marshall, Thanks for this great tool!

    The bug occurs for the following variable input line: AXFCLN = 3.0, 3.0, 48*, The 48* denotes a repetition of 48 null values Upon .write(), it is being turned into:
    AXFCLN = 3.0, 3.0, ','*,

    One clue is that everything is fine (output = input) if the trailing comma is removed prior to processing.

    Processing attached files as follows: nml=f90nml.read('repetition_of_null_win.nml') f90nml.patch('repetition_of_null_win.nml', nml, 'patched_data_win.nml') and likewise for the unix line-endings version.

    I'm using latest master commit 151438f on Oct 31, 2019 (ver 1.1.2) My sys.version is: 3.7.5 (tags/v3.7.5:5c02a39a0b, Oct 15 2019, 00:11:34) [MSC v.1916 64 bit (AMD64)] I'm running Win10/64/Pro.

    patched_data_unix.txt patched_data_win.txt repetition_of_null_win.txt repetition_of_null_unix.txt

    opened by frankeye 15
  • whitespace between indices for array in namelist cannot be parsed by nag

    whitespace between indices for array in namelist cannot be parsed by nag

    When writing namelists containing nested structures that need to be indexed, e.g.

    idx_nml
       v(1, 1) = 5
       v(2, 1) = 5
    /
    

    the resulting namelist file cannot be parsed by a program compiled with nag (tested here with version 6.1) , if there is a whitespace between the indices. I was wondering, if that whitespace can be omitted?

    opened by gitporst 15
  • f90nml.read() crashes for namelists without a terminating EOL.

    f90nml.read() crashes for namelists without a terminating EOL.

    I'm trying to transform a list of configuration files into a list of f90nml.Namelist objects. StopIteration is being raised by Tokenizer.update_chars().

    For example, let's say I have two nml files, foo.nml and bar.nml, this bug can be triggered by :

    [f90nml.read(s) for s in ['foo.nml','bar.nml']]
    
    opened by neutrinoceros 14
  • patching a repeated section on a namelist returns an unclear message

    patching a repeated section on a namelist returns an unclear message

    Related to esm-tools/esm_tools#843

    Problem description

    When loading a namelist with repeated sections, and then I try to patch one or several repeated sections, I get the following error:

    AttributeError: 'Cogroup' object has no attribute 'update'
    

    I understand it is not possible to update repeated sections, as it is not clear which one f90nml should patch. However, I find the error message misleading.

    Example

    To reproduce the error you can write the following fort.4 namelist:

    &NAMDIM
        NPROMA = -8, 
    /
    
    &NAMPAR0
        LSTATS = true,
        LDETAILED_STATS = false,
        LSYNCSTATS = false,
        MP_TYPE = 2,
        MBX_SIZE = 32000000,
        NPROC = 144,      ! Number of MPI ranks (cores of no OpenMP)
        NOUTPUT = 1,
    /
    
    &NAEPHY
        LEPHYS = true,
        LERADI = true,
        LELAIV = false,
    /
    
    &NAERAD
        NRPROMA = -8, 
        CRTABLEDIR = "./rtables/",  ! Modify this for your installation, note trailing /
    /
    
    &NAMPAR0
    /
    &NAEPHY
    /
    &NAERAD
    /
    

    Then you can use the following python3 script to reproduce the error:

    import f90nml
    
    nml = f90nml.read("fort.4")
    
    changes = { 
        "namdim": {
            "nproma": 0
        }   
    }
    
    nml.patch(changes)
    
    print(nml)
    print("Changing variables in non-repeated sections works!")
    print()
    print("Here comes the problem:")
    
    changes = { 
        "nampar0": {
            "lstats": False
        },  
        "naephy": {
            "leradi": False
        },
        "naerad": {
            "nrproma": 0
        },
    }
    
    nml.patch(changes)
    

    Suggested solution

    Ideally, the user should be reported which variables are problematic in this context. I have sketched a very rough solution with the desired behaviour for these lines: https://github.com/marshallward/f90nml/blob/2a8663aa7dfc06a446fbab6aae20d928f694f3fb/f90nml/namelist.py#L649-L658

    The rough solution:

        def patch(self, nml_patch):
            """Update the namelist from another partial or full namelist.
    
            This is different from the intrinsic `update()` method, which replaces
            a namelist section.  Rather, it updates the values within a section.
            """
    +       conflicting_repeated_secs = [] 
            for sec in nml_patch:
                if sec not in self:
                    self[sec] = Namelist()
    -           self[sec].update(nml_patch[sec])
    +           try: 
    +               self[sec].update(nml_patch[sec])
    +           except AttributeError:
    +               conflicting_repeated_secs.append(sec)
    +       if conflicting_repeated_secs:
    +           raise AttributeError(
    +               "The following sections are repeated in the namelist and cannot be "
    +               f"updated: {conflicting_repeated_secs}"
    +           )  
    
    opened by mandresm 1
  • patching of fortran types only works for first occurence

    patching of fortran types only works for first occurence

    Given the namelist file setup.nml

    &setup
     test%var1 = 1
     test%var2 = 2
    /
    

    and the patch

    patch={'setup': {'test':{'var2': 3}}}
    

    the call

    f90nml.patch('setup.nml', patch, 'patched_setup.nml')
    

    does not patch anything.

    However, patching the first occurrence of test% in setup.nml works

    patch={'setup': {'test':{'var1': 3}}}
    
    opened by rainbowsend 1
  • Problem when loading a namefile

    Problem when loading a namefile

    Hi I am trying to load a namefile from the simulation code Ramses (name = namelist.txt). It works most of the time but I have a run when this fails and gives the following message (see below). This may be just that my namefile is wrongly formatted. Is there a way for me to easily read the namelist and see where the formatting may be wrong?

    Apologies for the naive question.

    Thanks for any tips that may help.

    Eric

    ====================================== [within ipython]

    nm = "namelist.txt" f90nml.read(nm)


    AssertionError                            Traceback (most recent call last)
    Input In [10], in <cell line: 1>()
    ----> 1 f90nml.read(nm)
    
    File ~/.local/lib/python3.9/site-packages/f90nml/__init__.py:31, in read(nml_path)
         13 """Parse a Fortran namelist file and return its contents.
         14 
         15 File object usage:
       (...)
         28 >>> nml = parser.read(nml_file)
         29 """
         30 parser = Parser()
    ---> 31 return parser.read(nml_path)
    
    File ~/.local/lib/python3.9/site-packages/f90nml/parser.py:281, in Parser.read(self, nml_fname, nml_patch_in, patch_fname)
        279 nml_file = open(nml_fname, 'r') if nml_is_path else nml_fname
        280 try:
    --> 281     return self._readstream(nml_file, nml_patch)
        282 except StopIteration:
        283     raise ValueError('End-of-file reached before end of namelist.')
    
    File ~/.local/lib/python3.9/site-packages/f90nml/parser.py:388, in Parser._readstream(self, nml_file, nml_patch_in)
        385 # Set the next active variable
        386 if self.token in ('=', '(', '%'):
    --> 388     v_name, v_values = self._parse_variable(
        389         g_vars,
        390         patch_nml=grp_patch
        391     )
        393     if v_name in g_vars:
        394         v_prior_values = g_vars[v_name]
    
    File ~/.local/lib/python3.9/site-packages/f90nml/parser.py:554, in Parser._parse_variable(self, parent, patch_nml)
        551 self._update_tokens()
        552 self._update_tokens()
    --> 554 v_att, v_att_vals = self._parse_variable(
        555     v_parent,
        556     patch_nml=v_patch_nml
        557 )
        559 next_value = Namelist()
        560 next_value[v_att] = v_att_vals
    
    File ~/.local/lib/python3.9/site-packages/f90nml/parser.py:566, in Parser._parse_variable(self, parent, patch_nml)
        561     self._append_value(v_values, next_value, v_idx)
        563 else:
        564     # Construct the variable array
    --> 566     assert self.token == '='
        567     n_vals = None
        569     self._update_tokens()
    
    AssertionError: 
    
    opened by emsellem 3
  • Cannot parse a namelist without the right boundary of the array

    Cannot parse a namelist without the right boundary of the array

    I have a namelist like this ( q is an eight dimensional array, q(8) ) :

    &example
    q(1) = 1, 2, 3
    q(4) = 4, 5, 6
    /
    

    Fortan can parse correctly and get q = 1, 2, 3, 4, 5, 6, but f90nml cannot parse correctly: Namelist([('example', Namelist([('q', [1, None, None, 4])]))])

    opened by coreur 18
  • Parsing error using array assignment in namelist

    Parsing error using array assignment in namelist

    One of the namelists I am trying to parse using f90nml contains an array assignment of the following syntax

    &example
      arr(1:2)%foo =   1.0,   2.0
      arr(1:2)%bar =   3.0,   4.0
    /
    

    If I try to read/parse this namelist

    import f90nml
    f90nml.reads('''
    &example
      arr(1:2)%foo =   1.0,   2.0
      arr(1:2)%bar =   3.0,   4.0
    /
    ''')
    

    I will get the following error

    Traceback (most recent call last):
      File "<pyshell#1>", line 5, in <module>
        ''')
      File "C:\Python\Python37-64\lib\site-packages\f90nml\__init__.py", line 47, in reads
        return parser.reads(nml_string)
      File "C:\Python\Python37-64\lib\site-packages\f90nml\parser.py", line 280, in reads
        return self._readstream(iter(nml_string.splitlines()))
      File "C:\Python\Python37-64\lib\site-packages\f90nml\parser.py", line 359, in _readstream
        patch_nml=grp_patch
      File "C:\Python\Python37-64\lib\site-packages\f90nml\parser.py", line 459, in _parse_variable
        assert v_idx_bounds[0][1] - v_idx_bounds[0][0] == 1
    AssertionError
    
    bug 
    opened by Cory-Kramer 4
  • Assign absolute/relative path to string variable in CLI

    Assign absolute/relative path to string variable in CLI

    It doesn't seem that the CLI supports the use of '/'.

    Using

    f90nml -v DIR='./path/to/dir' -v ODIR='/path/to/dir' input

    will make DIR have the value '.', and ODIR empty. Escaping doesn't work either.

    opened by mateusffreitas 4
Releases(v1.4.1)
Owner
Marshall Ward
Marshall Ward
MiShell is a multi-platform, multi-architecture project based on the first version (MiShell32)

MiShell is a multi-platform, multi-architecture project based on the first version (MiShell32), which offers super super small reverse shell payloads great for injection in buffer overflow vulnerabil

Kamyar Hatamnezhad 0 Oct 27, 2022
A command-line based, minimal torrent streaming client made using Python and Webtorrent-cli. Stream your favorite shows straight from the command line.

A command-line based, minimal torrent streaming client made using Python and Webtorrent-cli. Installation pip install -r requirements.txt It use

Jonardon Hazarika 17 Dec 11, 2022
Password manager for the CLI simps.

CLI Password Manager Password manager for the CLI simps. Free software: MIT license

1 Dec 30, 2021
Pyrdle - Play Wordle in the CLI. Write an algorithm to play Wordle for you. Ruin all of the fun you've been having

Pyrdle - Play Wordle in the CLI. Write an algorithm to play Wordle for you. Ruin all of the fun you've been having

Charles Tapley Hoyt 11 Feb 11, 2022
The Pythone Script will generate a (.)sh file with reverse shell codes then you can execute the script on the target

Pythone Script will generate a (.)sh file with reverse shell codes then you can execute the script on the targetPythone Script will generate a (.)sh file with reverse shell codes then you can execute

Boy From Future 15 Sep 16, 2022
AlienFX is a CLI and GUI utility to control the lighting effects of your Alienware computer.

AlienFX is a Linux utility to control the lighting effects of your Alienware computer. At present there is a CLI version (alienfx) and a gtk GUI versi

Stephen Harris 218 Dec 26, 2022
Easily turn single threaded command line applications into a fast, multi-threaded application with CIDR and glob support.

Easily turn single threaded command line applications into a fast, multi-threaded application with CIDR and glob support.

Michael Skelton 1k Jan 07, 2023
Container images for portable development environments

Docker Dev Spin up a container to develop from anywhere! To run, just: docker run -ti aghost7/nodejs-dev:boron tmux new Alternatively, if on Linux: p

Jonathan Boudreau 163 Dec 22, 2022
Example of a CLI with python - know the extension of your files.

extensionCLI Example of a CLI with python - know the extension of your files. Usage: Install the CLI: pip3 install -e . Run the command with "ext" + t

ItanuRomero 5 Dec 29, 2022
A supercharged AWS command line interface (CLI).

SAWS Motivation AWS CLI Although the AWS CLI is a great resource to manage your AWS-powered services, it's tough to remember usage of: 70+ top-level c

Donne Martin 5.1k Jan 05, 2023
doq (python docstring generator) extension for coc.nvim

coc-pydocstring doq (python docstring generator) extension for coc.nvim Install CocInstall: :CocInstall coc-pydocstring vim-plug: Plug 'yaegassy/coc-p

yaegassy 27 Jan 04, 2023
flora-dev-cli (fd-cli) is command line interface software to interact with flora blockchain.

Install git clone https://github.com/Flora-Network/fd-cli.git cd fd-cli python3 -m venv venv source venv/bin/activate pip install -e . --extra-index-u

14 Sep 11, 2022
This tool is a free and unlimited python CLI for google translate. based on google_trans_new.

GoTransPy A free and unlimited python CLI for google translate based on google_trans_new. It's very easy to use and solve the problem that the old api

Youssef Mohamed 2 Jan 10, 2022
RSS reader client for CLI (Command Line Interface),

rReader is RSS reader client for CLI(Command Line Interface)

Lee JunHaeng 10 Dec 24, 2022
Python commandline tool for remembering linux/terminal commands

ehh Remember linux commands Commandline tool for remembering linux/terminal commands. It stores your favorite commands in ~/ehh.json in your homedir a

56 Nov 10, 2022
Python Library and CLI for exporting MySQL databases

expdb Python library and CLI for exporting MySQL databases Installation Pre-requisites MySQL server Python 3.9+ Using git Clone the repository to your

Devansh Singh 1 Nov 29, 2021
Python3 parser for Apple's crash reports

pyCrashReport in intended for analyzing crash reports from Apple devices into a clearer view, without all the thread listing and loaded images, just the actual data you really need to debug the probl

7 Aug 19, 2022
Doro is a CLI based pomodoro app and countdown timer application built using python.

Doro - CLI based pomodoro app Doro is a CLI based pomodoro app and countdown timer application built using python. Install $ pip install doro Usage Po

Suresh Kumar 14 May 23, 2022
Key-control - A tool for add keys to your Termux app

Key-Control Is a tool for add keys to your Termux app. Cara Penginstalan $ pkg u

Beereva.id 1 Feb 14, 2022
A basic molecule viewer written in Python, using curses; Thus, meant for linux terminals

asciiMOL A basic molecule viewer written in Python, using curses; Thus, meant for linux terminals. This is an alpha version, featuring: Opening defaul

Dominik Behrens 328 Dec 11, 2022