f90nml - A Fortran namelist parser, generator, and editor

Overview

f90nml - A Fortran namelist parser, generator, and editor

A Python module and command line tool for parsing Fortran namelist files

https://travis-ci.org/marshallward/f90nml.svg?branch=master https://ci.appveyor.com/api/projects/status/bcugyoqxiyyvemy8?svg=true https://coveralls.io/repos/marshallward/f90nml/badge.svg?branch=master

Documentation

The complete documentation for f90nml is available from Read The Docs.

http://f90nml.readthedocs.org/en/latest/

About f90nml

f90nml is a Python module and command line tool that provides a simple interface for the reading, writing, and modifying Fortran namelist files.

A namelist file is parsed and converted into an Namelist object, which behaves like a standard Python dict. Values are converted from Fortran data types to equivalent primitive Python types.

The command line tool f90nml can be used to modify individual values inside of a shell environment. It can also be used to convert the data between namelists and other configuration formats. JSON and YAML formats are currently supported.

Quick usage guide

To read a namelist file sample.nml which contains the following namelists:

&config_nml
   input = 'wind.nc'
   steps = 864
   layout = 8, 16
   visc = 1.0e-4
   use_biharmonic = .false.
/

we would use the following script:

import f90nml
nml = f90nml.read('sample.nml')

which would would point nml to the following dict:

nml = {
    'config_nml': {
        'input': 'wind.nc',
        'steps': 864,
        'layout': [8, 16],
        'visc': 0.0001,
        'use_biharmonic': False
    }
}

File objects can also be used as inputs:

with open('sample.nml') as nml_file:
    nml = f90nml.read(nml_file)

To modify one of the values, say steps, and save the output, just manipulate the nml contents and write to disk using the write function:

nml['config_nml']['steps'] = 432
nml.write('new_sample.nml')

Namelists can also be saved to file objects:

with open('target.nml') as nml_file:
   nml.write(nml_file)

To modify a namelist but preserve its comments and formatting, create a namelist patch and apply it to a target file using the patch function:

patch_nml = {'config_nml': {'visc': 1e-6}}
f90nml.patch('sample.nml', patch_nml, 'new_sample.nml')

Command line interface

A command line tool is provided to manipulate namelist files within the shell:

$ f90nml config.nml -g config_nml -v steps=432
&config_nml
   input = 'wind.nc'
   steps = 432
   layout = 8, 16
   visc = 1.0e-4
   use_biharmonic = .false.
/

See the documentation for details.

Installation

f90nml is available on PyPI and can be installed via pip:

$ pip install f90nml

The latest version of f90nml can be installed from source:

$ git clone https://github.com/marshallward/f90nml.git
$ cd f90nml
$ pip install .

Package distribution

f90nml is not distributed through any official packaging tools, but it is available on Arch Linux via the AUR:

$ git clone https://aur.archlinux.org/python-f90nml.git
$ cd python-f90nml
$ makepkg -sri

Volunteers are welcome to submit and maintain f90nml on other distributions.

Local install

Users without install privileges can append the --user flag to pip from the top f90nml directory:

$ pip install --user .

If pip is not available, then setup.py can still be used:

$ python setup.py install --user

When using setup.py locally, some users have reported that --prefix= may need to be appended to the command:

$ python setup.py install --user --prefix=

YAML support

The command line tool offers support for conversion between namelists and YAML formatted output. If PyYAML is already installed, then no other steps are required. To require YAML support, install the yaml extras package:

$ pip install f90nml[yaml]

To install as a user:

$ pip install --user .[yaml]

Contributing to f90nml

Users are welcome to submit bug reports, feature requests, and code contributions to this project through GitHub. More information is available in the Contributing guidelines.

Comments
  • Add file arguments

    Add file arguments

    Some API questions are how to handle force= and the default output name for patching; I decided to ignore force for files. For the latter issue, I raise ValueError when nml_fname is a file since there is no sane default.

    (personally, if it was my library, I would rip out the default name functionality entirely, and make it so that when there is no output file or path, it will still build and return the patched Namelist object without writing a patched namelist file. But that's just me)

    Addresses #25 (although there is no string API here, at least yet)

    opened by ExpHP 25
  • Order of repeated blocks not preserved when mixed with other named blocks

    Order of repeated blocks not preserved when mixed with other named blocks

    File file contains

    &block00
    i=1
    k=2
    &end
    &block01
    m=0
    n=1
    &end
    &block00
    j=3
    z=4
    &end
    

    The statement f90nml.write(f90nml.read(file),outfile) will result in outfile containing

    &block00
    i=1
    k=2
    /
    &block00
    j=3
    z=4
    /
    &block01
    m=0
    n=1
    /
    

    where order of namelist blocks has not been preserved.

    opened by jpf-x 24
  • Strings starting with + are parsed as a list

    Strings starting with + are parsed as a list

    I have a file formatted this way

    &iostuff
    output_folder = +out
    /
    

    When parsed, ['iostuff']['output_folder'] equals ['+', 'out']. I guess this is done so that if the variable out is defined somewhere else, this list is replaced by the corresponding numerical value. But if it isn't defined, the string '+out' should be returned instead. Maybe a simple solution would be having a way to tell f90nml that some variables are strings and should be considered as such.

    opened by amorison 24
  • Namelist with repeated groups doesn't convert to string

    Namelist with repeated groups doesn't convert to string

    Trying to convert a namelist with repeated groups to a string seems to fail. If I do the following:

    >>> import f90nml
    >>> nml = f90nml.reads("""
    ... &foo
    ...  a = 1
    ... /
    ... &foo
    ...  b = 2
    ... /
    ... """)
    >>> str(nml)
    "Namelist([('foo', [Namelist([('a', 1)]), Namelist([('b', 2)])])])"
    

    I was expecting str(nml) to return "&foo\n a = 1\n/\n&foo\n b = 2\n/\n", as is the case when the group names do not repeat. Interestingly, if I use nml.write('out.nml') the the output file is formatted correctly.

    opened by chunderbunny 21
  • Write bug in v0.20

    Write bug in v0.20

    I'm getting an odd bug when trying the write the following namelist:

     &a
     b%c_d_E(1)%ID = 1,
     b%c_d_E(2)%ID = 2,
     /
    

    Code is:

        import f90nml
        data = f90nml.read('test.nml')
        with open('test_RESAVED.nml','w') as f:
            data.write(f, force=True)
    

    Error message is:

    Traceback (most recent call last):
      File "test.py", line 309, in <module>
        data.write(f, force=True)
      File "C:\Anaconda3\lib\site-packages\f90nml\namelist.py", line 232, in write
        self.write_nmlgrp(grp_name, grp_vars, nml_file)
      File "C:\Anaconda3\lib\site-packages\f90nml\namelist.py", line 253, in write_nmlgrp
        for v_str in self.var_strings(v_name, v_val, v_start=v_start):
      File "C:\Anaconda3\lib\site-packages\f90nml\namelist.py", line 302, in var_strings
        v_strs = self.var_strings(v_title, f_vals, v_start=v_start_new)
      File "C:\Anaconda3\lib\site-packages\f90nml\namelist.py", line 313, in var_strings
        i_s = v_start[::-1][len(v_idx)]
    TypeError: 'NoneType' object is not subscriptable
    

    Oddly, when I change c_d_E to c_d_e, it works fine.

    I just updated to the latest release using pip. I've been able to read variables like this before, so something has changed.

    opened by jacobwilliams 20
  • Add a dependency on the yaml module

    Add a dependency on the yaml module

    Trying to use f90nml after installing on macOS via pip3 install f90nml with the -f yaml flag results in:

    f90nml: error: YAML module could not be found.
    

    I think you should just add an explicit dependency on pyyaml. Or, better yet, if there's a way to add a recommended (installed by default) dependency on pyyaml that someone could opt out of. (I'm not familiar enough with python packaging to know for certain if this is possible one way or the other.)

    opened by zbeekman 19
  • Change to accept NumPy arrays and similar objects

    Change to accept NumPy arrays and similar objects

    Hi there! We've been using f90nml in Pyrokinetics to read/write input files to plasma physics codes, and we ran into an issue where NumPy 0D arrays would propagate unnoticed into a Namelist and cause an exception to be thrown on calling Namelist.write. I've added a small change to Namelist._f90repr to detect 0D arrays and convert them to their equivalent primitive type. Nothing changes if NumPy is not installed.

    Let me know if you'd like me to add further test cases, or if you'd prefer to avoid including these sorts of edge cases in your library.

    opened by LiamPattinson 18
  • Optimizing performance

    Optimizing performance

    I was wondering if there were possibilities for optimizing performance for very large namelist files. I'm in the process of doing some benchmarks for some large files (say around 1500 lines, with multiple namelists and pretty much all variable types). I can only get around 1.7 calls of f90nml.read() per second. I assume that it's the parsing and/or creating of the structures that is the bottleneck (actually reading in the lines should only take a fraction of a second), but I'm going to investigate further.

    One thing I was wondering if parsing of different namelists in the same file could take place in parallel? I don't know if such a thing is possible or not, but if it is that might be something to explore. Maybe this is something I can try and contribute to, rather than just reporting bugs and asking for features!

    enhancement 
    opened by jacobwilliams 17
  • repetition of null value bug

    repetition of null value bug

    Hi Marshall, Thanks for this great tool!

    The bug occurs for the following variable input line: AXFCLN = 3.0, 3.0, 48*, The 48* denotes a repetition of 48 null values Upon .write(), it is being turned into:
    AXFCLN = 3.0, 3.0, ','*,

    One clue is that everything is fine (output = input) if the trailing comma is removed prior to processing.

    Processing attached files as follows: nml=f90nml.read('repetition_of_null_win.nml') f90nml.patch('repetition_of_null_win.nml', nml, 'patched_data_win.nml') and likewise for the unix line-endings version.

    I'm using latest master commit 151438f on Oct 31, 2019 (ver 1.1.2) My sys.version is: 3.7.5 (tags/v3.7.5:5c02a39a0b, Oct 15 2019, 00:11:34) [MSC v.1916 64 bit (AMD64)] I'm running Win10/64/Pro.

    patched_data_unix.txt patched_data_win.txt repetition_of_null_win.txt repetition_of_null_unix.txt

    opened by frankeye 15
  • whitespace between indices for array in namelist cannot be parsed by nag

    whitespace between indices for array in namelist cannot be parsed by nag

    When writing namelists containing nested structures that need to be indexed, e.g.

    idx_nml
       v(1, 1) = 5
       v(2, 1) = 5
    /
    

    the resulting namelist file cannot be parsed by a program compiled with nag (tested here with version 6.1) , if there is a whitespace between the indices. I was wondering, if that whitespace can be omitted?

    opened by gitporst 15
  • f90nml.read() crashes for namelists without a terminating EOL.

    f90nml.read() crashes for namelists without a terminating EOL.

    I'm trying to transform a list of configuration files into a list of f90nml.Namelist objects. StopIteration is being raised by Tokenizer.update_chars().

    For example, let's say I have two nml files, foo.nml and bar.nml, this bug can be triggered by :

    [f90nml.read(s) for s in ['foo.nml','bar.nml']]
    
    opened by neutrinoceros 14
  • patching a repeated section on a namelist returns an unclear message

    patching a repeated section on a namelist returns an unclear message

    Related to esm-tools/esm_tools#843

    Problem description

    When loading a namelist with repeated sections, and then I try to patch one or several repeated sections, I get the following error:

    AttributeError: 'Cogroup' object has no attribute 'update'
    

    I understand it is not possible to update repeated sections, as it is not clear which one f90nml should patch. However, I find the error message misleading.

    Example

    To reproduce the error you can write the following fort.4 namelist:

    &NAMDIM
        NPROMA = -8, 
    /
    
    &NAMPAR0
        LSTATS = true,
        LDETAILED_STATS = false,
        LSYNCSTATS = false,
        MP_TYPE = 2,
        MBX_SIZE = 32000000,
        NPROC = 144,      ! Number of MPI ranks (cores of no OpenMP)
        NOUTPUT = 1,
    /
    
    &NAEPHY
        LEPHYS = true,
        LERADI = true,
        LELAIV = false,
    /
    
    &NAERAD
        NRPROMA = -8, 
        CRTABLEDIR = "./rtables/",  ! Modify this for your installation, note trailing /
    /
    
    &NAMPAR0
    /
    &NAEPHY
    /
    &NAERAD
    /
    

    Then you can use the following python3 script to reproduce the error:

    import f90nml
    
    nml = f90nml.read("fort.4")
    
    changes = { 
        "namdim": {
            "nproma": 0
        }   
    }
    
    nml.patch(changes)
    
    print(nml)
    print("Changing variables in non-repeated sections works!")
    print()
    print("Here comes the problem:")
    
    changes = { 
        "nampar0": {
            "lstats": False
        },  
        "naephy": {
            "leradi": False
        },
        "naerad": {
            "nrproma": 0
        },
    }
    
    nml.patch(changes)
    

    Suggested solution

    Ideally, the user should be reported which variables are problematic in this context. I have sketched a very rough solution with the desired behaviour for these lines: https://github.com/marshallward/f90nml/blob/2a8663aa7dfc06a446fbab6aae20d928f694f3fb/f90nml/namelist.py#L649-L658

    The rough solution:

        def patch(self, nml_patch):
            """Update the namelist from another partial or full namelist.
    
            This is different from the intrinsic `update()` method, which replaces
            a namelist section.  Rather, it updates the values within a section.
            """
    +       conflicting_repeated_secs = [] 
            for sec in nml_patch:
                if sec not in self:
                    self[sec] = Namelist()
    -           self[sec].update(nml_patch[sec])
    +           try: 
    +               self[sec].update(nml_patch[sec])
    +           except AttributeError:
    +               conflicting_repeated_secs.append(sec)
    +       if conflicting_repeated_secs:
    +           raise AttributeError(
    +               "The following sections are repeated in the namelist and cannot be "
    +               f"updated: {conflicting_repeated_secs}"
    +           )  
    
    opened by mandresm 1
  • patching of fortran types only works for first occurence

    patching of fortran types only works for first occurence

    Given the namelist file setup.nml

    &setup
     test%var1 = 1
     test%var2 = 2
    /
    

    and the patch

    patch={'setup': {'test':{'var2': 3}}}
    

    the call

    f90nml.patch('setup.nml', patch, 'patched_setup.nml')
    

    does not patch anything.

    However, patching the first occurrence of test% in setup.nml works

    patch={'setup': {'test':{'var1': 3}}}
    
    opened by rainbowsend 1
  • Problem when loading a namefile

    Problem when loading a namefile

    Hi I am trying to load a namefile from the simulation code Ramses (name = namelist.txt). It works most of the time but I have a run when this fails and gives the following message (see below). This may be just that my namefile is wrongly formatted. Is there a way for me to easily read the namelist and see where the formatting may be wrong?

    Apologies for the naive question.

    Thanks for any tips that may help.

    Eric

    ====================================== [within ipython]

    nm = "namelist.txt" f90nml.read(nm)


    AssertionError                            Traceback (most recent call last)
    Input In [10], in <cell line: 1>()
    ----> 1 f90nml.read(nm)
    
    File ~/.local/lib/python3.9/site-packages/f90nml/__init__.py:31, in read(nml_path)
         13 """Parse a Fortran namelist file and return its contents.
         14 
         15 File object usage:
       (...)
         28 >>> nml = parser.read(nml_file)
         29 """
         30 parser = Parser()
    ---> 31 return parser.read(nml_path)
    
    File ~/.local/lib/python3.9/site-packages/f90nml/parser.py:281, in Parser.read(self, nml_fname, nml_patch_in, patch_fname)
        279 nml_file = open(nml_fname, 'r') if nml_is_path else nml_fname
        280 try:
    --> 281     return self._readstream(nml_file, nml_patch)
        282 except StopIteration:
        283     raise ValueError('End-of-file reached before end of namelist.')
    
    File ~/.local/lib/python3.9/site-packages/f90nml/parser.py:388, in Parser._readstream(self, nml_file, nml_patch_in)
        385 # Set the next active variable
        386 if self.token in ('=', '(', '%'):
    --> 388     v_name, v_values = self._parse_variable(
        389         g_vars,
        390         patch_nml=grp_patch
        391     )
        393     if v_name in g_vars:
        394         v_prior_values = g_vars[v_name]
    
    File ~/.local/lib/python3.9/site-packages/f90nml/parser.py:554, in Parser._parse_variable(self, parent, patch_nml)
        551 self._update_tokens()
        552 self._update_tokens()
    --> 554 v_att, v_att_vals = self._parse_variable(
        555     v_parent,
        556     patch_nml=v_patch_nml
        557 )
        559 next_value = Namelist()
        560 next_value[v_att] = v_att_vals
    
    File ~/.local/lib/python3.9/site-packages/f90nml/parser.py:566, in Parser._parse_variable(self, parent, patch_nml)
        561     self._append_value(v_values, next_value, v_idx)
        563 else:
        564     # Construct the variable array
    --> 566     assert self.token == '='
        567     n_vals = None
        569     self._update_tokens()
    
    AssertionError: 
    
    opened by emsellem 3
  • Cannot parse a namelist without the right boundary of the array

    Cannot parse a namelist without the right boundary of the array

    I have a namelist like this ( q is an eight dimensional array, q(8) ) :

    &example
    q(1) = 1, 2, 3
    q(4) = 4, 5, 6
    /
    

    Fortan can parse correctly and get q = 1, 2, 3, 4, 5, 6, but f90nml cannot parse correctly: Namelist([('example', Namelist([('q', [1, None, None, 4])]))])

    opened by coreur 18
  • Parsing error using array assignment in namelist

    Parsing error using array assignment in namelist

    One of the namelists I am trying to parse using f90nml contains an array assignment of the following syntax

    &example
      arr(1:2)%foo =   1.0,   2.0
      arr(1:2)%bar =   3.0,   4.0
    /
    

    If I try to read/parse this namelist

    import f90nml
    f90nml.reads('''
    &example
      arr(1:2)%foo =   1.0,   2.0
      arr(1:2)%bar =   3.0,   4.0
    /
    ''')
    

    I will get the following error

    Traceback (most recent call last):
      File "<pyshell#1>", line 5, in <module>
        ''')
      File "C:\Python\Python37-64\lib\site-packages\f90nml\__init__.py", line 47, in reads
        return parser.reads(nml_string)
      File "C:\Python\Python37-64\lib\site-packages\f90nml\parser.py", line 280, in reads
        return self._readstream(iter(nml_string.splitlines()))
      File "C:\Python\Python37-64\lib\site-packages\f90nml\parser.py", line 359, in _readstream
        patch_nml=grp_patch
      File "C:\Python\Python37-64\lib\site-packages\f90nml\parser.py", line 459, in _parse_variable
        assert v_idx_bounds[0][1] - v_idx_bounds[0][0] == 1
    AssertionError
    
    bug 
    opened by Cory-Kramer 4
  • Assign absolute/relative path to string variable in CLI

    Assign absolute/relative path to string variable in CLI

    It doesn't seem that the CLI supports the use of '/'.

    Using

    f90nml -v DIR='./path/to/dir' -v ODIR='/path/to/dir' input

    will make DIR have the value '.', and ODIR empty. Escaping doesn't work either.

    opened by mateusffreitas 4
Releases(v1.4.1)
Owner
Marshall Ward
Marshall Ward
CLI Web-CAT interface for people who use VIM.

CLI Web-CAT CLI Web-CAT interface. Installation git clone https://github.com/phuang1024/cliwebcat cd cliwebcat python setup.py bdist_wheel sdist cd di

Patrick 4 Apr 11, 2022
A library for creating text-based graphs in the terminal

tplot is a Python package for creating text-based graphs. Useful for visualizing data to the terminal or log files.

Jeroen Delcour 164 Dec 14, 2022
Dart Version Manager CLI implemented with Python and Typer.

Dart Version Manager CLI implemented with Python and Typer.

EducUp 6 Jun 26, 2022
Terminal with builtin ortholinear keyboard and touch screen as a home automation interface.

OLKB-Terminal Terminal with builtin ortholinear keyboard and touch screen as a home automation interface. Features Step and STLs available for non-com

Jeff Eberl 50 Oct 07, 2022
cmsis-pack-manager is a python module, Rust crate and command line utility for managing current device information that is stored in many CMSIS PACKs

cmsis-pack-manager cmsis-pack-manager is a python module, Rust crate and command line utility for managing current device information that is stored i

pyocd 20 Dec 21, 2022
AML Command Transfer. A lightweight tool to transfer any command line to Azure Machine Learning Services

AML Command Transfer (ACT) ACT is a lightweight tool to transfer any command from the local machine to AML or ITP, both of which are Azure Machine Lea

Microsoft 11 Aug 10, 2022
Execute shell command lines in parallel on Slurm, S(on) of Grid Engine (SGE), PBS/Torque clusters

qbatch Execute shell command lines in parallel on Slurm, S(on) of Grid Engine (SGE), PBS/Torque clusters qbatch is a tool for executing commands in pa

Jon Pipitone 26 Dec 12, 2022
Runs a command in P4wnP1 and displays the output on OLED screen (SH1106)

p4wnp1-oled-terminal Runs a command in P4wnP1 and displays the output on OLED screen (SH1106) Works on Raspberry Pi Zero 2 W Tested successfully on RP

PawnSolo 1 Dec 14, 2021
🔖 Lemnos: A simple, light-weight command-line to-do list manager.

🔖 Lemnos: CLI To-do List Manager This is a simple program that allows one to manage a to-do list via the command-line. Example $ python3 todo.py add

Rohan Sikand 1 Dec 07, 2022
lfb (light file browser) is a terminal file browser

lfb (light file browser) is a terminal file browser. The whole program is a mess as of now. In the feature I will remove the need for external dependencies, tidy up the code, make an actual readme, a

2 Apr 09, 2022
Wordle-textual - Play Wordle from the CLI, using Textual

Wordle, playable from the CLI This project seeks to emulate Wordle in your shell

PhenoM4n4n 3 Mar 29, 2022
A startpage configured aesthetically with terminal-esque link formatting

Terminal-y Startpage Setup Clone the repository, then make an unformatted.txt file following the specifications in example.txt. Run format.py Open ind

belkarx 13 May 01, 2022
Terminal-based keyboard testing

kbdtest kbdtest is a simple Python program that tests keyboard input using an interactive, terminal-based, visual keyboard display. It was originally

Ruunyox 12 Jul 19, 2022
Urial (URI Addition tooL) intelligently updates URIs stored in Finder comments of macOS files

Urial Urial (URI addition tool) is a simple but intelligent command-line tool to add or replace URIs found inside macOS Finder comments. Table of cont

Mike Hucka 3 Sep 14, 2022
Run an FFmpeg command and see the percentage progress and ETA.

Run an FFmpeg command and see the percentage progress and ETA.

25 Dec 22, 2022
Todo list console based application. Todo's save to a seperate file.

Todo list console based application. Todo's save to a seperate file.

1 Dec 24, 2021
NudeNet wrapper made to provide a simple cli interface to the library

Nudenet Wrapper. Small warpper script for NudeNet Made to provide a small and easy to use cli interface with the library. You can indicate a single im

1 Oct 20, 2021
[WIP]An ani-cli like cli tool for movies and webseries

mov-cli A cli to browse and watch movies. Installation This project is a work in progress. However, you can try it out python git clone https://github

166 Dec 30, 2022
triggercmd is a CLI client for the TRIGGERcmd cloud service.

TriggerCMD CLI client triggercmd is a CLI client for the TRIGGERcmd cloud service. installation the triggercmd package is available in PyPI. to instal

Gustavo Soares 7 Oct 18, 2022
Python CLI script to solve wordles.

Wordle Solver Python CLI script to solve wordles. You need at least python 3.8 installed to run this. No dependencies. Sample Usage Let's say the word

Rachel Brindle 1 Jan 16, 2022