jsonextended package

Module contents

a module to extend the python json package functionality;

  • decoding/encoding between the on-disk JSON structure and in-memory nested dictionary structure, including:
    • treating path structures, with nested directories and multiple .json files, as a single json.
    • on-disk indexing of the json structure (using the ijson package)
    • extended data type serialisation (numpy.ndarray, Decimals, pint.Quantities,…)
  • viewing and manipulating the nested dictionaries:
    • enhanced pretty printer
    • Javascript rendered, expandable tree in the Jupyter Notebook
    • filter, merge, flatten, unflatten functions
  • Units schema concept to apply and convert physical units (using the pint package)
  • Parser abstract class for dealing with converting other file formats to JSON

Notes

On-disk indexing of the json structure, before reading into memory, to reduce memory overhead when dealing with large json structures/files (using the ijson package), e.g.

path = get_test_path() %memit jdict1 = to_dict(path,[‘dir1’,’file2’,’meta’],in_memory=True) maximum of 3: 12.242188 MB per loop

%memit jdict1 = to_dict(path,[‘dir1’,’file2’,’meta’],in_memory=False) maximum of 3: 6.996094 MB per loop

Examples

>>> from jsonextended import ejson, edict, utils

>>> path = utils.get_test_path()
>>> path.is_dir()
True

>>> ejson.jkeys(path)
['dir1', 'dir2', 'dir3']

>>> jdict1 = ejson.to_dict(path)
>>> edict.pprint(jdict1,depth=2)
dir1:
  dir1_1: {...}
  file1: {...}
  file2: {...}
dir2:
  file1: {...}
dir3:

>>> jdict2 = ejson.to_dict(path,['dir1','file1'])
>>> edict.pprint(jdict2,depth=1)
initial: {...}
meta: {...}
optimised: {...}
units: {...}

>>> filtered = edict.filter_keys(jdict2,['vol*'],use_wildcards=True)
>>> edict.pprint(filtered)
initial:
  crystallographic:
    volume: 924.62752781
  primitive:
    volume: 462.313764
optimised:
  crystallographic:
    volume: 1063.98960509
  primitive:
    volume: 531.994803

>>> edict.pprint(edict.flatten(filtered))
(initial, crystallographic, volume):   924.62752781
(initial, primitive, volume):          462.313764
(optimised, crystallographic, volume): 1063.98960509
(optimised, primitive, volume):        531.994803