biothings.utils

biothings.utils.aws

biothings.utils.aws.create_bucket(name, region=None, aws_key=None, aws_secret=None, acl=None, ignore_already_exists=False)

Create a S3 bucket “name” in optional “region”. If aws_key and aws_secret are set, S3 client will these, otherwise it’ll use default system-wide setting. “acl” defines permissions on the bucket: “private” (default), “public-read”, “public-read-write” and “authenticated-read”

biothings.utils.aws.download_s3_file(s3key, localfile=None, aws_key=None, aws_secret=None, s3_bucket=None, overwrite=False)

biothings.utils.aws.get_s3_file(s3key, localfile=None, return_what=False, aws_key=None, aws_secret=None, s3_bucket=None)

biothings.utils.aws.get_s3_file_contents(s3key, aws_key=None, aws_secret=None, s3_bucket=None) → bytes

biothings.utils.aws.get_s3_folder(s3folder, basedir=None, aws_key=None, aws_secret=None, s3_bucket=None)

biothings.utils.aws.get_s3_static_website_url(s3key, aws_key=None, aws_secret=None, s3_bucket=None)

biothings.utils.aws.get_s3_url(s3key, aws_key=None, aws_secret=None, s3_bucket=None)

biothings.utils.aws.key_exists(bucket, s3key, aws_key=None, aws_secret=None)

biothings.utils.aws.send_s3_big_file(localfile, s3key, overwrite=False, acl=None, aws_key=None, aws_secret=None, s3_bucket=None, storage_class=None)

Multiparts upload for file bigger than 5GiB

biothings.utils.aws.send_s3_file(localfile, s3key, overwrite=False, permissions=None, metadata=None, content=None, content_type=None, aws_key=None, aws_secret=None, s3_bucket=None, redirect=None)

save a localfile to s3 bucket with the given key. bucket is set via S3_BUCKET it also save localfile’s lastmodified time in s3 file’s metadata

Parameters

redirect (str) – if not None, set the redirect property of the object so it produces a 301 when accessed

biothings.utils.aws.send_s3_folder(folder, s3basedir=None, acl=None, overwrite=False, aws_key=None, aws_secret=None, s3_bucket=None)

biothings.utils.aws.set_static_website(name, aws_key=None, aws_secret=None, index='index.html', error='error.html')

biothings.utils.backend

biothings.utils.common

This module contains util functions may be shared by both BioThings data-hub and web components. In general, do not include utils depending on any third-party modules.

class biothings.utils.common.BiothingsJSONEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)

Bases: JSONEncoder

A class to dump Python Datetime object. json.dumps(data, cls=DateTimeJSONEncoder, indent=indent)

Constructor for JSONEncoder, with sensible defaults.

If skipkeys is false, then it is a TypeError to attempt encoding of keys that are not str, int, float or None. If skipkeys is True, such items are simply skipped.

If ensure_ascii is true, the output is guaranteed to be str objects with all incoming non-ASCII characters escaped. If ensure_ascii is false, the output can contain non-ASCII characters.

If check_circular is true, then lists, dicts, and custom encoded objects will be checked for circular references during encoding to prevent an infinite recursion (which would cause an RecursionError). Otherwise, no such check takes place.

If allow_nan is true, then NaN, Infinity, and -Infinity will be encoded as such. This behavior is not JSON specification compliant, but is consistent with most JavaScript based encoders and decoders. Otherwise, it will be a ValueError to encode such floats.

If sort_keys is true, then the output of dictionaries will be sorted by key; this is useful for regression tests to ensure that JSON serializations can be compared on a day-to-day basis.

If indent is a non-negative integer, then JSON array elements and object members will be pretty-printed with that indent level. An indent level of 0 will only insert newlines. None is the most compact representation.

If specified, separators should be an (item_separator, key_separator) tuple. The default is (’, ‘, ‘: ‘) if indent is None and (‘,’, ‘: ‘) otherwise. To get the most compact JSON representation, you should specify (‘,’, ‘:’) to eliminate whitespace.

If specified, default is a function that gets called for objects that can’t otherwise be serialized. It should return a JSON encodable version of the object or raise a TypeError.

default(o)

Implement this method in a subclass such that it returns a serializable object for o, or calls the base implementation (to raise a TypeError).

For example, to support arbitrary iterators, you could implement default like this:

def default(self, o):
    try:
        iterable = iter(o)
    except TypeError:
        pass
    else:
        return list(iterable)
    # Let the base class default method raise the TypeError
    return JSONEncoder.default(self, o)

class biothings.utils.common.LogPrint(log_f, log=1, timestamp=0)

Bases: object

If this class is set to sys.stdout, it will output both log_f and __stdout__. log_f is a file handler.

close()

fileno()

flush()

pause()

resume()

start()

write(text)

biothings.utils.common.SubStr(input_string, start_string='', end_string='', include=0)

Return the substring between start_string and end_string. If start_string is ‘’, cut string from the beginning of input_string. If end_string is ‘’, cut string to the end of input_string. If either start_string or end_string can not be found from input_string, return ‘’. The end_pos is the first position of end_string after start_string. If multi-occurence,cut at the first position. include=0(default), does not include start/end_string; include=1: include start/end_string.

biothings.utils.common.addsuffix(filename, suffix, noext=False)

Add suffix in front of “.extension”, so keeping the same extension. if noext is True, remove extension from the filename.

async biothings.utils.common.aiogunzipall(folder, pattern, job_manager, pinfo)

Gunzip all files in folder matching pattern. job_manager is used for parallelisation, and pinfo is a pre-filled dict used by job_manager to report jobs in the hub (see bt.utils.manager.JobManager)

biothings.utils.common.anyfile(infile, mode='r')

return a file handler with the support for gzip/zip comppressed files. if infile is a two value tuple, then first one is the compressed file; the second one is the actual filename in the compressed file. e.g., (‘a.zip’, ‘aa.txt’)

biothings.utils.common.ask(prompt, options='YN')

Prompt Yes or No,return the upper case ‘Y’ or ‘N’.

class biothings.utils.common.dotdict

Bases: dict

biothings.utils.common.dump(obj, filename, protocol=4, compress='gzip')

Saves a compressed object to disk protocol version 4 is the default for py3.8, supported since py3.4

biothings.utils.common.dump2gridfs(obj, filename, db, protocol=2)

Save a compressed (support gzip only) object to MongoDB gridfs.

biothings.utils.common.file_newer(source, target)

return True if source file is newer than target file.

biothings.utils.common.filter_dict(d, keys)

Remove keys from dict “d”. “keys” is a list of string, dotfield notation can be used to express nested keys. If key to remove doesn’t exist, silently ignore it

biothings.utils.common.find_classes_subclassing(mods, baseclass)

Given a module or a list of modules, inspect and find all classes which are a subclass of the given baseclass, inside those modules

biothings.utils.common.find_doc(k, keys)

Used by jsonld insertion in www.api.es._insert_jsonld

biothings.utils.common.get_class_from_classpath(class_path)

biothings.utils.common.get_compressed_outfile(filename, compress='gzip')

Get a output file handler with given compress method. currently support gzip/bz2/lzma, lzma only available in py3

biothings.utils.common.get_dotfield_value(dotfield, d)

Explore dictionary d using dotfield notation and return value. Example:

d = {"a":{"b":1}}.
get_dotfield_value("a.b",d) => 1

biothings.utils.common.get_random_string()

biothings.utils.common.get_timestamp()

biothings.utils.common.gunzip(f, pattern='*.gz')

biothings.utils.common.gunzipall(folder, pattern='*.gz')

gunzip all *.gz files in “folder”

class biothings.utils.common.inf

Bases: object

Represents Inf type, but not as a float

biothings.utils.common.is_filehandle(fh)

return True/False if fh is a file-like object

biothings.utils.common.is_float(f)

return True if input is a float.

biothings.utils.common.is_int(s)

return True or False if input string is integer or not.

biothings.utils.common.is_scalar(f)

biothings.utils.common.is_seq(li)

return True if input is either a list or a tuple.

biothings.utils.common.is_str(s)

return True or False if input is a string or not. python3 compatible.

biothings.utils.common.iter_n(iterable, n, with_cnt=False)

Iterate an iterator by chunks (of n) if with_cnt is True, return (chunk, cnt) each time ref http://stackoverflow.com/questions/8991506/iterate-an-iterator-by-chunks-of-n-in-python

biothings.utils.common.json_encode(obj)

Tornado-aimed json encoder, it does the same job as tornado.escape.json_encode but also deals with datetime encoding

biothings.utils.common.json_serial(obj)

JSON serializer for objects not serializable by default json code

biothings.utils.common.list2dict(a_list, keyitem, alwayslist=False)

Return a dictionary with specified keyitem as key, others as values. keyitem can be an index or a sequence of indexes. For example:

li = [['A','a',1],
     ['B','a',2],
     ['A','b',3]]
list2dict(li, 0)---> {'A':[('a',1),('b',3)],
                     'B':('a',2)}

If alwayslist is True, values are always a list even there is only one item in it.

list2dict(li, 0, True)---> {'A':[('a',1),('b',3)],
                            'B':[('a',2),]}

biothings.utils.common.loadobj(filename, mode='file')

Loads a compressed object from disk file (or file-like handler) or MongoDB gridfs file (mode=’gridfs’)

obj = loadobj('data.pyobj')
obj = loadobj(('data.pyobj', mongo_db), mode='gridfs')

biothings.utils.common.md5sum(fname)

biothings.utils.common.merge(x, dx)

Merge dictionary dx (Δx) into dictionary x. If __REPLACE__ key is present in any level z in dx, z in x is replaced, instead of merged, with z in dx.

class biothings.utils.common.nan

Bases: object

Represents NaN type, but not as a float

biothings.utils.common.newer(t0, t1, fmt='%Y%m%d')

t0 and t1 are string of timestamps matching “format” pattern. Return True if t1 is newer than t0.

biothings.utils.common.open_anyfile(infile, mode='r')

a context manager can be used in “with” stmt. accepts a filehandle or anything accepted by anyfile function.

with open_anyfile(‘test.txt’) as in_f:

do_something()

biothings.utils.common.open_compressed_file(filename)

Get a read-only file-handler for compressed file, currently support gzip/bz2/lzma, lzma only available in py3

biothings.utils.common.rmdashfr(top)

Recursively delete dirs and files from “top” directory, then delete “top” dir

biothings.utils.common.run_once()

should_run_task_1 = run_once() print(should_run_task_1()) -> True print(should_run_task_1()) -> False print(should_run_task_1()) -> False print(should_run_task_1()) -> False

should_run_task_2 = run_once() print(should_run_task_2(‘2a’)) -> True print(should_run_task_2(‘2b’)) -> True print(should_run_task_2(‘2a’)) -> False print(should_run_task_2(‘2b’)) -> False …

biothings.utils.common.safe_unicode(s, mask='#')

replace non-decodable char into “#”.

biothings.utils.common.safewfile(filename, prompt=True, default='C', mode='w')

return a file handle in ‘w’ mode,use alternative name if same name exist. if prompt == 1, ask for overwriting,appending or changing name, else, changing to available name automatically.

biothings.utils.common.setup_logfile(logfile)

biothings.utils.common.sizeof_fmt(num, suffix='B')

biothings.utils.common.split_ids(q)

split input query string into list of ids. any of ``”

|,+”`` as the separator,

but perserving a phrase if quoted (either single or double quoted) more detailed rules see: http://docs.python.org/2/library/shlex.html#parsing-rules

e.g.:

>>> split_ids('CDK2 CDK3')
 ['CDK2', 'CDK3']
>>> split_ids('"CDK2 CDK3"

CDk4’)

[‘CDK2 CDK3’, ‘CDK4’]

class biothings.utils.common.splitstr

Bases: str

Type representing strings with space in it

biothings.utils.common.timesofar(t0, clock=0, t1=None)

return the string(eg.’3m3.42s’) for the passed real time/CPU time so far from given t0 (return from t0=time.time() for real time/ t0=time.clock() for CPU time).

biothings.utils.common.traverse(obj, leaf_node=False)

Output path-dictionary pairs. For example, input: {

‘exac_nontcga’: {‘af’: 0.00001883}, ‘gnomad_exome’: {‘af’: {‘af’: 0.0000119429, ‘af_afr’: 0.000123077}}, ‘snpeff’: {‘ann’: [{‘effect’: ‘intron_variant’,

‘feature_id’: ‘NM_014672.3’}, {‘effect’: ‘intron_variant’, ‘feature_id’: ‘NM_001256678.1’}]}

} will be translated to a generator: (

(“exac_nontcga”, {“af”: 0.00001883}), (“gnomad_exome.af”, {“af”: 0.0000119429, “af_afr”: 0.000123077}), (“gnomad_exome”, {“af”: {“af”: 0.0000119429, “af_afr”: 0.000123077}}), (“snpeff.ann”, {“effect”: “intron_variant”, “feature_id”: “NM_014672.3”}), (“snpeff.ann”, {“effect”: “intron_variant”, “feature_id”: “NM_001256678.1”}), (“snpeff.ann”, [{ … },{ … }]), (“snpeff”, {“ann”: [{ … },{ … }]}), (‘’, {‘exac_nontcga’: {…}, ‘gnomad_exome’: {…}, ‘snpeff’: {…}})

) or when traversing leaf nodes: (

(‘exac_nontcga.af’, 0.00001883), (‘gnomad_exome.af.af’, 0.0000119429), (‘gnomad_exome.af.af_afr’, 0.000123077), (‘snpeff.ann.effect’, ‘intron_variant’), (‘snpeff.ann.feature_id’, ‘NM_014672.3’), (‘snpeff.ann.effect’, ‘intron_variant’), (‘snpeff.ann.feature_id’, ‘NM_001256678.1’)

)

biothings.utils.common.uncompressall(folder)

Try to uncompress any known archive files in folder

biothings.utils.common.untargzall(folder, pattern='*.tar.gz')

gunzip and untar all *.tar.gz files in “folder”

biothings.utils.common.unxzall(folder, pattern='*.xz')

unxz all xz files in “folder”, in “folder”

biothings.utils.common.unzipall(folder, pattern='*.zip')

unzip all zip files in “folder”, in “folder”

biothings.utils.configuration

class biothings.utils.configuration.ConfigAttrMeta(confmod: biothings.utils.configuration.MetaField = <factory>, section: biothings.utils.configuration.Text = <factory>, description: biothings.utils.configuration.Paragraph = <factory>, readonly: biothings.utils.configuration.Flag = <factory>, hidden: biothings.utils.configuration.Flag = <factory>, invisible: biothings.utils.configuration.Flag = <factory>)

Bases: object

asdict()

commit()

confmod: MetaField

description: Paragraph

feed(field, value)

hidden: Flag

invisible: Flag

readonly: Flag

reset()

section: Text

update(meta)

class biothings.utils.configuration.ConfigLine(seq)

Bases: UserString

PATTERNS = (('hidden', re.compile('^#-\\s*hide\\s*-#\\s*$'), <function ConfigLine.<lambda>>), ('invisible', re.compile('^#-\\s*invisible\\s*-#\\s*$'), <function ConfigLine.<lambda>>), ('readonly', re.compile('^#-\\s*readonly\\s*-#\\s*$'), <function ConfigLine.<lambda>>), ('section', re.compile('^#\\*\\s*(.*)\\s*\\*#\\s*$'), <function ConfigLine.<lambda>>), ('description', re.compile('.*\\s*#\\s+(.*)$'), <function ConfigLine.<lambda>>))

match()

class biothings.utils.configuration.ConfigLines(initlist=None)

Bases: UserList

parse(attrs=())

class biothings.utils.configuration.ConfigurationDefault(default, desc)

Bases: object

exception biothings.utils.configuration.ConfigurationError

Bases: Exception

class biothings.utils.configuration.ConfigurationValue(code)

Bases: object

type to wrap default value when it’s code and needs to be interpreted later code is passed to eval() in the context of the whole “config” dict (so for instance, paths declared before in the configuration file can be used in the code passed to eval) code will also be executed through exec() if eval() raised a syntax error. This would happen when code contains statements, not just expression. In that case, a variable should be created in these statements (named the same as the original config variable) so the proper value can be through ConfigurationManager.

get_value(name, conf)

Return value by eval’ing code in self.code, in the context of given configuration dict (namespace), for given config parameter name.

class biothings.utils.configuration.ConfigurationWrapper(default_config, conf)

Bases: object

Wraps and manages configuration access and edit. A singleton instance is available throughout all hub apps using biothings.config or biothings.hub.config after calling import biothings.hub. In addition to providing config value access, either from config files or database, config manager can supersede attributes of a class with values coming from the database, allowing dynamic configuration of hub’s elements.

When constructing a ConfigurationWrapper instance, variables will be defined with default values coming from default_config, then they can be overridden by conf’s values, or new variables will be added if not defined in default_conf. Only metadata come from default_config will be used.

get_value_from_db(name)

get_value_from_file(name)

property modified

property readonly

reset(name=None)

show()

store_value_to_db(name, value)

supersede(klass)

supersede class variable with db values

class biothings.utils.configuration.Flag(value=None)

Bases: MetaField

default

alias of bool

feed(value)

class biothings.utils.configuration.MetaField(value=None)

Bases: object

clear()

default

alias of None

feed(value)

property value

class biothings.utils.configuration.Paragraph(value=None)

Bases: MetaField

default

alias of list

feed(value)

property value

class biothings.utils.configuration.Text(value=None)

Bases: MetaField

feed(value)

biothings.utils.configuration.is_jsonable(x)

biothings.utils.configuration.set_default_folder(data_archive_root, sub_folder)

set default sub folder based on data_archive_root

biothings.utils.dataload

Utility functions for parsing flatfiles, mapping to JSON, cleaning.

class biothings.utils.dataload.MinType

Bases: object

biothings.utils.dataload.alwayslist(value)

If input value if not a list/tuple type, return it as a single value list.

biothings.utils.dataload.boolean_convert(d, convert_keys=None, level=0)

Explore document d and specified convert keys to boolean. Use dotfield notation for inner keys

biothings.utils.dataload.dict_apply(d, key, value, sort=True)

add value to d[key], append it if key exists

>>> d = {'a': 1}
>>> dict_apply(d, 'a', 2)
 {'a': [1, 2]}
>>> dict_apply(d, 'a', 3)
 {'a': [1, 2, 3]}
>>> dict_apply(d, 'b', 2)
 {'a': 1, 'b': 2}

biothings.utils.dataload.dict_attrmerge(dict_li, removedup=True, sort=True, special_fns=None)

dict_attrmerge([{‘a’: 1, ‘b’:[2,3]},

{‘a’: [1,2], ‘b’:[3,5], ‘c’=4}])

sould return

{‘a’: [1,2], ‘b’:[2,3,5], ‘c’=4}

special_fns is a dictionary of {attr: merge_fn}

used for some special attr, which need special merge_fn e.g., {‘uniprot’: _merge_uniprot}

biothings.utils.dataload.dict_convert(_dict, keyfn=None, valuefn=None)

Return a new dict with each key converted by keyfn (if not None), and each value converted by valuefn (if not None).

biothings.utils.dataload.dict_nodup(_dict, sort=True)

biothings.utils.dataload.dict_sweep(d, vals=None, remove_invalid_list=False)

Remove keys whos values are “.”, “-”, “”, “NA”, “none”, ” ” and remove empty dictionaries

Parameters

• d (dict) – a dictionary

• vals (str or list) – a string or list of strings to sweep, or None to use the default values

• remove_invalid_list (boolean) –

  when true, will remove key for which list has only one value, which is part of “vals”. Ex:

  test_dict = {'gene': [None, None], 'site': ["Intron", None], 'snp_build' : 136}
  

  with remove_invalid_list == False:

  {'gene': [None], 'site': ['Intron'], 'snp_build': 136}
  

  with remove_invalid_list == True:

  {'site': ['Intron'], 'snp_build': 136}
  

biothings.utils.dataload.dict_to_list(gene_d)

return a list of genedoc from genedoc dictionary and make sure the “_id” field exists.

biothings.utils.dataload.dict_traverse(d, func, traverse_list=False)

Recursively traverse dictionary d, calling func(k,v) for each key/value found. func must return a tuple(new_key,new_value)

biothings.utils.dataload.dict_walk(dictionary, key_func)

Recursively apply key_func to dict’s keys

biothings.utils.dataload.dupline_seperator(dupline, dup_sep, dup_idx=None, strip=False)

for a line like this:

a   b1,b2  c1,c2

return a generator of this list (breaking out of the duplicates in each field):

[(a,b1,c1),
 (a,b2,c1),
 (a,b1,c2),
 (a,b2,c2)]

Example:

dupline_seperator(dupline=['a', 'b1,b2', 'c1,c2'],
                  dup_idx=[1,2],
                  dup_sep=',')

if dup_idx is None, try to split on every field. if strip is True, also tripe out of extra spaces.

biothings.utils.dataload.file_merge(infiles, outfile=None, header=1, verbose=1)

merge a list of input files with the same format. if header will be removed from the 2nd files in the list.

biothings.utils.dataload.float_convert(d, include_keys=None, exclude_keys=None)

Convert elements in a document to floats.

By default, traverse all keys If include_keys is specified, only convert the list from include_keys a.b, a.b.c If exclude_keys is specified, only exclude the list from exclude_keys

Parameters

• d – a dictionary to traverse keys on

• include_keys – only convert these keys (optional)

• exclude_keys – exclude all other keys except these keys (optional)

Returns

generate key, value pairs

biothings.utils.dataload.id_strip(id_list)

biothings.utils.dataload.int_convert(d, include_keys=None, exclude_keys=None)

Convert elements in a document to integers.

By default, traverse all keys If include_keys is specified, only convert the list from include_keys a.b, a.b.c If exclude_keys is specified, only exclude the list from exclude_keys

Parameters

• d – a dictionary to traverse keys on

• include_keys – only convert these keys (optional)

• exclude_keys – exclude all other keys except these keys (optional)

Returns

generate key, value pairs

biothings.utils.dataload.list2dict(a_list, keyitem, alwayslist=False)

Return a dictionary with specified keyitem as key, others as values. keyitem can be an index or a sequence of indexes. For example:

li=[['A','a',1],
    ['B','a',2],
    ['A','b',3]]
list2dict(li,0)---> {'A':[('a',1),('b',3)],
                     'B':('a',2)}

If alwayslist is True, values are always a list even there is only one item in it:

list2dict(li,0,True)---> {'A':[('a',1),('b',3)],
                          'B':[('a',2),]}

biothings.utils.dataload.list_itemcnt(a_list)

Return number of occurrence for each type of item in the list.

biothings.utils.dataload.list_nondup(a_list)

biothings.utils.dataload.list_split(d, sep)

Split fields by sep into comma separated lists, strip.

biothings.utils.dataload.listitems(a_list, *idx)

Return multiple items from list by given indexes.

biothings.utils.dataload.listsort(a_list, by, reverse=False, cmp=None, key=None)

Given list is a list of sub(list/tuple.) Return a new list sorted by the ith(given from “by” item) item of each sublist.

biothings.utils.dataload.llist(li, sep='\t')

Nicely output the list with each item a line.

biothings.utils.dataload.merge_dict(dict_li, attr_li, missingvalue=None)

Merging multiple dictionaries into a new one. Example:

In [136]: d1 = {'id1': 100, 'id2': 200}
In [137]: d2 = {'id1': 'aaa', 'id2': 'bbb', 'id3': 'ccc'}
In [138]: merge_dict([d1,d2], ['number', 'string'])
Out[138]:
{'id1': {'number': 100, 'string': 'aaa'},
'id2': {'number': 200, 'string': 'bbb'},
'id3': {'string': 'ccc'}}
In [139]: merge_dict([d1,d2], ['number', 'string'], missingvalue='NA')
Out[139]:
{'id1': {'number': 100, 'string': 'aaa'},
'id2': {'number': 200, 'string': 'bbb'},
'id3': {'number': 'NA', 'string': 'ccc'}}

biothings.utils.dataload.merge_duplicate_rows(rows, db)

@param rows: rows to be grouped by @param db: database name, string

biothings.utils.dataload.merge_root_keys(doc1, doc2, exclude=None)

Ex: d1 = {“_id”:1,”a”:”a”,”b”:{“k”:”b”}}

d2 = {“_id”:1,”a”:”A”,”b”:{“k”:”B”},”c”:123}

Both documents have the same _id, and 2 root keys, “a” and “b”. Using this storage, the resulting document will be:

{‘_id’: 1, ‘a’: [‘A’, ‘a’], ‘b’: [{‘k’: ‘B’}, {‘k’: ‘b’}],”c”:123}

biothings.utils.dataload.merge_struct(v1, v2, aslistofdict=None)

biothings.utils.dataload.normalized_value(value, sort=True)

Return a “normalized” value: 1. if a list, remove duplicate and sort it 2. if a list with one item, convert to that single item only 3. if a list, remove empty values 4. otherwise, return value as it is.

biothings.utils.dataload.rec_handler(infile, block_end='\n', skip=0, include_block_end=False, as_list=False)

A generator to return a record (block of text) at once from the infile. The record is separated by one or more empty lines by default. skip can be used to skip top n-th lines if include_block_end is True, the line matching block_end will also be returned. if as_list is True, return a list of lines in one record.

biothings.utils.dataload.safe_type(f, val)

Convert an input string to int/float/… using passed function. If the conversion fails then None is returned. If value of a type other than a string then the original value is returned.

biothings.utils.dataload.tab2dict(datafile, cols, key, alwayslist=False, **kwargs)

biothings.utils.dataload.tab2dict_iter(datafile, cols, key, alwayslist=False, **kwargs)

biothings.utils.dataload.tab2list(datafile, cols, **kwargs)

biothings.utils.dataload.tabfile_feeder(datafile, header=1, sep='\t', includefn=None, coerce_unicode=True, assert_column_no=None)

a generator for each row in the file.

biothings.utils.dataload.tabfile_tester(datafile, header=1, sep='\t')

biothings.utils.dataload.to_boolean(val, true_str=None, false_str=None)

Normlize str value to boolean value

biothings.utils.dataload.to_float(val)

convert an input string to int

biothings.utils.dataload.to_int(val)

convert an input string to float

biothings.utils.dataload.to_number(val)

convert an input string to int/float.

biothings.utils.dataload.traverse_keys(d, include_keys=None, exclude_keys=None)

Return all key, value pairs for a document.

By default, traverse all keys If include_keys is specified, only traverse the list from include_kes a.b, a.b.c If exclude_keys is specified, only exclude the list from exclude_keys

if a key in include_keys/exclude_keys is not found in d, it’s skipped quietly.

Parameters

• d – a dictionary to traverse keys on

• include_keys – only traverse these keys (optional)

• exclude_keys – exclude all other keys except these keys (optional)

Returns

generate key, value pairs

biothings.utils.dataload.unique_ids(src_module)

biothings.utils.dataload.unlist(d)

biothings.utils.dataload.unlist_incexcl(d, include_keys=None, exclude_keys=None)

Unlist elements in a document.

If there is 1 value in the list, set the element to that value. Otherwise, leave the list unchanged.

By default, traverse all keys If include_keys is specified, only traverse the list from include_keys a.b, a.b.c If exclude_keys is specified, only exclude the list from exclude_keys

Parameters

• d – a dictionary to unlist

• include_keys – only unlist these keys (optional)

• exclude_keys – exclude all other keys except these keys (optional)

Returns

generate key, value pairs

biothings.utils.dataload.update_dict_recur(d, u)

Update dict d with dict u’s values, recursively (so existing values in d but not in u are kept even if nested)

biothings.utils.dataload.updated_dict(_dict, attrs)

Same as dict.update, but return the updated dictionary.

biothings.utils.dataload.value_convert(_dict, fn, traverse_list=True)

For each value in _dict, apply fn and then update _dict with return the value. if traverse_list is True and a value is a list, apply fn to each item of the list.

biothings.utils.dataload.value_convert_incexcl(d, fn, include_keys=None, exclude_keys=None)

Convert elements in a document using a function fn.

By default, traverse all keys If include_keys is specified, only convert the list from include_keys a.b, a.b.c If exclude_keys is specified, only exclude the list from exclude_keys

Parameters

• d – a dictionary to traverse keys on

• fn – function to convert elements with

• include_keys – only convert these keys (optional)

• exclude_keys – exclude all other keys except these keys (optional)

Returns

generate key, value pairs

biothings.utils.dataload.value_convert_to_number(d, skipped_keys=None)

convert string numbers into integers or floats skip converting certain keys in skipped_keys list

biothings.utils.diff

biothings.utils.doc_traversal

Some utility functions that do document traversal

class biothings.utils.doc_traversal.Queue

Bases: object

isempty()

pop()

get next obj from queue

push(obj)

put obj on queue

exception biothings.utils.doc_traversal.QueueEmptyError

Bases: Exception

class biothings.utils.doc_traversal.Stack

Bases: object

isempty()

pop()

push(obj)

put obj on stack

exception biothings.utils.doc_traversal.StackEmptyError

Bases: Exception

biothings.utils.doc_traversal.breadth_first_recursive_traversal(doc, path=None)

doesn’t exactly implement breadth first ordering it seems, not sure why…

biothings.utils.doc_traversal.breadth_first_traversal(doc)

Yield a 2 element tuple for every k, v pair in document items (nodes are visited in breadth first order k is itself a tuple of keys annotating the path for this node (v) to root v is the node value

biothings.utils.doc_traversal.depth_first_recursive_traversal(doc, path=None)

biothings.utils.doc_traversal.depth_first_traversal(doc)

Yield a 2 element tuple for every k, v pair in document items (nodes are visited in depth first order k is itself a tuple of keys annotating the path for this node (v) to root v is the node value

biothings.utils.docs

biothings.utils.docs.exists_or_null(doc, field, val=None)

biothings.utils.docs.flatten_doc(doc, outfield_sep='.', sort=True)

This function will flatten an elasticsearch document (really any json object). outfield_sep is the separator between the fields in the return object. sort specifies whether the output object should be sorted alphabetically before returning

(otherwise output will remain in traveral order)

biothings.utils.docs.flatten_doc_2(doc, outfield_sep='.', sort=True)

biothings.utils.dotfield

biothings.utils.dotfield.compose_dot_fields_by_fields(genedoc, fields)

reverse funtion of parse_dot_fields

biothings.utils.dotfield.make_object(attr, value)

Create dictionary following the input dot notation and the value Example:

make_object('a.b.c', 100) -->
or make_object(['a','b','c'], 100) -->
{a:{b:{c:100}}}

biothings.utils.dotfield.merge_object(obj1, obj2)

biothings.utils.dotfield.parse_dot_fields({'a': 1, 'b.c': 2, 'b.a.c': 3})

should return {‘a’: 1, ‘b’: {‘a’: {‘c’: 3}, ‘c’: 2}}

biothings.utils.dotstring

biothings.utils.dotstring.key_value(dictionary, key)

Return a generator for all values in a dictionary specific by a dotstirng (key)

if key is not found from the dictionary, None is returned.

Parameters

• dictionary – a dictionary to return values from

• key – key that specifies a value in the dictionary

Returns

generator for values that match the given key

biothings.utils.dotstring.last_element(d, key_list)

Return the last element and key for a document d given a docstring.

A document d is passed with a list of keys key_list. A generator is then returned for all elements that match all keys. Not that there may be a 1-to-many relationship between keys and elements due to lists in the document.

Parameters

• d – document d to return elements from

• key_list – list of keys that specify elements in the document d

Returns

generator for elements that match all keys

biothings.utils.dotstring.list_length(d, field)

Return the length of a list specified by field.

If field represents a list in the document, then return its length. Otherwise return 0.

Parameters

• d – a dictionary

• field – the dotstring field specifying a list

biothings.utils.dotstring.remove_key(dictionary, key)

Remove field specified by the docstring key

Parameters

• dictionary – a dictionary to remove the value from

• key – key that specifies an element in the dictionary

Returns

dictionary after changes have been made

biothings.utils.dotstring.set_key_value(dictionary, key, value)

Set values all values in dictionary matching a dotstring key to a specified value.

if key is not found in dictionary, it just skip quietly.

Parameters

• dictionary – a dictionary to set values in

• key – key that specifies an element in the dictionary

Returns

dictionary after changes have been made

biothings.utils.es

class biothings.utils.es.Collection(colname, db)

Bases: object

count()

find(filter=None, projection=None, *args, **kwargs)

find_one(*args, **kwargs)

insert_one(document, *args, **kwargs)

remove(query)

replace_one(filter, replacement, upsert=False, *args, **kwargs)

save(doc, *args, **kwargs)

update(*args, **kwargs)

update_many(filter, update, upsert=False, *args, **kwargs)

update_one(filter, update, upsert=False, *args, **kwargs)

class biothings.utils.es.Database

Bases: IDatabase

CONFIG = None

property address

Returns sufficient information so a connection to a database can be created. Information can be a dictionary, object, etc… and depends on the actual backend

create_collection(colname)

Create a table/colleciton named colname. If backend is using a schema-based database (ie. SQL), backend should enforce the schema with at least field “_id” as the primary key (as a string).

class biothings.utils.es.ESIndex(client, index_name)

Bases: object

An Elasticsearch Index Wrapping A Client. Counterpart for pymongo.collection.Collection

property doc_type

class biothings.utils.es.ESIndexer(index, doc_type='_doc', es_host='localhost:9200', step=500, step_size=10, number_of_shards=1, number_of_replicas=0, check_index=True, **kwargs)

Bases: object

build_index(**kwargs)

check_index()

Check if index is an alias, and update self._index to point to actual index

TODO: the overall design of ESIndexer is not great. If we are exposing ES

implementation details (such as the abilities to create and delete indices, create and update aliases, etc.) to the user of this Class, then this method doesn’t seem that out of place.

clean_field(field, dryrun=True, step=5000)

remove a top-level field from ES index, if the field is the only field of the doc, remove the doc as well. step is the size of bulk update on ES try first with dryrun turned on, and then perform the actual updates with dryrun off.

count(**kwargs)

count_src(**kwargs)

create_index(**kwargs)

create_repository(repo_name, settings)

delete_doc(id)

delete a doc from the index based on passed id.

delete_docs(ids, step=None)

delete a list of docs in bulk.

delete_index(index=None)

doc_feeder(**kwargs)

doc_feeder_using_helper(**kwargs)

exists(**kwargs)

return True/False if a biothing id exists or not.

exists_index(index: Optional[str] = None)

find_biggest_doc(fields_li, min=5, return_doc=False)

return the doc with the max number of fields from fields_li.

get_alias(index: Optional[str] = None, alias_name: Optional[str] = None) → List[str]

Get indices with alias associated with given index name or alias name

Parameters

• index – name of index

• alias_name – name of alias

Returns

Mapping of index names with their aliases

get_biothing(**kwargs)

get_docs(**kwargs)

Return matching docs for given ids iterable, if not found return None. A generator is returned to the matched docs. If only_source is False, the entire document is returned, otherwise only the source is returned.

get_id_list(**kwargs)

get_indice_names_by_settings(index: Optional[str] = None, sort_by_creation_date=False, reverse=False) → List[str]

Get list of indices names associated with given index name, using indices’ settings

Parameters

• index – name of index

• sort_by_creation_date – sort the result by indice’s creation_date

• reverse – control the direction of the sorting

Returns

list of index names (str)

get_indices_from_snapshots(repo_name, snapshot_name)

get_internal_number_of_replicas()

get_mapping()

return the current index mapping

get_mapping_meta()

return the current _meta field.

get_repository(repo_name)

get_restore_status(index_name=None)

get_settings(index: Optional[str] = None) → Mapping[str, Mapping]

Get indices with settings associated with given index name

Parameters

index – name of index

Returns

Mapping of index names with their settings

get_snapshot_status(repo, snapshot)

get_snapshots(repo_name, snapshot_name)

index(doc, id=None, action='index')

add a doc to the index. If id is not None, the existing doc will be updated.

index_bulk(docs, step=None, action='index')

mexists(**kwargs)

optimize(**kwargs)

optimize the default index.

restore(repo_name, snapshot_name, index_name=None, purge=False)

set_internal_number_of_replicas(number_of_replicas=None)

snapshot(repo, snapshot, mode=None, **params)

update(id, extra_doc, upsert=True)

update an existing doc with extra_doc. allow to set upsert=True, to insert new docs.

update_alias(alias_name: str, index: Optional[str] = None)

Create or update an ES alias pointing to an index

Creates or updates an alias in Elasticsearch, associated with the given index name or the underlying index of the ESIndexer instance.

When the alias name does not exist, it will be created. If an existing alias already exists, it will be updated to only associate with the index.

When the alias name already exists, an exception will be raised, UNLESS the alias name is the same as index name that the ESIndexer is initialized with. In this case, the existing index with the name collision will be deleted, and the alias will be created in its place. This feature is intended for seamless migration from an index to an alias associated with an index for zero-downtime installs.

Parameters

• alias_name – name of the alias

• index – name of the index to associate with alias. If None, the index of the ESIndexer instance is used.

Raises

IndexerException –

update_docs(partial_docs, upsert=True, step=None, **kwargs)

update a list of partial_docs in bulk. allow to set upsert=True, to insert new docs.

update_mapping(m)

update_mapping_meta(meta)

exception biothings.utils.es.IndexerException

Bases: Exception

exception biothings.utils.es.MappingError

Bases: Exception

biothings.utils.es.generate_es_mapping(inspect_doc, init=True, level=0)

Generate an ES mapping according to “inspect_doc”, which is produced by biothings.utils.inspect module

biothings.utils.es.get_api()

biothings.utils.es.get_cmd()

biothings.utils.es.get_data_plugin()

biothings.utils.es.get_es(es_host, timeout=120, max_retries=3, retry_on_timeout=False)

biothings.utils.es.get_event()

biothings.utils.es.get_hub_config()

biothings.utils.es.get_hub_db_conn()

biothings.utils.es.get_last_command()

biothings.utils.es.get_source_fullname(col_name)

biothings.utils.es.get_src_build()

biothings.utils.es.get_src_build_config()

biothings.utils.es.get_src_conn()

biothings.utils.es.get_src_dump()

biothings.utils.es.get_src_master()

biothings.utils.es.verify_ids(doc_iter, es_host, index, doc_type=None, step=100000)

verify how many docs from input interator/list overlapping with existing docs.

biothings.utils.es.wrapper(func)

this wrapper allows passing index and doc_type from wrapped method.

biothings.utils.exclude_ids

class biothings.utils.exclude_ids.ExcludeFieldsById(exclusion_ids, field_lst, min_list_size=1000)

Bases: object

This class provides a framework to exclude fields for certain identifiers. Up to three arguments are passed to this class, an identifier list, a list of fields to remove, and minimum list size. The identifier list is a list of document identifiers to act on. The list of fields are fields that will be removed; they are specified using a dotstring notation. The minimum list size is the minimum number of elements that should be in a list in order for it to be removed. The ‘drugbank’, ‘chebi’, and ‘ndc’ data sources were manually tested with this class.

Fields to truncate are specified by field_lst. The dot-notation is accepted.

biothings.utils.hub

biothings.utils.hub_db

hub_db module is a place-holder for internal hub database functions. Hub DB contains informations about sources, configurations variables, etc… It’s for internal usage. When biothings.config_for_app() is called, this module will be “filled” with the actual implementations from the specified backend (speficied in config.py, or defaulting to MongoDB).

Hub DB can be implemented over different backend, it’s orginally been done using MongoDB, so the dialect is very inspired by pymongo. Any hub db backend implementation must implement the functions and classes below. See biothings.utils.mongo and biothings.utils.sqlit3 for some examples.

class biothings.utils.hub_db.ChangeListener

Bases: object

read()

class biothings.utils.hub_db.ChangeWatcher

Bases: object

classmethod add(listener)

col_entity = {'cmd': 'command', 'hub_config': 'config', 'src_build': 'build', 'src_build_config': 'build_config', 'src_dump': 'source', 'src_master': 'master'}

do_publish = False

event_queue = <Queue at 0x7f842af26740 maxsize=0>

listeners = {}

classmethod monitor(func, entity, op)

classmethod publish()

classmethod wrap(getfunc)

class biothings.utils.hub_db.Collection(colname, db)

Bases: object

Defines a minimal subset of MongoDB collection behavior. Note: Collection instances must be pickleable (if not, __getstate__ can be implemented to deal with those attributes for instance)

Init args can differ depending on the backend requirements. colname is the only one required.

count()

Return the number of documents in the collection

database()

Return the database name

find(*args, **kwargs)

Return an iterable of documents matching criterias defined in *args[0] (which will be a dict). Query dialect is a minimal one, inspired by MongoDB. Dict can contain the name of a key, and the value being searched for. Ex: {“field1”:”value1”} will return all documents where field1 == “value1”. Nested key (field1.subfield1) aren’t supported (no need to implement). Exact matches only are required.

If no query is passed, or if query is an empty dict, return all documents.

find_one(*args, **kwargs)

Return one document from the collection. *args will contain a dict with the query parameters. See also find()

insert_one(doc)

Insert a document in the collection. Raise an error if already inserted

property name

Return the collection/table name

remove(query)

Delete all documents matching ‘query’

replace_one(query, doc)

Replace a document matching ‘query’ (or the first found one) with passed doc

save(doc)

Shortcut to update_one() or insert_one(). Save the document, by either inserting if it doesn’t exist, or update existing one

update(query, what)

Same as update_one() but operate on all documents matching ‘query’

update_one(query, what, upsert=False)

Update one document (or the first matching query). See find() for query parameter. “what” tells how to update the document. $set/$unset/$push operators must be implemented (refer to MongoDB documentation for more). Nested keys operation aren’t necesary.

class biothings.utils.hub_db.IDatabase

Bases: object

This class declares an interface and partially implements some of it, mimicking mongokit.Connection class. It’s used to keep used document model. Any internal backend should implement (derives) this interface

property address

Returns sufficient information so a connection to a database can be created. Information can be a dictionary, object, etc… and depends on the actual backend

collection_names()

Return a list of all collections (or tables) found in this database

create_collection(colname)

Create a table/colleciton named colname. If backend is using a schema-based database (ie. SQL), backend should enforce the schema with at least field “_id” as the primary key (as a string).

biothings.utils.hub_db.backup(folder='.', archive=None)

Dump the whole hub_db database in given folder. “archive” can be pass to specify the target filename, otherwise, it’s randomly generated

Note

this doesn’t backup source/merge data, just the internal data used by the hub

biothings.utils.hub_db.get_api()

Return a Collection instance for api collection/table

biothings.utils.hub_db.get_cmd()

Return a Collection instance for commands collection/table

biothings.utils.hub_db.get_data_plugin()

Return a Collection instance for data_plugin collection/table

biothings.utils.hub_db.get_event()

Return a Collection instance for events collection/table

biothings.utils.hub_db.get_hub_config()

Return a Collection instance storing configuration values

biothings.utils.hub_db.get_hub_db_conn()

Return a Database instance (connection to hub db)

biothings.utils.hub_db.get_last_command()

Return the latest cmd document (according to _id)

biothings.utils.hub_db.get_source_fullname(col_name)

Assuming col_name is a collection created from an upload process, find the main source & sub_source associated.

biothings.utils.hub_db.get_src_build()

Return a Collection instance for src_build collection/table

biothings.utils.hub_db.get_src_build_config()

Return a Collection instance for src_build_hnonfig collection/table

biothings.utils.hub_db.get_src_dump()

Return a Collection instance for src_dump collection/table

biothings.utils.hub_db.get_src_master()

Return a Collection instance for src_master collection/table

biothings.utils.hub_db.restore(archive, drop=False)

Restore database from given archive. If drop is True, then delete existing collections

biothings.utils.hub_db.setup(config)

biothings.utils.info

class biothings.utils.info.DevInfo

Bases: object

get()

class biothings.utils.info.FieldNote(path)

Bases: object

get_field_notes()

Return the cached field notes associated with this instance.

biothings.utils.inspect

This module contains util functions may be shared by both BioThings data-hub and web components. In general, do not include utils depending on any third-party modules. Note: unittests available in biothings.tests.hub

class biothings.utils.inspect.BaseMode

Bases: object

key = None

merge(target, tomerge)

Merge two different maps together (from tomerge into target)

post(mapt, mode, clean)

report(struct, drep, orig_struct=None)

Given a data structure “struct” being inspected, report (fill) “drep” dictionary with useful values for this mode, under drep[self.key] key. Sometimes “struct” is already converted to its analytical value at this point (inspect may count number of dict and would force to pass struct as “1”, instead of the whole dict, where number of keys could be then be reported), “orig_struct” is that case contains the original structure that was to be reported, whatever the pre-conversion step did.

template = {}

class biothings.utils.inspect.DeepStatsMode

Bases: StatsMode

key = '_stats'

merge(target_stats, tomerge_stats)

Merge two different maps together (from tomerge into target)

post(mapt, mode, clean)

report(val, drep, orig_struct=None)

Given a data structure “struct” being inspected, report (fill) “drep” dictionary with useful values for this mode, under drep[self.key] key. Sometimes “struct” is already converted to its analytical value at this point (inspect may count number of dict and would force to pass struct as “1”, instead of the whole dict, where number of keys could be then be reported), “orig_struct” is that case contains the original structure that was to be reported, whatever the pre-conversion step did.

template = {'_stats': {'__vals': [], '_count': 0, '_max': -inf, '_min': inf}}

class biothings.utils.inspect.IdentifiersMode

Bases: RegexMode

ids = None

key = '_ident'

matchers = None

class biothings.utils.inspect.RegexMode

Bases: BaseMode

matchers = []

merge(target, tomerge)

Merge two different maps together (from tomerge into target)

report(val, drep, orig_struct=None)

Given a data structure “struct” being inspected, report (fill) “drep” dictionary with useful values for this mode, under drep[self.key] key. Sometimes “struct” is already converted to its analytical value at this point (inspect may count number of dict and would force to pass struct as “1”, instead of the whole dict, where number of keys could be then be reported), “orig_struct” is that case contains the original structure that was to be reported, whatever the pre-conversion step did.

class biothings.utils.inspect.StatsMode

Bases: BaseMode

flatten_stats(stats)

key = '_stats'

maxminiflist(val, func)

merge(target_stats, tomerge_stats)

Merge two different maps together (from tomerge into target)

report(struct, drep, orig_struct=None)

Given a data structure “struct” being inspected, report (fill) “drep” dictionary with useful values for this mode, under drep[self.key] key. Sometimes “struct” is already converted to its analytical value at this point (inspect may count number of dict and would force to pass struct as “1”, instead of the whole dict, where number of keys could be then be reported), “orig_struct” is that case contains the original structure that was to be reported, whatever the pre-conversion step did.

sumiflist(val)

template = {'_stats': {'_count': 0, '_max': -inf, '_min': inf, '_none': 0}}

biothings.utils.inspect.compute_metadata(mapt, mode)

biothings.utils.inspect.get_converters(modes, logger=<module 'logging' from '/home/docs/.asdf/installs/python/3.10.4/lib/python3.10/logging/__init__.py'>)

biothings.utils.inspect.get_mode_layer(mode)

biothings.utils.inspect.inspect(struct, key=None, mapt=None, mode='type', level=0, logger=<module 'logging' from '/home/docs/.asdf/installs/python/3.10.4/lib/python3.10/logging/__init__.py'>)

Explore struct and report types contained in it.

Parameters

• struct – is the data structure to explore

• mapt – if not None, will complete that type map with passed struct. This is useful when iterating over a dataset of similar data, trying to find a good type summary contained in that dataset.

• level – is for internal purposes, mostly debugging

• mode – see inspect_docs() documentation

biothings.utils.inspect.inspect_docs(docs, mode='type', clean=True, merge=False, logger=<module 'logging' from '/home/docs/.asdf/installs/python/3.10.4/lib/python3.10/logging/__init__.py'>, pre_mapping=False, limit=None, sample=None, metadata=True, auto_convert=True)

Inspect docs and return a summary of its structure:

Parameters

• mode –

  possible values are:

  • ”type”: (default) explore documents and report strict data structure

  • ”mapping”: same as type but also perform test on data so guess best mapping

    (eg. check if a string is splitable, etc…). Implies merge=True

  • ”stats”: explore documents and compute basic stats (count,min,max,sum)

  • ”deepstats”: same as stats but record values and also compute mean,stdev,median

    (memory intensive…)

  • ”jsonschema”, same as “type” but returned a json-schema formatted result

  mode can also be a list of modes, eg. [“type”,”mapping”]. There’s little overhead computing multiple types as most time is spent on actually getting the data.

• clean – don’t delete recorded vqlues or temporary results

• merge – merge scalar into list when both exist (eg. {“val”:..} and [{“val”:…}]

• limit – can limit the inspection to the x first docs (None = no limit, inspects all)

• sample – in combination with limit, randomly extract a sample of ‘limit’ docs (so not necessarily the x first ones defined by limit). If random.random() is greater than sample, doc is inspected, otherwise it’s skipped

• metadata – compute metadata on the result

• auto_convert – run converters automatically (converters are used to convert one mode’s output to another mode’s output, eg. type to jsonschema)

biothings.utils.inspect.merge_record(target, tomerge, mode)

biothings.utils.inspect.merge_scalar_list(mapt, mode)

biothings.utils.inspect.run_converters(_map, converters, logger=<module 'logging' from '/home/docs/.asdf/installs/python/3.10.4/lib/python3.10/logging/__init__.py'>)

biothings.utils.inspect.stringify_inspect_doc(dmap)

biothings.utils.inspect.typify_inspect_doc(dmap)

dmap is an inspect which was converted to be stored in a database, namely actual python types were stringify to be storabled. This function does the oposite and restore back python types within the inspect doc

biothings.utils.jsondiff

The MIT License (MIT)

Copyright (c) 2014 Ilya Volkov

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

biothings.utils.jsondiff.make(src, dst, **kwargs)

biothings.utils.jsonpatch

Apply JSON-Patches (RFC 6902)

class biothings.utils.jsonpatch.AddOperation(operation)

Bases: PatchOperation

Adds an object property or an array element.

apply(obj)

Abstract method that applies patch operation to specified object.

class biothings.utils.jsonpatch.CopyOperation(operation)

Bases: PatchOperation

Copies an object property or an array element to a new location

apply(obj)

Abstract method that applies patch operation to specified object.

exception biothings.utils.jsonpatch.InvalidJsonPatch

Bases: JsonPatchException

Raised if an invalid JSON Patch is created

class biothings.utils.jsonpatch.JsonPatch(patch)

Bases: object

A JSON Patch is a list of Patch Operations.

>>> patch = JsonPatch([
...     {'op': 'add', 'path': '/foo', 'value': 'bar'},
...     {'op': 'add', 'path': '/baz', 'value': [1, 2, 3]},
...     {'op': 'remove', 'path': '/baz/1'},
...     {'op': 'test', 'path': '/baz', 'value': [1, 3]},
...     {'op': 'replace', 'path': '/baz/0', 'value': 42},
...     {'op': 'remove', 'path': '/baz/1'},
... ])
>>> doc = {}
>>> result = patch.apply(doc)
>>> expected = {'foo': 'bar', 'baz': [42]}
>>> result == expected
True

JsonPatch object is iterable, so you could easily access to each patch statement in loop:

>>> lpatch = list(patch)
>>> expected = {'op': 'add', 'path': '/foo', 'value': 'bar'}
>>> lpatch[0] == expected
True
>>> lpatch == patch.patch
True

Also JsonPatch could be converted directly to bool if it contains any operation statements:

>>> bool(patch)
True
>>> bool(JsonPatch([]))
False

This behavior is very handy with make_patch() to write more readable code:

>>> old = {'foo': 'bar', 'numbers': [1, 3, 4, 8]}
>>> new = {'baz': 'qux', 'numbers': [1, 4, 7]}
>>> patch = make_patch(old, new)
>>> if patch:
...     # document have changed, do something useful
...     patch.apply(old)    
{...}

apply(orig_obj, in_place=False, ignore_conflicts=False, verify=False)

Applies the patch to given object.

Parameters

• obj (dict) – Document object.

• in_place (bool) – Tweaks way how patch would be applied - directly to specified obj or to his copy.

Returns

Modified obj.

classmethod from_diff(src, dst)

Creates JsonPatch instance based on comparing of two document objects. Json patch would be created for src argument against dst one.

Parameters

• src (dict) – Data source document object.

• dst (dict) – Data source document object.

Returns

JsonPatch instance.

>>> src = {'foo': 'bar', 'numbers': [1, 3, 4, 8]}
>>> dst = {'baz': 'qux', 'numbers': [1, 4, 7]}
>>> patch = JsonPatch.from_diff(src, dst)
>>> new = patch.apply(src)
>>> new == dst
True

classmethod from_string(patch_str)

Creates JsonPatch instance from string source.

Parameters

patch_str (str) – JSON patch as raw string.

Returns

JsonPatch instance.

to_string()

Returns patch set as JSON string.

exception biothings.utils.jsonpatch.JsonPatchConflict

Bases: JsonPatchException

Raised if patch could not be applied due to conflict situation such as: - attempt to add object key then it already exists; - attempt to operate with nonexistence object key; - attempt to insert value to array at position beyond of it size; - etc.

exception biothings.utils.jsonpatch.JsonPatchException

Bases: Exception

Base Json Patch exception

exception biothings.utils.jsonpatch.JsonPatchTestFailed

Bases: JsonPatchException, AssertionError

A Test operation failed

class biothings.utils.jsonpatch.MoveOperation(operation)

Bases: PatchOperation

Moves an object property or an array element to new location.

apply(obj)

Abstract method that applies patch operation to specified object.

class biothings.utils.jsonpatch.PatchOperation(operation)

Bases: object

A single operation inside a JSON Patch.

apply(obj)

Abstract method that applies patch operation to specified object.

class biothings.utils.jsonpatch.RemoveOperation(operation)

Bases: PatchOperation

Removes an object property or an array element.

apply(obj)

Abstract method that applies patch operation to specified object.

class biothings.utils.jsonpatch.ReplaceOperation(operation)

Bases: PatchOperation

Replaces an object property or an array element by new value.

apply(obj)

Abstract method that applies patch operation to specified object.

class biothings.utils.jsonpatch.TestOperation(operation)

Bases: PatchOperation

Test value by specified location.

apply(obj)

Abstract method that applies patch operation to specified object.

biothings.utils.jsonpatch.apply_patch(doc, patch, in_place=False, ignore_conflicts=False, verify=False)

Apply list of patches to specified json document.

Parameters

• doc (dict) – Document object.

• patch (list or str) – JSON patch as list of dicts or raw JSON-encoded string.

• in_place (bool) – While True patch will modify target document. By default patch will be applied to document copy.

• ignore_conflicts (bool) – Ignore JsonConflicts errors

• verify (bool) – works with ignore_conflicts = True, if errors and verify is True (recommanded), make sure the resulting objects is the same as the original one. ignore_conflicts and verify are used to run patches multiple times and get rif of errors when operations can’t be performed multiple times because the object has already been patched This will force in_place to False in order the comparison to occur.

Returns

Patched document object.

Return type

dict

>>> doc = {'foo': 'bar'}
>>> patch = [{'op': 'add', 'path': '/baz', 'value': 'qux'}]
>>> other = apply_patch(doc, patch)
>>> doc is not other
True
>>> other == {'foo': 'bar', 'baz': 'qux'}
True
>>> patch = [{'op': 'add', 'path': '/baz', 'value': 'qux'}]
>>> apply_patch(doc, patch, in_place=True) == {'foo': 'bar', 'baz': 'qux'}
True
>>> doc == other
True

biothings.utils.jsonpatch.get_loadjson()

adds the object_pairs_hook parameter to json.load when possible

The “object_pairs_hook” parameter is used to handle duplicate keys when loading a JSON object. This parameter does not exist in Python 2.6. This methods returns an unmodified json.load for Python 2.6 and a partial function with object_pairs_hook set to multidict for Python versions that support the parameter.

biothings.utils.jsonpatch.make_patch(src, dst)

Generates patch by comparing of two document objects. Actually is a proxy to JsonPatch.from_diff() method.

Parameters

• src (dict) – Data source document object.

• dst (dict) – Data source document object.

>>> src = {'foo': 'bar', 'numbers': [1, 3, 4, 8]}
>>> dst = {'baz': 'qux', 'numbers': [1, 4, 7]}
>>> patch = make_patch(src, dst)
>>> new = patch.apply(src)
>>> new == dst
True

biothings.utils.jsonpatch.multidict(ordered_pairs)

Convert duplicate keys values to lists.

biothings.utils.jsonpatch.reapply_patch(doc, patch)

Apply or (safely) re-apply patch to doc

biothings.utils.jsonschema

biothings.utils.jsonschema.generate_json_schema(dmap)

biothings.utils.jsonschema.test()

biothings.utils.loggers

class biothings.utils.loggers.Colors(value)

Bases: Enum

An enumeration.

CRITICAL = '#7b0099'

DEBUG = '#a1a1a1'

ERROR = 'danger'

INFO = 'good'

NOTSET = '#d6d2d2'

WARNING = 'warning'

class biothings.utils.loggers.EventRecorder(*args, **kwargs)

Bases: StreamHandler

Initialize the handler.

If stream is not specified, sys.stderr is used.

emit(record)

Emit a record.

If a formatter is specified, it is used to format the record. The record is then written to the stream with a trailing newline. If exception information is present, it is formatted using traceback.print_exception and appended to the stream. If the stream has an ‘encoding’ attribute, it is used to determine how to do the output to the stream.

class biothings.utils.loggers.GZipRotator

Bases: object

class biothings.utils.loggers.LookUpList(initlist)

Bases: UserList

find(val)

find_index(val)

class biothings.utils.loggers.Range(start: Union[int, float] = 0, end: Union[int, float] = inf)

Bases: object

end: Union[int, float] = inf

start: Union[int, float] = 0

class biothings.utils.loggers.ReceiverGroup(initlist=None)

Bases: UserList

class biothings.utils.loggers.Record(range, value)

Bases: NamedTuple

Create new instance of Record(range, value)

range: Range

Alias for field number 0

value: Enum

Alias for field number 1

class biothings.utils.loggers.ShellLogger(*args, **kwargs)

Bases: Logger

Custom “levels” for input going to the shell and output coming from it (just for naming)

Initialize the logger with a name and an optional level.

INPUT = 1001

OUTPUT = 1000

input(msg, *args, **kwargs)

output(msg, *args, **kwargs)

class biothings.utils.loggers.SlackHandler(webhook, mentions)

Bases: StreamHandler

Initialize the handler.

If stream is not specified, sys.stderr is used.

emit(record)

Emit a record.

If a formatter is specified, it is used to format the record. The record is then written to the stream with a trailing newline. If exception information is present, it is formatted using traceback.print_exception and appended to the stream. If the stream has an ‘encoding’ attribute, it is used to determine how to do the output to the stream.

static send(webhook, message, level, mentions=())

class biothings.utils.loggers.SlackMentionPolicy(policy)

Bases: object

mentions(level)

class biothings.utils.loggers.SlackMessage

Bases: object

build()

markdown(text, prefixes=(), suffixes=())

plaintext(text, color)

class biothings.utils.loggers.Squares(value)

Bases: Enum

An enumeration.

CRITICAL = ':large_purple_square:'

DEBUG = ':white_large_square:'

ERROR = ':large_red_square:'

INFO = ':large_blue_square:'

NOTSET = ''

WARNING = ':large_orange_square:'

class biothings.utils.loggers.WSLogHandler(listener)

Bases: StreamHandler

when listener is a bt.hub.api.handlers.ws.LogListener instance, log statements are propagated through existing websocket

Initialize the handler.

If stream is not specified, sys.stderr is used.

emit(record)

Emit a record.

If a formatter is specified, it is used to format the record. The record is then written to the stream with a trailing newline. If exception information is present, it is formatted using traceback.print_exception and appended to the stream. If the stream has an ‘encoding’ attribute, it is used to determine how to do the output to the stream.

payload(record)

class biothings.utils.loggers.WSShellHandler(listener)

Bases: WSLogHandler

when listener is a bt.hub.api.handlers.ws.LogListener instance, log statements are propagated through existing websocket

Initialize the handler.

If stream is not specified, sys.stderr is used.

payload(record)

biothings.utils.loggers.configurate_file_handler(logger, logfile, formater=None, force=False)

biothings.utils.loggers.create_logger(log_folder, logger_name, level=10)

biothings.utils.loggers.get_logger(logger_name, log_folder=None, handlers=('console', 'file', 'slack'), timestamp=None, force=False)

Configure a logger object from logger_name and return (logger, logfile)

biothings.utils.loggers.setup_default_log(default_logger_name, log_folder, level=10)

biothings.utils.manager

biothings.utils.mongo

biothings.utils.parallel

biothings.utils.parallel_mp

biothings.utils.parsers

biothings.utils.parsers.json_array_parser(patterns: Optional[Iterable[str]] = None) → Callable[[str], Generator[dict, None, None]]

Create JSON Array Parser given filename patterns

For use with manifest.json based plugins. The data comes in a JSON that is an JSON array, containing multiple documents.

Parameters

patterns – glob-compatible patterns for filenames, like .json, data.json

Returns

parser_func

biothings.utils.parsers.ndjson_parser(patterns: Optional[Iterable[str]] = None) → Callable[[str], Generator[dict, None, None]]

Create NDJSON Parser given filename patterns

For use with manifest.json based plugins. Caveat: Only handles valid NDJSON (no extra newlines, UTF8, etc.)

Parameters

patterns – glob-compatible patterns for filenames, like .ndjson, data.ndjson

Returns

Generator that takes in a data_folder and returns documents from

NDJSON files that matches the filename patterns

Return type

parser_func

biothings.utils.redis

class biothings.utils.redis.RedisClient(connection_params)

Bases: object

check()

client = None

classmethod get_client(params)

get_db(db_name=None)

Return a redict client instance from a database name or database number (if db_name is an integer)

initialize(deep=False)

Careful: this may delete data. Prepare Redis instance to work with biothings hub: - database 0: this db is used to store a mapping between

database index and database name (so a database can be accessed by name). This method will flush this db and prepare it.

• any other databases will be flushed if deep is True, making the redis server fully dedicated to

property mapdb

pick_db()

Return a database number, preferably not used (db doesn’t exist). If no database available (all are used), will be one and flush it…

exception biothings.utils.redis.RedisClientError

Bases: Exception

biothings.utils.serializer

class biothings.utils.serializer.URL(seq)

Bases: UserString

remove(param='format')

biothings.utils.serializer.orjson_default(o)

The default function passed to orjson to serialize non-serializable objects

biothings.utils.serializer.to_json(data, indent=False, sort_keys=False)

biothings.utils.serializer.to_json_0(data)

deprecated

biothings.utils.serializer.to_json_file(data, fobj, indent=False, sort_keys=False)

biothings.utils.serializer.to_msgpack(data)

biothings.utils.serializer.to_yaml(data, stream=None, Dumper=<class 'yaml.dumper.SafeDumper'>, default_flow_style=False)

biothings.utils.shelve

biothings.utils.sqlite3

class biothings.utils.sqlite3.Collection(colname, db)

Bases: object

count()

property database

drop()

find(*args, **kwargs)

find_one(*args, **kwargs)

get_conn()

insert_one(doc)

property name

remove(query)

replace_one(query, doc, upsert=False)

save(doc)

update(query, what)

update_one(query, what, upsert=False)

class biothings.utils.sqlite3.Database

Bases: IDatabase

property address

Returns sufficient information so a connection to a database can be created. Information can be a dictionary, object, etc… and depends on the actual backend

collection_names()

Return a list of all collections (or tables) found in this database

create_collection(colname)

Create a table/colleciton named colname. If backend is using a schema-based database (ie. SQL), backend should enforce the schema with at least field “_id” as the primary key (as a string).

create_if_needed(table)

get_conn()

biothings.utils.sqlite3.get_api()

biothings.utils.sqlite3.get_cmd()

biothings.utils.sqlite3.get_data_plugin()

biothings.utils.sqlite3.get_event()

biothings.utils.sqlite3.get_hub_config()

biothings.utils.sqlite3.get_hub_db_conn()

biothings.utils.sqlite3.get_last_command()

biothings.utils.sqlite3.get_source_fullname(col_name)

Assuming col_name is a collection created from an upload process, find the main source & sub_source associated.

biothings.utils.sqlite3.get_src_build()

biothings.utils.sqlite3.get_src_build_config()

biothings.utils.sqlite3.get_src_dump()

biothings.utils.sqlite3.get_src_master()

biothings.utils.version

Functions to return versions of things.

biothings.utils.version.check_new_version(folder, max_commits=10)

Given a folder pointing to a Git repo, return a dict containing info about remote commits not qpplied yet to the repo, or empty dict if nothing new.

biothings.utils.version.get_biothings_commit()

Gets the biothings commit information.

biothings.utils.version.get_python_exec_version()

return Python version

biothings.utils.version.get_python_version()

Get a list of python packages installed and their versions.

biothings.utils.version.get_repository_information(app_dir=None)

Get the repository information for the local repository, if it exists.

biothings.utils.version.get_software_info(app_dir=None)

return current application info

biothings.utils.version.get_source_code_info(src_file)

Given a path to a source code, try to find information about repository, revision, URL pointing to that file, etc… Return None if nothing can be determined. Tricky cases:

• src_file could refer to another repo, within current repo (namely a remote data plugin, cloned within the api’s plugins folder

• src_file could point to a folder, when for instance a dataplugin is analized. This is because we can’t point to an uploader file since it’s dynamically generated

biothings.utils.version.get_version(folder)

return revision of a git folder

biothings.utils.version.set_versions(config, app_folder)

Propagate versions (git branch name) in config module. Also set app and biothings folder paths (though not exposed as a config param since they are lower-cased, see biothings.__init__.py, regex PARAM_PAT)