biothings.hub.dataplugin

biothings.hub.dataplugin.assistant

class biothings.hub.dataplugin.assistant.AdvancedPluginLoader(plugin_name)

Bases: BasePluginLoader

can_load_plugin()

Return True if loader is able to load plugin (check data folder content)

load_plugin()

Load plugin and register its components

loader_type = 'advanced'

exception biothings.hub.dataplugin.assistant.AssistantException

Bases: Exception

class biothings.hub.dataplugin.assistant.AssistantManager(data_plugin_manager, dumper_manager, uploader_manager, keylookup=None, default_export_folder='hub/dataload/sources', *args, **kwargs)

Bases: BaseSourceManager

configure(klasses=[<class 'biothings.hub.dataplugin.assistant.GithubAssistant'>, <class 'biothings.hub.dataplugin.assistant.LocalAssistant'>])

create_instance(klass, url)

export(plugin_name, folder=None, what=['dumper', 'uploader', 'mapping'], purge=False)

Export generated code for a given plugin name, in given folder (or use DEFAULT_EXPORT_FOLDER if None). Exported information can be: - dumper: dumper class generated from the manifest - uploader: uploader class generated from the manifest - mapping: mapping generated from inspection or from the manifest If “purge” is true, any existing folder/code will be deleted first, otherwise, will raise an error if some folder/files already exist.

export_dumper(plugin_name, folder)

export_mapping(plugin_name, folder)

export_uploader(plugin_name, folder)

load(autodiscover=True)

Load plugins registered in internal Hub database and generate/register dumpers & uploaders accordingly. If autodiscover is True, also search DATA_PLUGIN_FOLDER for existing plugin directories not registered yet in the database, and register them automatically.

load_plugin(plugin)

register_classes(klasses)

Register each class in self.register dict. Key will be used to retrieve the source class, create an instance and run method from it. It must be implemented in subclass as each manager may need to access its sources differently,based on different keys.

register_url(url)

setup_log()

Setup and return a logger instance

submit(url)

unregister_url(url=None, name=None)

update_plugin_name(plugin, new_name)

class biothings.hub.dataplugin.assistant.AssistedDumper

Bases: object

DATA_PLUGIN_FOLDER = None

class biothings.hub.dataplugin.assistant.AssistedUploader

Bases: object

DATA_PLUGIN_FOLDER = None

class biothings.hub.dataplugin.assistant.BaseAssistant(url)

Bases: object

can_handle()

Return true if assistant can handle the code

data_plugin_manager = None

dumper_manager = None

handle()

Access self.url and do whatever is necessary to bring code to life within the hub… (hint: that may involve creating a dumper on-the-fly and register that dumper to a manager…)

keylookup = None

load_plugin()

Load plugin and register its components

property loader

Return loader object able to interpret plugin’s folder content

loaders = {'advanced': <class 'biothings.hub.dataplugin.assistant.AdvancedPluginLoader'>, 'manifest': <class 'biothings.hub.dataplugin.assistant.ManifestBasedPluginLoader'>}

property plugin_name

Return plugin name, parsed from self.url and set self._src_folder as path to folder containing dataplugin source code

plugin_type = None

register_loader()

setup_log()

Setup and return a logger instance

uploader_manager = None

class biothings.hub.dataplugin.assistant.BasePluginLoader(plugin_name)

Bases: object

can_load_plugin()

Return True if loader is able to load plugin (check data folder content)

get_plugin_obj()

invalidate_plugin(error)

load_plugin()

Load plugin and register its components

loader_type = None

setup_log()

Setup and return a logger instance

class biothings.hub.dataplugin.assistant.GithubAssistant(url)

Bases: BaseAssistant

can_handle()

Return true if assistant can handle the code

get_classdef()

handle()

Access self.url and do whatever is necessary to bring code to life within the hub… (hint: that may involve creating a dumper on-the-fly and register that dumper to a manager…)

property plugin_name

Return plugin name, parsed from self.url and set self._src_folder as path to folder containing dataplugin source code

plugin_type = 'github'

exception biothings.hub.dataplugin.assistant.LoaderException

Bases: Exception

class biothings.hub.dataplugin.assistant.LocalAssistant(url)

Bases: BaseAssistant

can_handle()

Return true if assistant can handle the code

get_classdef()

handle()

Access self.url and do whatever is necessary to bring code to life within the hub… (hint: that may involve creating a dumper on-the-fly and register that dumper to a manager…)

property plugin_name

Return plugin name, parsed from self.url and set self._src_folder as path to folder containing dataplugin source code

plugin_type = 'local'

class biothings.hub.dataplugin.assistant.ManifestBasedPluginLoader(plugin_name)

Bases: BasePluginLoader

can_load_plugin()

Return True if loader is able to load plugin (check data folder content)

dumper_registry = {'docker': <class 'biothings.hub.dataload.dumper.DockerContainerDumper'>, 'ftp': <class 'biothings.hub.dataload.dumper.LastModifiedFTPDumper'>, 'http': <class 'biothings.hub.dataload.dumper.LastModifiedHTTPDumper'>, 'https': <class 'biothings.hub.dataload.dumper.LastModifiedHTTPDumper'>}

get_code_for_mod_name(mod_name)

Returns string literal and name of function, given a path

Parameters:

mod_name – string with module name and function name, separated by colon

Returns:

containing

• indented string literal for the function specified

• name of the function

Return type:

Tuple[str, str]

get_dumper_dynamic_class(dumper_section, metadata)

get_uploader_dynamic_class(uploader_section, metadata, sub_source_name='')

get_uploader_dynamic_classes(uploader_section, metadata, data_plugin_folder)

interpret_manifest(manifest, data_plugin_folder)

load_plugin()

Load plugin and register its components

loader_type = 'manifest'

biothings.hub.dataplugin.manager

class biothings.hub.dataplugin.manager.DataPluginManager(job_manager, datasource_path='dataload.sources', *args, **kwargs)

Bases: DumperManager

load(plugin_name, *args, **kwargs)

class biothings.hub.dataplugin.manager.GitDataPlugin(src_name=None, src_root_folder=None, log_folder=None, archive=None)

Bases: GitDumper

prepare_src_dump()

class biothings.hub.dataplugin.manager.ManualDataPlugin(*args, **kwargs)

Bases: ManualDumper

async dump(*args, **kwargs)

Dump (ie. download) resource as needed this should be called after instance creation ‘force’ argument will force dump, passing this to create_todump_list() method.

prepare_src_dump()