weaver.lib package

Submodules

weaver.lib.engine module

class weaver.lib.engine.Engine

Bases: object

A generic database system. Specific database platforms will inherit from this class.

connect(force_reconnect=False)

connection

create_db()

Create a new database based on settings supplied in Database object engine.db.

create_db_statement()

Return SQL statement to create a database.

cursor

Get db cursor.

database_name(name=None)

Return name of the database.

datatypes = []

db = None

debug = False

disconnect()

drop_statement(objecttype, objectname)

Return drop table or database SQL statement.

execute(statement, commit=True)

Execute given statement.

executemany(statement, values, commit=True)

Execute given statement with multiple values.

exists(database, table_name)

Check to see if the given table exists.

final_cleanup()

Close the database connection.

get_connection()

This method should be overloaded by specific implementations of Engine.

get_cursor()

Get db cursor.

get_input()

Manually get user input for connection information when script is run from terminal.

gis_import(table)

instructions = 'Enter your database connection information:'

name = ''

pkformat = '%s PRIMARY KEY %s '

required_opts = []

script = None

set_engine_encoding()

set_table_delimiter(file_path)

table = None

table_exists(dbname, tablename)

This can be overridden to return True if a table exists. It returns False by default.

table_name(name=None, dbname=None)

Return full tablename.

to_csv(sort=True, path='/home/docs/checkouts/readthedocs.org/user_builds/weaver/checkouts/latest/docs', table_name=None)

use_cache = True

warning(warning)

warnings = []

weaver.lib.engine.file_exists(path)

Return true if a file exists and its size is greater than 0.

weaver.lib.engine.filename_from_url(url)

Extract and returns the filename from the url.

weaver.lib.engine.gen_from_source(source)

Return generator from a source tuple.

Source tuples are of the form (callable, args) where callable(*args) returns either a generator or another source tuple. This allows indefinite regeneration of data sources.

weaver.lib.engine.reporthook(count, block_size, total_size)

Generate the progress bar.

Uses file size to calculate the percentage of file size downloaded. If the total_size of the file being downloaded is not in the header, provide progress as size of bytes downloaded in either KB, MB and GB.

weaver.lib.engine.skip_rows(rows, source)

Skip over the header lines by reading them before processing.

weaver.lib.get_opts module

weaver.lib.models module

Data Retriever Data Model

This module contains basic class definitions for the Retriever platform.

weaver.lib.tools module

weaver.lib.tools.open_csvw(csv_file, encode=True)

Open a csv writer forcing the use of Linux line endings on Windows.

Also sets dialect to ‘excel’ and escape characters to ‘’

weaver.lib.tools.open_fr(file_name, encoding='ISO-8859-1', encode=True)

Open file for reading respecting Python version and OS differences.

Sets newline to Linux line endings on Windows and Python 3 When encode=False does not set encoding on nix and Python 3 to keep as bytes

weaver.lib.tools.open_fw(file_name, encoding='ISO-8859-1', encode=True)

Open file for writing respecting Python version and OS differences.

Sets newline to Linux line endings on Python 3 When encode=False does not set encoding on nix and Python 3 to keep as bytes

weaver.lib.tools.to_str(object, object_encoding=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>)

weaver.lib.warning module

class weaver.lib.warning.Warning(location, warning)

Bases: object

Module contents

weaver.lib.check_for_updates()

Check for updates to datasets.

This updates the HOME_DIR scripts directory with the latest script versions

weaver.lib.join_postgres(dataset, user='postgres', password='', host='localhost', port=5432, database='postgres', database_name=None, table_name=None, compile=False, debug=False, quiet=False, use_cache=True)

Install scripts in postgres.

weaver.lib.join_sqlite(dataset, file=None, table_name=None, compile=False, debug=False, quiet=False, use_cache=True)

Install scripts in sqlite.

weaver.lib.datasets(keywords=None, licenses=None)

Return list of all available datasets.

weaver.lib.dataset_names()

Return list of all available dataset names.

weaver.lib.reload_scripts()

Load scripts from scripts directory and return list of modules.

weaver.lib.reset_weaver(scope='scripts', ask_permission=True)

Remove stored information on scripts, data, and connections.