Source code docs¶
This is all auto-generated documentation from the IIQTools source code. It defines the different modules, their APIs, and some examples.
iiqtools.exceptions¶
Centralized location for all custom Exceptions
-
exception
iiqtools.exceptions.
CliError
(command, stdout, stderr, exit_code, message='Command Failure')[source]¶ Raised when an CLI command has a non-zero exit code.
Parameters: - command (String) – The CLI command that was ran
- stdout (String) – The output from the standard out stream
- stderr (String) – The output from the standard error stream
- exit_code – The exit/return code from the command
-
exception
iiqtools.exceptions.
DatabaseError
(message, pgcode)[source]¶ Raised when an error occurs when interacting with the InsightIQ database
Attribute pgcode: The error code used by PostgreSQL. https://www.postgresql.org/docs/9.3/static/errcodes-appendix.html Attribute message: The error message
iiqtools.utils¶
The utils module for the IIQTools package. These are common-need/generic functions for making scripts/tools for use with InsightIQ.
cli_parsers¶
This module contains functions that parse stdout of a CLI command into a usable Python data structure.
-
iiqtools.utils.cli_parsers.
df_to_dict
(output)[source]¶ Parse the output from the command df into a dictionary
Returns: Dictionary Parameters: output (String) – Required The pile of stuff outputted by running df
database¶
Utilities for interacting with the InsightIQ database
-
class
iiqtools.utils.database.
Column
[source]¶ A database column consiting of the column’s name, and it’s type
Type: namedtuple
Parameters: - name – The column’s name
- type – The database type for the given column (i.e. int, float, double)
-
class
iiqtools.utils.database.
Database
(user='postgres', dbname='insightiq')[source]¶ Simplifies communication with the database.
The goal of this object is to make basic interactions with the database simpler than directly using the psycopg2 library. It does this by reducing the number of API methods, providing handy built-in methods for common needs (like listing tables of a database), auto-commit of transactions, and auto-rollback of bad SQL transactions. This object is not indented for power users, or long lived processes (like the InsightIQ application); it’s designed for shorter lived “scripts”.
Parameters: - user (String, default postgres) – The username when connection to the databse
- dbname (String, default insightiq) – The specific database to connection to. InsightIQ utilizes a different database for every monitored cluster, plus one generic database for the application (named “insightiq”).
-
execute
(sql, params=None)[source]¶ Run a single SQL command
Returns: Generator
Parameters: - sql (String) – Required The SQL syntax to execute
- params (Iterable) – The values to use in a parameterized SQL query
This method is implemented as a Python Generator: https://wiki.python.org/moin/Generators This means you are suppose to iterate over the results:
db = Database() for row in db.execute("select * from some_table;"): print row
If you want all the rows as a single thing, just use
list
:db = Database() data = list(db.execute("select * from some_table;")
But WARNING that might cause your program to run out of memory and crash! That reason is why this method is a generator by default ;)
To perform a parameterized query (i.e. avoid SQL injection), provided the parameters as an iterable:
db = Database() # passing in "foo_column" alone would try and string format every # character of "foo_column" into your SQL statement. # Instead, make "foo_column" a tuple by wrapping it like ("foo_column",) # Note: the trailing comma is required. data = list(db.execute("select %s from some_table", ("foo_column",)))
-
executemany
(sql, params)[source]¶ Run the SQL for every iteration of the supplied params
This method behaves exactly like execute, except that it can perform multiple SQL commands in a single transaction. The point of this method is so you can retain Atomicity when you must execute the same SQL with different parameters. This method isn’t intended to be faster than looping over the normal execute method with the different parameters.
Returns: Generator
Parameters: - sql (String) – Required The SQL syntax to execute
- params (Iterable) – Required The parameterized values to iterate
-
isolation_level
¶ Set the isolation level of your connnection to the database
-
primary_key
(table)[source]¶ Given a table, return the primary key
Note
If you supply a timeseries table that DOES NOT have an EPOC timestamp in the name, you will get zero results. For timeseries tables, supply a table that contains the EPOC timestamps to see the primary key.
Returns: Tuple of namedtuples -> (Column(name, type), Column(name, type)) Parameters: table (String) – Required The table to obtain the primary key from
generic¶
This module contains miscellaneous utility functions
-
iiqtools.utils.generic.
check_path
(cli_value)[source]¶ Validate that the supplied path is an actual file system directory.
This function is intended to be used with the argparse lib as an argument type.
Raises: argparse.ArgumentTypeError Returns: String Parameters: cli_value (String) – The value supplied by the end user
logger¶
-
iiqtools.utils.logger.
get_logger
(log_path=None, stream_lvl=0, file_lvl=20)[source]¶ Factory for making logging objects
The verbosity of the logs are configurable as defined by the official Python documentation: https://docs.python.org/2/library/logging.html#logging-levels
Returns: logging.Logger
Raises: AssertionError on bad parameter input
Parameters: - log_path (String) – Required The absolute file path to write logs to
- stream_lvl (Integer, default 20) – Set to print log messages to the terminal.
- file_lvl – How verbose the log file messages are. This value cannot be zero.
shell¶
This module reduces boilerplate when interacting with the command shell, i.e. BASH.
Example A:
>>> result = shell.run_cmd('ls')
>>> print result.stdout
README.txt
someOtherFile.txt
>>> print result.stderr
>>> result.exit_code
0
Example B:
>>> # run_cmd does not support the Unix Pipeline
>>> result = shell.run_cmd('cat foo.txt | grep "not supported"')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
raise CliError(cli_syntax, stdout, stderr, exit_code)
iiqtools.exceptions.CliError: Command Failure: cat foo.txt | grep "not supported"
-
class
iiqtools.utils.shell.
CliResult
[source]¶ The outcome from running a CLI command
Type: collections.namedtuple
Parameters: - command (String) – The CLI command that was ran
- stdout (String) – The output from the standard out stream
- stderr (String) – The output from the standard error stream
- exit_code – The exit/return code from the command
-
iiqtools.utils.shell.
run_cmd
(cli_syntax)[source]¶ Execute a simple CLI command.
This function blocks until the CLI command returns and does not support the Unix pipeline.
Returns: CliResult - namedtuple Raises: CliError (when exit code is not zero) Parameters: cli_syntax (String) – The CLI command to run.
versions¶
This module make obtaining and comparing version strings easy!
-
class
iiqtools.utils.versions.
PatchInfo
[source]¶ Describes the state of patches for InsightIQ
Parameters: - iiq_dir (String) – The file system path where InsightIQ source is located.
- patches_dir (String) – The file system path where patches for InsightIQ are stored.
- specific_patch (String) – Only populated when a patch is being installed/removed/read.
- is_installed – If
specific_patch
is installed or not. - readme (String) – The README.txt for
specific_patch
if applicable. - all_patches (Tuple) – All currently installed patches.
Type is_installed: Boolean
-
class
iiqtools.utils.versions.
Version
(version, name)[source]¶ Implements comparison operators for common version strings
Only versions strings upwards of 4 digits and consisting only of numbers is supported. Version strings breakdown into
major
,minor
,patch
, andbuild
.Example:
>>> version_as_string = '1.2.3' >>> version = Version(name='myPackage', version=version_as_string) >>> version.major 1 >>> version.patch 3 >>> type(version.minor) <type 'int'> >>> type(version.build) <type 'NoneType'>
Comparing Versions:
>>> v1 = Version(name='myPackage', version='1.2.3') >>> v2 = Version(name='newPackage', version='1.4.0') >>> v1 < v2 True >>> v2 >= v1 True >>> v2 == v1 False
The Version object also support comparison of string versions:
>>> v1 = Version(name='foo', version='4.5') >>> v1 > '5.0' False
It’s worth noting, more specific versions (that would otherwise be equal), are considered greater:
>>> v1 = Version(name='bar', version='1.2.0') >>> v1 > '1.2' True
This is because a version without a value is None, and zero is greater than None in Python:
>>> 0 > None True
-
iiqtools.utils.versions.
get_iiq_version
()[source]¶ Obtain the version of InsightIQ installed
Returns: iiqtools.utils.versions.Version
-
iiqtools.utils.versions.
get_iiqtools_version
()[source]¶ Obtain the version of iiqtools installed
Returns: iiqtools.utils.versions.Version
-
iiqtools.utils.versions.
get_patch_info
(specific_patch, log)[source]¶ Obtain the current state of patches for InsightIQ
Returns: PatchInfo (namedtuple)
Parameters: - specific_patch (String) – Required The name of a patch that’s being installed/removed/read.
- log (logging.Logger) – Required The logging object. This param is really here to make unit testing easier -> https://en.wikipedia.org/wiki/Dependency_injection
insightiq_api¶
This module is for performing privileged API calls to InsightIQ.
-
exception
iiqtools.utils.insightiq_api.
ConnectionError
[source]¶ Unable to establish an connection to the OneFS API
-
class
iiqtools.utils.insightiq_api.
InsightiqApi
(username, password, verify=False)[source]¶ An authenticated connection to the InsightIQ API
This object is a simple wrapper around a requests Session. The point of wrapping the requests Session object is to remove boiler plate code in making API calls to InsightIQ, and to auto-handle authenticating to the API. The most noteworthy changes to this object and how you use the requests Session object is that you must provide the username and password when instantiating the object, and when you make a request, you only supply the URI end point (i.e. not the http://my-host.org:8080 part).
Supports use of
with
statements, which will automatically handle creating and closing the HTTP session with InsightIQ.Example:
>>> with InsightiApi(username='administrator', password='foo') as iiq: response = iiq.get('/api/clusters')
Parameters: - username (String) – Required The name of the administrative account for InsightIQ
- password (String) – Required The password for the administrative account being used.
- verify (Boolean) – Perform SSL/TLS cert validation using system certs. Setting to True will likely cause issues when using a self-signed SSL/TLS cert. Default is False.
-
delete
(endpoint, params=None, data=None, headers=None, **kwargs)[source]¶ Perform an HTTP DELETE request
Returns: PyObject
Parameters: - endpoint (String) – Required The URI end point of the InsightIQ API to call
- params (Dictionary) – The HTTP parameters to send in the HTTP request
- data (PyObject) – The HTTP body content to send in the request. The Python object supplied (i.e. list, dict, etc) will be auto-converted to JSON string.
- headers (Dictionary) – Any additional HTTP headers to send in the request
-
get
(endpoint, params=None, data=None, headers=None, **kwargs)[source]¶ Perform an HTTP GET request
Returns: PyObject
Parameters: - endpoint (String) – Required The URI end point of the InsightIQ API to call
- params (Dictionary) – The HTTP parameters to send in the HTTP request
- data (PyObject) – The HTTP body content to send in the request. The Python object supplied (i.e. list, dict, etc) will be auto-converted to JSON string.
- headers (Dictionary) – Any additional HTTP headers to send in the request
-
head
(endpoint, params=None, data=None, headers=None, **kwargs)[source]¶ Perform an HTTP HEAD request
Returns: PyObject
Parameters: - endpoint (String) – Required The URI end point of the InsightIQ API to call
- params (Dictionary) – The HTTP parameters to send in the HTTP request
- data (PyObject) – The HTTP body content to send in the request. The Python object supplied (i.e. list, dict, etc) will be auto-converted to JSON string.
- headers (Dictionary) – Any additional HTTP headers to send in the request
-
post
(endpoint, params=None, data=None, headers=None, **kwargs)[source]¶ Perform an HTTP POST request
Returns: PyObject
Parameters: - endpoint (String) – Required The URI end point of the InsightIQ API to call
- params (Dictionary) – The HTTP parameters to send in the HTTP request
- data (PyObject) – The HTTP body content to send in the request. The Python object supplied (i.e. list, dict, etc) will be auto-converted to JSON string.
- headers (Dictionary) – Any additional HTTP headers to send in the request
-
put
(endpoint, params=None, data=None, headers=None, **kwargs)[source]¶ Perform an HTTP PUT request
Returns: PyObject
Parameters: - endpoint (String) – Required The URI end point of the InsightIQ API to call
- params (Dictionary) – The HTTP parameters to send in the HTTP request
- data (PyObject) – The HTTP body content to send in the request. The Python object supplied (i.e. list, dict, etc) will be auto-converted to JSON string.
- headers (Dictionary) – Any additional HTTP headers to send in the request
-
class
iiqtools.utils.insightiq_api.
Parameters
(*args, **kwargs)[source]¶ Object for working with HTTP query parameters
This object supports the Python dictionary API, and lets you define the same HTTP query parameter more than once. Additional definitions for the same query parameter creates a new entry in the underlying list. This decision makes it simple to iterate Parameters to build up the HTTP query string because you do not have to iterate parameter values. Because you can define the same parameter more than once, using the standard dictionary API will only impact the first occurrence of that parameter. To modify a specific parameter, you must use the methods in this class which extend the dictionary API.
This documentation is specific to how Parameters extends the normal Python dictionary API. For documentation about the Python dictionary API, please checkout their official page here.
Example creating duplicate parameters:
>>> params = Parameters() >>> for value in range(3): ... params.add('myParam', value) ... Parameters([['myParam', 0], ['myParam', 1], ['myParam', 2]])
What NOT to do:
>>> params = Parameters() >>> for doh in range(3): ... params['homer'] = doh ... >>> params Parameters([['homer', 2], ['homer', 2], ['homer', 2]])
Iterating Parameters to build a query string:
>>> query = [] >>> params = Parameters(one=1, two=2) >>> for name, value in params.items(): ... query.append('%s=%s' % (name, value)) ... >>> query_str = '&'.join(query) >>> query_str 'one=1&two=2'
Parameters: - args (List, Tuple, or Dictionary) – Data to initialize the Parameters object with.
- kwargs (Dictionary) – Data to initialize the Parameters object with.
-
NAME
¶ The array index for a parameter name; avoids magic numbers
-
VALUE
¶ The array index for a parameter value; avoids magic numbers
-
add
(name, value)[source]¶ Add a duplicate parameter
Returns: None
Parameters: - name (String) – Required The name of the parameter
- value (PyObject) – Required The value for the duplicate parameter
-
delete_parameter
(name, occurrence)[source]¶ Delete a specific parameter that is defined more than once. Thread safe.
Returns: None
Parameters: - name (String) – Required The parameter to delete.
- occurrence (Integer) – Required The N-th instance of a parameter. Zero based numbering.
-
get_all
(name)[source]¶ Return the key/value pairs for a parameter. Order is maintained.
Returns: List Parameters: name (String) – Required The name of the query parameter
-
modify_parameter
(name, new_value, occurrence)[source]¶ Change the value of a specific parameter that is defined more than once. Thread safe.
Returns: None
Parameters: - name (String) – Required The parameter to delete.
- new_value (PyObject) – Required The value for the parameter
- occurrence (Integer) – Required The N-th instance of a parameter. Zero based numbering.
iiqtools.iiqtools_gather_info¶
This module contains all the business logic for collecting logs and configuration information about InsightIQ for remote troubleshooting.
-
iiqtools.iiqtools_gather_info.
add_from_memory
(the_tarfile, data_name, data)[source]¶ Simplify adding in-memory information to the tar file
Returns: None
Parameters: - the_tarfile (tarfile.open) – The open tarfile object
- data_name (String) – The reference to the data; i.e. it’s name when you uncompress the file
- data (String) – The contents of the in-memory information
-
iiqtools.iiqtools_gather_info.
call_iiq_api
(uri)[source]¶ Make an internal API call to the InsightIQ API
Returns: JSON String Parameters: uri (string) – The API end point to call
-
iiqtools.iiqtools_gather_info.
cli_cmd_info
(command, parser)[source]¶ Standardizes the JSON format for any data collected via a CLI command
Returns: JSON String Parameters: command (String) – The CLI command to execute
-
iiqtools.iiqtools_gather_info.
clusters_info
()[source]¶ Obtain a pile of data about all monitored clusters in InsightIQ
Returns: JSON String
-
iiqtools.iiqtools_gather_info.
datastore_info
()[source]¶ Obtain data about the datastore for InsightIQ
Returns: JSON String
-
iiqtools.iiqtools_gather_info.
get_tarfile
(output_dir, case_number, the_time=None)[source]¶ Centralizes logic for making tgz file for InsightIQ logs
Returns: tarfile.TarFile
Parameters: - output_dir (String) – Required The directory to save the tar file in
- case_number (String or Integer) – Required The SR that the logs are for; used in file name.
- the_time (EPOC time stamp) – An optional EPOC timestamp to use when naming the file. If not supplied, this function calls time.time().
-
iiqtools.iiqtools_gather_info.
ifconfig_info
()[source]¶ Obtain data about the network interfaces on the host OS
Returns: JSON String
-
iiqtools.iiqtools_gather_info.
iiq_version_info
()[source]¶ Obtain info about the version of InsightIQ installed on the host OS
Returns: JSON String
-
iiqtools.iiqtools_gather_info.
ldap_info
()[source]¶ Obtain the config for LDAP in InsightIQ
Returns: JSON String
-
iiqtools.iiqtools_gather_info.
memory_info
()[source]¶ Obtain data about RAM on the host OS
Returns: JSON String
-
iiqtools.iiqtools_gather_info.
mount_info
()[source]¶ Obtain data about mounted file systems on the host OS
Returns: JSON String
iiqtools.iiqtools_patch¶
This module contains all the business logic for collecting logs and configuration information about InsightIQ for remote troubleshooting.
-
iiqtools.iiqtools_gather_info.
add_from_memory
(the_tarfile, data_name, data)[source] Simplify adding in-memory information to the tar file
Returns: None
Parameters: - the_tarfile (tarfile.open) – The open tarfile object
- data_name (String) – The reference to the data; i.e. it’s name when you uncompress the file
- data (String) – The contents of the in-memory information
-
iiqtools.iiqtools_gather_info.
call_iiq_api
(uri)[source] Make an internal API call to the InsightIQ API
Returns: JSON String Parameters: uri (string) – The API end point to call
-
iiqtools.iiqtools_gather_info.
cli_cmd_info
(command, parser)[source] Standardizes the JSON format for any data collected via a CLI command
Returns: JSON String Parameters: command (String) – The CLI command to execute
-
iiqtools.iiqtools_gather_info.
clusters_info
()[source] Obtain a pile of data about all monitored clusters in InsightIQ
Returns: JSON String
-
iiqtools.iiqtools_gather_info.
datastore_info
()[source] Obtain data about the datastore for InsightIQ
Returns: JSON String
-
iiqtools.iiqtools_gather_info.
get_tarfile
(output_dir, case_number, the_time=None)[source] Centralizes logic for making tgz file for InsightIQ logs
Returns: tarfile.TarFile
Parameters: - output_dir (String) – Required The directory to save the tar file in
- case_number (String or Integer) – Required The SR that the logs are for; used in file name.
- the_time (EPOC time stamp) – An optional EPOC timestamp to use when naming the file. If not supplied, this function calls time.time().
-
iiqtools.iiqtools_gather_info.
ifconfig_info
()[source] Obtain data about the network interfaces on the host OS
Returns: JSON String
-
iiqtools.iiqtools_gather_info.
iiq_version_info
()[source] Obtain info about the version of InsightIQ installed on the host OS
Returns: JSON String
-
iiqtools.iiqtools_gather_info.
ldap_info
()[source] Obtain the config for LDAP in InsightIQ
Returns: JSON String
-
iiqtools.iiqtools_gather_info.
main
(the_cli_args)[source] Entry point for running script
-
iiqtools.iiqtools_gather_info.
memory_info
()[source] Obtain data about RAM on the host OS
Returns: JSON String
-
iiqtools.iiqtools_gather_info.
mount_info
()[source] Obtain data about mounted file systems on the host OS
Returns: JSON String
-
iiqtools.iiqtools_gather_info.
parse_cli
(cli_args)[source] Handles parsing the CLI, and gives us –help for (basically) free
Returns: argparse.Namespace Parameters: cli_args (List) – The arguments passed to the script
-
iiqtools.iiqtools_gather_info.
reports_info
()[source] Obtain info about any scheduled reports in InsightIQ
Returns: JSON String
iiqtools.iiqtools_tar_to_zip¶
This script converts the format of datastore exports (not the CSV exports) from tar to zip. In InsightIQ 4.1, the format was changed to fix bug 162840, in which an attempt to import a large datastore export would time out. The only change to the exported data is the format. So to use an export from an older instance (before 4.1) all you have to convert the format. In other words, the data is still the same, it’s just a different compression format in InsightIQ 4.1.
-
class
iiqtools.iiqtools_tar_to_zip.
BufferedZipFile
(file, mode='r', compression=0, allowZip64=False)[source]¶ A subclass of zipfile.ZipFile that can read from a file-like object and stream the contents into a new zip file.
-
writebuffered
(filename, file_handle, file_size)[source]¶ Stream write data to the zip archive
Parameters: - filename (String) – Required The name to give the data once added to the zip file
- file_handle (Anything that supports the read method) – Required The file-like object to read
- file_size (Integer) – Required The size of the file in bytes
-
-
iiqtools.iiqtools_tar_to_zip.
check_tar
(value)[source]¶ Validate that the supplied tar file is an InsightIQ datastore export file.
Raises: argparse.ArgumentTypeError Returns: String Parameters: value (String) – Required The CLI value to validate
-
iiqtools.iiqtools_tar_to_zip.
get_timestamp_from_export
(source_tar)[source]¶ Allows us to create the new zip archive with the correct timestamp
Returns: String Parameters: source_tar (String) – Required The tar that’s being converted to a zip
-
iiqtools.iiqtools_tar_to_zip.
joinname
(export_dir, file_name)[source]¶ The tar/zip used by InsightIQ expects the data nested in a directory. This function handles absolute and relative paths for file_name.
Returns: String
Parameters: - export_dir (String) – Required The directory name to nest the file under
- file_name (String) – Required The name of the filed nested in the directory
iiqtools.iiqtools_version¶
This module contains the business logic for print the versions of InsightIQ and IIQTool that’s installed.