Utils¶
Exploration Functions¶
Module containing factory functions for parameter exploration
-
pypet.utils.explore.
cartesian_product
(parameter_dict, combined_parameters=())[source]¶ Generates a Cartesian product of the input parameter dictionary.
For example:
>>> print cartesian_product({'param1':[1,2,3], 'param2':[42.0, 52.5]}) {'param1':[1,1,2,2,3,3],'param2': [42.0,52.5,42.0,52.5,42.0,52.5]}
- Parameters
parameter_dict – Dictionary containing parameter names as keys and iterables of data to explore.
combined_parameters –
Tuple of tuples. Defines the order of the parameters and parameters that are linked together. If an inner tuple contains only a single item, you can spare the inner tuple brackets.
For example:
>>> print cartesian_product( {'param1': [42.0, 52.5], 'param2':['a', 'b'], 'param3' : [1,2,3]}, ('param3',('param1', 'param2'))) {param3':[1,1,2,2,3,3],'param1' : [42.0,52.5,42.0,52.5,42.0,52.5], 'param2':['a','b','a','b','a','b']}
- Returns
Dictionary with cartesian product lists.
-
pypet.utils.explore.
find_unique_points
(explored_parameters)[source]¶ Takes a list of explored parameters and finds unique parameter combinations.
If parameter ranges are hashable operates in O(N), otherwise O(N**2).
- Parameters
explored_parameters – List of explored parameters
- Returns
List of tuples, first entry being the parameter values, second entry a list containing the run position of the unique combination.
Utility Functions¶
HDF5 File Compression¶
You can use the following function to compress an existing HDF5 file that already contains a trajectory. This only works under Linux.
-
pypet.
compact_hdf5_file
(filename, name=None, index=None, keep_backup=True)[source]¶ Can compress an HDF5 to reduce file size.
The properties on how to compress the new file are taken from a given trajectory in the file. Simply calls
ptrepack
from the command line. (Se also https://pytables.github.io/usersguide/utilities.html#ptrepackdescr)Currently only supported under Linux, no guarantee for Windows usage.
- Parameters
filename – Name of the file to compact
name – The name of the trajectory from which the compression properties are taken
index – Instead of a name you could also specify an index, i.e -1 for the last trajectory in the file.
keep_backup – If a back up version of the original file should be kept. The backup file is named as the original but _backup is appended to the end.
- Returns
The return/error code of ptrepack
Progressbar¶
Simple progressbar that can be used during a for-loop (no initialisation necessary). It displays progress and estimates remaining time.
-
pypet.
progressbar
(index, total, percentage_step=10, logger='print', log_level=20, reprint=True, time=True, length=20, fmt_string=None, reset=False)[source]¶ Plots a progress bar to the given logger for large for loops.
To be used inside a for-loop at the end of the loop:
for irun in range(42): my_costly_job() # Your expensive function progressbar(index=irun, total=42, reprint=True) # shows a growing progressbar
There is no initialisation of the progressbar necessary before the for-loop. The progressbar will be reset automatically if used in another for-loop.
- Parameters
index – Current index of for-loop
total – Total size of for-loop
percentage_step – Steps with which the bar should be plotted
logger – Logger to write to - with level INFO. If string ‘print’ is given, the print statement is used. Use
None
if you don’t want to print or log the progressbar statement.log_level – Log level with which to log.
reprint – If no new line should be plotted but carriage return (works only for printing)
time – If the remaining time should be estimated and displayed
length – Length of the bar in = signs.
fmt_string – A string which contains exactly one %s in order to incorporate the progressbar. If such a string is given,
fmt_string % progressbar
is printed/logged.reset – If the progressbar should be restarted. If progressbar is called with a lower index than the one before, the progressbar is automatically restarted.
- Returns
The progressbar string or None if the string has not been updated.
Multiprocessing Directory Creation¶
Function that calls os.makedirs
but takes care about race conditions if multiple
processes or threads try to create the directories at the same time.
Merging many Trajectories¶
You can easily merge several trajectories located in one directory into one with
-
pypet.
merge_all_in_folder
(folder, ext='.hdf5', dynamic_imports=None, storage_service=None, force=False, ignore_data=(), move_data=False, delete_other_files=False, keep_info=True, keep_other_trajectory_info=True, merge_config=True, backup=True)[source]¶ Merges all files in a given folder.
IMPORTANT: Does not check if there are more than 1 trajectory in a file. Always uses the last trajectory in file and ignores the other ones.
Trajectories are merged according to the alphabetical order of the files, i.e. the resulting merged trajectory is found in the first file (according to lexicographic ordering).
- Parameters
folder – folder (not recursive) where to look for files
ext – only files with the given extension are used
dynamic_imports – Dynamic imports for loading
storage_service – storage service to use, leave None to use the default one
force – If loading should be forced.
delete_other_files – Deletes files of merged trajectories
All other parameters as in f_merge_many of the trajectory.
- Returns
The merged traj
Manual Runs¶
If you don’t want to use an Environment but manually schedule runs, take a look at the following decorator:
-
pypet.
manual_run
(turn_into_run=True, store_meta_data=True, clean_up=True)[source]¶ Can be used to decorate a function as a manual run function.
This can be helpful if you want the run functionality without using an environment.
- Parameters
turn_into_run – If the trajectory should become a single run with more specialized functionality during a single run.
store_meta_data – If meta-data like runtime should be automatically stored
clean_up – If all data added during the single run should be removed, only works if
turn_into_run=True
.
General Equality Function and Comparisons of Parameters and Results¶
Module containing utility functions to compare parameters and results
-
pypet.utils.comparisons.
get_all_attributes
(instance)[source]¶ Returns an attribute value dictionary much like __dict__ but incorporates __slots__
-
pypet.utils.comparisons.
nested_equal
(a, b)[source]¶ Compares two objects recursively by their elements.
Also handles numpy arrays, pandas data and sparse matrices.
First checks if the data falls into the above categories. If not, it is checked if a or b are some type of sequence or mapping and the contained elements are compared. If this is not the case, it is checked if a or b do provide a custom __eq__ that evaluates to a single boolean value. If this is not the case, the attributes of a and b are compared. If this does not help either, normal == is used.
Assumes hashable items are not mutable in a way that affects equality. Based on the suggestion from HERE, thanks again Lauritz V. Thaulow :-)