imagine.observables package¶
Submodules¶
imagine.observables.dataset module¶
Datasets are auxiliary classes used to facilitate the reading and inclusion of observational data in the IMAGINE pipeline
-
class
imagine.observables.dataset.
Dataset
[source]¶ Bases:
imagine.tools.class_tools.BaseClass
Base class for writing helpers to convert arbitrary observational datasets into IMAGINE’s standardized format
-
REQ_ATTRS
= ['NAME']¶
-
cov
¶
-
data
¶ Array in the shape (1, N)
-
frequency
¶
-
key
¶ Key used in the Observables_dictionary
-
name
¶
-
-
class
imagine.observables.dataset.
TabularDataset
(data, name, data_column=None, units=None, coordinates_type='galactic', lon_column='lon', lat_column='lat', x_column='x', y_column='y', z_column='z', error_column=None, frequency='nan', tag='nan')[source]¶ Bases:
imagine.observables.dataset.Dataset
Base class for tabular datasets, where the data is input in either in a Python dictionary-like object (
dict
,astropy.table.Table
,pandas.DataFrame
, etc).Parameters: data (dict-like) – Can be a
dict
,astropy.table.Table
,pandas.DataFrame
, or similar object containing the data.name (str) – Standard name of this type of observable. E.g. ‘fd’, ‘sync’, ‘dm’.
data_column (str) – Key used to access the relevant dataset from the provided data (i.e. data[data_column]).
units (astropy.units.Unit or str) – Units used for the data.
coordinates_type (str) – Type of coordinates used. Can be ‘galactic’ or ‘cartesian’.
lon_column (str) – Key used to access the Galactic longitudes (in deg) from data.
lat_column (str) – Key used to access the Galactic latitudes (in deg) from data.
lat_column (str) – Key used to access the Galactic latitudes (in deg) from data.
x_column, y_column, z_column (str) – Keys used to access the coordinates (in kpc) from data.
frequency (astropy.units.Quantity) – Frequency of the measurement (if relevant)
tag (str) –
- Extra information associated with the observable.
- ‘I’ - total intensity (in unit K-cmb)
- ‘Q’ - Stokes Q (in unit K-cmb, IAU convention)
- ‘U’ - Stokes U (in unit K-cmb, IAU convention)
- ‘PI’ - polarisation intensity (in unit K-cmb)
- ‘PA’ - polarisation angle (in unit rad, IAU convention)
-
class
imagine.observables.dataset.
HEALPixDataset
(data, error=None, cov=None, Nside=None)[source]¶ Bases:
imagine.observables.dataset.Dataset
Base class for HEALPix datasets, which are input as a simple 1D-array without explicit coordinate information
-
class
imagine.observables.dataset.
FaradayDepthHEALPixDataset
(data, error=None, cov=None, Nside=None)[source]¶ Bases:
imagine.observables.dataset.HEALPixDataset
Stores a Faraday depth map into an IMAGINE-compatible dataset
Parameters: - data (numpy.ndarray) – 1D-array containing the HEALPix map
- Nside (int, optional) – For extra internal consistency checking. If Nside is present, it will be checked whether \(12\times N_{side}^2\) matches
-
data
¶ Data in ObservablesDict-compatible shape
-
key
¶ Standard key associated with this observable
-
NAME
= 'fd'¶
-
class
imagine.observables.dataset.
SynchrotronHEALPixDataset
(data, frequency, type, error=None, cov=None, Nside=None)[source]¶ Bases:
imagine.observables.dataset.HEALPixDataset
Stores a synchrotron emission map into an IMAGINE-compatible dataset. This can be Stokes parameters, total and polarised intensity, and polarisation angle.
The parameter type and the units of the map in data must follow:
- ‘I’ - total intensity (in unit K-cmb)
- ‘Q’ - Stokes Q (in unit K-cmb, IAU convention)
- ‘U’ - Stokes U (in unit K-cmb, IAU convention)
- ‘PI’ - polarisation intensity (in unit K-cmb)
- ‘PA’ - polarisation angle (in unit rad, IAU convention)
Parameters: - data (numpy.ndarray) – 1D-array containing the HEALPix map
- frequency (astropy.units.Quantity) – Frequency of the measurement (if relevant)
- Nside (int, optional) – For extra internal consistency checking. If Nside is present, it will be checked whether \(12\times N_{side}^2\) matches data.size
- type (str) – The type of map being supplied in data.
-
data
¶ Data in ObservablesDict-compatible shape
-
key
¶ Standard key associated with this observable
-
NAME
= 'sync'¶
-
class
imagine.observables.dataset.
DispersionMeasureHEALPixDataset
(data, error=None, cov=None, Nside=None)[source]¶ Bases:
imagine.observables.dataset.HEALPixDataset
Stores a dispersion measure map into an IMAGINE-compatible dataset
Parameters: - data (numpy.ndarray) – 1D-array containing the HEALPix map
- Nside (int, optional) – For extra internal consistency checking. If Nside is present, it will be checked whether \(12\times N_{side}^2\) matches
-
data
¶ Data in ObservablesDict-compatible shape
-
key
¶ Standard key associated with this observable
-
NAME
= 'dm'¶
imagine.observables.observable module¶
In the Observable class we define three data types, i.e., - ‘measured’ - ‘simulated’ - ‘covariance’ where ‘measured’ indicates the hosted data is from measurements, which has a single realization, ‘simulated’ indicates the hosted data is from simulations, which has multiple realizations, ‘covariance’ indicates the hosted data is a covariance matrix, which has a single realization but by default should not be stored/read/written by a single computing node.
‘measred’ data puts its identical copies on all computing nodes, which means each node has a full storage of ‘measured’ data.
‘simulated’ data puts different realizations on different nodes, which means each node has part of the full realizations, but at least a full version of one single realization.
‘covariance’ data distributes itself into all computing nodes, which means to have a full set of ‘covariance’ data, we have to collect pieces from all the computing nodes.
-
class
imagine.observables.observable.
Observable
(data=None, dtype=None, coords=None)[source]¶ Bases:
object
Observable class is designed for storing/manipulating distributed information. For the testing suits, please turn to “imagine/tests/observable_tests.py”.
Parameters: - data (numpy.ndarray) – distributed/copied data
- dtype (str) – Data type, must be either: ‘measured’, ‘simulated’ or ‘covariance’
-
append
(new_data)[source]¶ appending new data happens only to SIMULATED dtype the new data to be appended should also be distributed which makes the appending operation naturally in parallel
rewrite flag will be switched off once rewriten has been performed
-
data
¶ Data stored in the LOCAL processor (numpy.ndarray, read-only).
-
dtype
¶ ‘measured’, ‘simulated’ or ‘covariance’
Type: Data type, can be either
-
ensemble_mean
¶
-
global_data
¶ Data gathered from ALL processors (numpy.ndarray, read-only). Note that only master node hosts the global data, while slave nodes hosts None.
-
rw_flag
¶ Rewriting flag, if true, append method will perform rewriting
-
shape
¶ Shape of the GLOBAL array, i.e. considering all processors (numpy.ndarray, read-only).
-
size
¶ Local data size (int, read-only) this size means the dimension of input data not the sample size of realizations
imagine.observables.observable_dict module¶
For convenience we define dictionary of Observable objects as ObservableDict from which one can define define the classes Measurements, Covariances, Simulations and Masks, which can be used to store:
- measured data sets
- measured/simulated covariances
- simulated ensemble sets
- mask “maps” (but actually mask lists)
- Conventions for observables entries
- Faraday depth: (‘fd’,’nan’,str(size/Nside),’nan’)
- Dispersion measure: (‘dm’,’nan’,str(pix),’nan’)
- Synchrotron emission: (‘sync’,str(freq),str(pix),X)
- where X stands for:
- ‘I’ - total intensity (in unit K-cmb)
- ‘Q’ - Stokes Q (in unit K-cmb, IAU convention)
- ‘U’ - Stokes U (in unit K-cmb, IAU convention)
- ‘PI’ - polarisation intensity (in unit K-cmb)
- ‘PA’ - polarisation angle (in unit rad, IAU convention)
- Remarks:
- str(freq), polarisation-related-flag are redundant for Faraday depth
- and dispersion measure so we put ‘nan’ instead
- str(pix/nside) stores either Healpix Nside, or just number of
- pixels/points we do this for flexibility, in case users have non-HEALPix based in/output
- Remarks on the observable tags
str(freq), polarisation-related-flag are redundant for Faraday depth and dispersion measure so we put ‘nan’ instead
str(pix/nside) stores either Healpix Nisde, or just number of pixels/points we do this for flexibility, in case users have non-HEALPix-map-like in/output
- Masking convention
- masked erea associated with pixel value 0, unmasked area with pixel value 1
- Masking method
- mask only applies to observables/covariances observable after masking will be re-recorded as plain data type
Note
Distribution with MPI
- all data are either distributed or copied, where “copied” means each node stores the identical copy which is convenient for hosting measured data and mask map
- Covariances has Field object with global shape “around” (data_size//mpisize, data_size) “around” means to distribute matrix correctly as described in “imagine/tools/mpi_helper.py”
-
class
imagine.observables.observable_dict.
ObservableDict
[source]¶ Bases:
imagine.tools.class_tools.BaseClass
Base class from which imagine.observables.observable_dict.Measurements, imagine.observables.observable_dict.Covariances, imagine.observables.observable_dict.Simulations and imagine.observables.observable_dict.Masks are derived.
See imagine.observables.observable_dict module documentation for further details.
-
append
(name, new_data, plain=False)[source]¶ Adds/updates name and data
Parameters: - name (str tuple) – Should follow the convention:
(data-name,str(data-freq),str(data-Nside/size),str(ext))
. If data is independent from frequency, set ‘nan’. ext can be ‘I’,’Q’,’U’,’PI’,’PA’, ‘nan’ or other customized tags. - new_data – distributed/copied
numpy.ndarray
orObservable
- plain (bool) – If True, means unstructured data. If False (default case), means HEALPix-like sky map.
- name (str tuple) – Should follow the convention:
-
apply_mask
(mask_dict)[source]¶ Parameters: mask_dict (imagine.observables.observable_dict.Masks) – Masks object
-
archive
¶
-
-
class
imagine.observables.observable_dict.
Masks
[source]¶ Bases:
imagine.observables.observable_dict.ObservableDict
Stores HEALPix mask maps
See imagine.observables.observable_dict module documentation for further details.
-
append
(name, new_data, plain=False)[source]¶ Adds/updates name and data
Parameters: - name (str tuple) – Should follow the convention:
(data-name,str(data-freq),str(data-Nside/"tab"),str(ext))
. If data is independent from frequency, set ‘nan’. ext can be ‘I’,’Q’,’U’,’PI’,’PA’, ‘nan’ or other customized tags. - new_data – distributed/copied
numpy.ndarray
orObservable
- plain (bool) – If True, means unstructured data. If False (default case), means HEALPix-like sky map.
- name (str tuple) – Should follow the convention:
-
-
class
imagine.observables.observable_dict.
Measurements
[source]¶ Bases:
imagine.observables.observable_dict.ObservableDict
Stores observational data sets
See imagine.observables.observable_dict module documentation for further details.
-
append
(name=None, new_data=None, plain=False, dataset=None)[source]¶ Adds/updates name and data
Parameters: - dataset (imagine.observables.dataset.Dataset) – The IMAGINE dataset already adjusts the format of the data and sets the adequate key. If dataset is present, all other arguments will be ignored.
- name (str tuple) – Should follow the convention:
(data-name,str(data-freq),str(data-Nside)/"tab",str(ext))
. If data is independent from frequency, set ‘nan’. ext can be ‘I’,’Q’,’U’,’PI’,’PA’, ‘nan’ or other customized tags. - new_data (numpy.ndarray or imagine.observables.observable.Observable) – distributed/copied
numpy.ndarray
orObservable
- plain (bool) – If True, means unstructured/tabular data. If False (default case), means HEALPix-like sky map.
-
-
class
imagine.observables.observable_dict.
Simulations
[source]¶ Bases:
imagine.observables.observable_dict.ObservableDict
Stores simulated ensemble sets
See imagine.observables.observable_dict module documentation for further details.
-
append
(name, new_data, plain=False)[source]¶ Adds/updates name and data
Parameters: - name (str tuple) – Should follow the convention:
(data-name,str(data-freq),str(data-Nside)/"tab",str(ext))
. If data is independent from frequency, set ‘nan’. ext can be ‘I’,’Q’,’U’,’PI’,’PA’, ‘nan’ or other customized tags. - new_data – distributed/copied
numpy.ndarray
orObservable
- plain (bool) – If True, means unstructured data. If False (default case), means HEALPix-like sky map.
- name (str tuple) – Should follow the convention:
-
-
class
imagine.observables.observable_dict.
Covariances
[source]¶ Bases:
imagine.observables.observable_dict.ObservableDict
Stores observational covariances
See imagine.observables.observable_dict module documentation for further details.
-
append
(name=None, new_data=None, plain=False, dataset=None)[source]¶ Adds/updates name and data
Parameters: - name (str tuple) – Should follow the convention:
(data-name,str(data-freq),str(data-Nside)/"tab",str(ext))
. If data is independent from frequency, set ‘nan’. ext can be ‘I’,’Q’,’U’,’PI’,’PA’, ‘nan’ or other customized tags. - data – distributed/copied ndarray/Observable
- plain (bool) – If True, means unstructured data. If False (default case), means HEALPix-like sky map.
- name (str tuple) – Should follow the convention:
-
Module contents¶
-
class
imagine.observables.
Dataset
[source]¶ Bases:
imagine.tools.class_tools.BaseClass
Base class for writing helpers to convert arbitrary observational datasets into IMAGINE’s standardized format
-
REQ_ATTRS
= ['NAME']¶
-
cov
¶
-
data
¶ Array in the shape (1, N)
-
frequency
¶
-
key
¶ Key used in the Observables_dictionary
-
name
¶
-
-
class
imagine.observables.
TabularDataset
(data, name, data_column=None, units=None, coordinates_type='galactic', lon_column='lon', lat_column='lat', x_column='x', y_column='y', z_column='z', error_column=None, frequency='nan', tag='nan')[source]¶ Bases:
imagine.observables.dataset.Dataset
Base class for tabular datasets, where the data is input in either in a Python dictionary-like object (
dict
,astropy.table.Table
,pandas.DataFrame
, etc).Parameters: data (dict-like) – Can be a
dict
,astropy.table.Table
,pandas.DataFrame
, or similar object containing the data.name (str) – Standard name of this type of observable. E.g. ‘fd’, ‘sync’, ‘dm’.
data_column (str) – Key used to access the relevant dataset from the provided data (i.e. data[data_column]).
units (astropy.units.Unit or str) – Units used for the data.
coordinates_type (str) – Type of coordinates used. Can be ‘galactic’ or ‘cartesian’.
lon_column (str) – Key used to access the Galactic longitudes (in deg) from data.
lat_column (str) – Key used to access the Galactic latitudes (in deg) from data.
lat_column (str) – Key used to access the Galactic latitudes (in deg) from data.
x_column, y_column, z_column (str) – Keys used to access the coordinates (in kpc) from data.
frequency (astropy.units.Quantity) – Frequency of the measurement (if relevant)
tag (str) –
- Extra information associated with the observable.
- ‘I’ - total intensity (in unit K-cmb)
- ‘Q’ - Stokes Q (in unit K-cmb, IAU convention)
- ‘U’ - Stokes U (in unit K-cmb, IAU convention)
- ‘PI’ - polarisation intensity (in unit K-cmb)
- ‘PA’ - polarisation angle (in unit rad, IAU convention)
-
class
imagine.observables.
HEALPixDataset
(data, error=None, cov=None, Nside=None)[source]¶ Bases:
imagine.observables.dataset.Dataset
Base class for HEALPix datasets, which are input as a simple 1D-array without explicit coordinate information
-
class
imagine.observables.
FaradayDepthHEALPixDataset
(data, error=None, cov=None, Nside=None)[source]¶ Bases:
imagine.observables.dataset.HEALPixDataset
Stores a Faraday depth map into an IMAGINE-compatible dataset
Parameters: - data (numpy.ndarray) – 1D-array containing the HEALPix map
- Nside (int, optional) – For extra internal consistency checking. If Nside is present, it will be checked whether \(12\times N_{side}^2\) matches
-
data
¶ Data in ObservablesDict-compatible shape
-
key
¶ Standard key associated with this observable
-
NAME
= 'fd'¶
-
class
imagine.observables.
SynchrotronHEALPixDataset
(data, frequency, type, error=None, cov=None, Nside=None)[source]¶ Bases:
imagine.observables.dataset.HEALPixDataset
Stores a synchrotron emission map into an IMAGINE-compatible dataset. This can be Stokes parameters, total and polarised intensity, and polarisation angle.
The parameter type and the units of the map in data must follow:
- ‘I’ - total intensity (in unit K-cmb)
- ‘Q’ - Stokes Q (in unit K-cmb, IAU convention)
- ‘U’ - Stokes U (in unit K-cmb, IAU convention)
- ‘PI’ - polarisation intensity (in unit K-cmb)
- ‘PA’ - polarisation angle (in unit rad, IAU convention)
Parameters: - data (numpy.ndarray) – 1D-array containing the HEALPix map
- frequency (astropy.units.Quantity) – Frequency of the measurement (if relevant)
- Nside (int, optional) – For extra internal consistency checking. If Nside is present, it will be checked whether \(12\times N_{side}^2\) matches data.size
- type (str) – The type of map being supplied in data.
-
data
¶ Data in ObservablesDict-compatible shape
-
key
¶ Standard key associated with this observable
-
NAME
= 'sync'¶
-
class
imagine.observables.
DispersionMeasureHEALPixDataset
(data, error=None, cov=None, Nside=None)[source]¶ Bases:
imagine.observables.dataset.HEALPixDataset
Stores a dispersion measure map into an IMAGINE-compatible dataset
Parameters: - data (numpy.ndarray) – 1D-array containing the HEALPix map
- Nside (int, optional) – For extra internal consistency checking. If Nside is present, it will be checked whether \(12\times N_{side}^2\) matches
-
data
¶ Data in ObservablesDict-compatible shape
-
key
¶ Standard key associated with this observable
-
NAME
= 'dm'¶
-
class
imagine.observables.
Observable
(data=None, dtype=None, coords=None)[source]¶ Bases:
object
Observable class is designed for storing/manipulating distributed information. For the testing suits, please turn to “imagine/tests/observable_tests.py”.
Parameters: - data (numpy.ndarray) – distributed/copied data
- dtype (str) – Data type, must be either: ‘measured’, ‘simulated’ or ‘covariance’
-
append
(new_data)[source]¶ appending new data happens only to SIMULATED dtype the new data to be appended should also be distributed which makes the appending operation naturally in parallel
rewrite flag will be switched off once rewriten has been performed
-
data
¶ Data stored in the LOCAL processor (numpy.ndarray, read-only).
-
dtype
¶ ‘measured’, ‘simulated’ or ‘covariance’
Type: Data type, can be either
-
ensemble_mean
¶
-
global_data
¶ Data gathered from ALL processors (numpy.ndarray, read-only). Note that only master node hosts the global data, while slave nodes hosts None.
-
rw_flag
¶ Rewriting flag, if true, append method will perform rewriting
-
shape
¶ Shape of the GLOBAL array, i.e. considering all processors (numpy.ndarray, read-only).
-
size
¶ Local data size (int, read-only) this size means the dimension of input data not the sample size of realizations
-
class
imagine.observables.
ObservableDict
[source]¶ Bases:
imagine.tools.class_tools.BaseClass
Base class from which imagine.observables.observable_dict.Measurements, imagine.observables.observable_dict.Covariances, imagine.observables.observable_dict.Simulations and imagine.observables.observable_dict.Masks are derived.
See imagine.observables.observable_dict module documentation for further details.
-
append
(name, new_data, plain=False)[source]¶ Adds/updates name and data
Parameters: - name (str tuple) – Should follow the convention:
(data-name,str(data-freq),str(data-Nside/size),str(ext))
. If data is independent from frequency, set ‘nan’. ext can be ‘I’,’Q’,’U’,’PI’,’PA’, ‘nan’ or other customized tags. - new_data – distributed/copied
numpy.ndarray
orObservable
- plain (bool) – If True, means unstructured data. If False (default case), means HEALPix-like sky map.
- name (str tuple) – Should follow the convention:
-
apply_mask
(mask_dict)[source]¶ Parameters: mask_dict (imagine.observables.observable_dict.Masks) – Masks object
-
archive
¶
-
-
class
imagine.observables.
Masks
[source]¶ Bases:
imagine.observables.observable_dict.ObservableDict
Stores HEALPix mask maps
See imagine.observables.observable_dict module documentation for further details.
-
append
(name, new_data, plain=False)[source]¶ Adds/updates name and data
Parameters: - name (str tuple) – Should follow the convention:
(data-name,str(data-freq),str(data-Nside/"tab"),str(ext))
. If data is independent from frequency, set ‘nan’. ext can be ‘I’,’Q’,’U’,’PI’,’PA’, ‘nan’ or other customized tags. - new_data – distributed/copied
numpy.ndarray
orObservable
- plain (bool) – If True, means unstructured data. If False (default case), means HEALPix-like sky map.
- name (str tuple) – Should follow the convention:
-
-
class
imagine.observables.
Measurements
[source]¶ Bases:
imagine.observables.observable_dict.ObservableDict
Stores observational data sets
See imagine.observables.observable_dict module documentation for further details.
-
append
(name=None, new_data=None, plain=False, dataset=None)[source]¶ Adds/updates name and data
Parameters: - dataset (imagine.observables.dataset.Dataset) – The IMAGINE dataset already adjusts the format of the data and sets the adequate key. If dataset is present, all other arguments will be ignored.
- name (str tuple) – Should follow the convention:
(data-name,str(data-freq),str(data-Nside)/"tab",str(ext))
. If data is independent from frequency, set ‘nan’. ext can be ‘I’,’Q’,’U’,’PI’,’PA’, ‘nan’ or other customized tags. - new_data (numpy.ndarray or imagine.observables.observable.Observable) – distributed/copied
numpy.ndarray
orObservable
- plain (bool) – If True, means unstructured/tabular data. If False (default case), means HEALPix-like sky map.
-
-
class
imagine.observables.
Simulations
[source]¶ Bases:
imagine.observables.observable_dict.ObservableDict
Stores simulated ensemble sets
See imagine.observables.observable_dict module documentation for further details.
-
append
(name, new_data, plain=False)[source]¶ Adds/updates name and data
Parameters: - name (str tuple) – Should follow the convention:
(data-name,str(data-freq),str(data-Nside)/"tab",str(ext))
. If data is independent from frequency, set ‘nan’. ext can be ‘I’,’Q’,’U’,’PI’,’PA’, ‘nan’ or other customized tags. - new_data – distributed/copied
numpy.ndarray
orObservable
- plain (bool) – If True, means unstructured data. If False (default case), means HEALPix-like sky map.
- name (str tuple) – Should follow the convention:
-
-
class
imagine.observables.
Covariances
[source]¶ Bases:
imagine.observables.observable_dict.ObservableDict
Stores observational covariances
See imagine.observables.observable_dict module documentation for further details.
-
append
(name=None, new_data=None, plain=False, dataset=None)[source]¶ Adds/updates name and data
Parameters: - name (str tuple) – Should follow the convention:
(data-name,str(data-freq),str(data-Nside)/"tab",str(ext))
. If data is independent from frequency, set ‘nan’. ext can be ‘I’,’Q’,’U’,’PI’,’PA’, ‘nan’ or other customized tags. - data – distributed/copied ndarray/Observable
- plain (bool) – If True, means unstructured data. If False (default case), means HEALPix-like sky map.
- name (str tuple) – Should follow the convention:
-