imagine.observables package

Submodules

imagine.observables.dataset module

Datasets are auxiliary classes used to facilitate the reading and inclusion of observational data in the IMAGINE pipeline

class imagine.observables.dataset.Dataset[source]

Bases: imagine.tools.class_tools.BaseClass

Base class for writing helpers to convert arbitrary observational datasets into IMAGINE’s standardized format

REQ_ATTRS = ['NAME']
cov
data

Array in the shape (1, N)

frequency
key

Key used in the Observables_dictionary

name
class imagine.observables.dataset.TabularDataset(data, name, data_column=None, units=None, coordinates_type='galactic', lon_column='lon', lat_column='lat', x_column='x', y_column='y', z_column='z', error_column=None, frequency='nan', tag='nan')[source]

Bases: imagine.observables.dataset.Dataset

Base class for tabular datasets, where the data is input in either in a Python dictionary-like object (dict, astropy.table.Table, pandas.DataFrame, etc).

Parameters:
  • data (dict-like) – Can be a dict, astropy.table.Table, pandas.DataFrame, or similar object containing the data.

  • name (str) – Standard name of this type of observable. E.g. ‘fd’, ‘sync’, ‘dm’.

  • data_column (str) – Key used to access the relevant dataset from the provided data (i.e. data[data_column]).

  • units (astropy.units.Unit or str) – Units used for the data.

  • coordinates_type (str) – Type of coordinates used. Can be ‘galactic’ or ‘cartesian’.

  • lon_column (str) – Key used to access the Galactic longitudes (in deg) from data.

  • lat_column (str) – Key used to access the Galactic latitudes (in deg) from data.

  • lat_column (str) – Key used to access the Galactic latitudes (in deg) from data.

  • x_column, y_column, z_column (str) – Keys used to access the coordinates (in kpc) from data.

  • frequency (astropy.units.Quantity) – Frequency of the measurement (if relevant)

  • tag (str) –

    Extra information associated with the observable.
    • ‘I’ - total intensity (in unit K-cmb)
    • ‘Q’ - Stokes Q (in unit K-cmb, IAU convention)
    • ‘U’ - Stokes U (in unit K-cmb, IAU convention)
    • ‘PI’ - polarisation intensity (in unit K-cmb)
    • ‘PA’ - polarisation angle (in unit rad, IAU convention)
class imagine.observables.dataset.HEALPixDataset(data, error=None, cov=None, Nside=None)[source]

Bases: imagine.observables.dataset.Dataset

Base class for HEALPix datasets, which are input as a simple 1D-array without explicit coordinate information

class imagine.observables.dataset.FaradayDepthHEALPixDataset(data, error=None, cov=None, Nside=None)[source]

Bases: imagine.observables.dataset.HEALPixDataset

Stores a Faraday depth map into an IMAGINE-compatible dataset

Parameters:
  • data (numpy.ndarray) – 1D-array containing the HEALPix map
  • Nside (int, optional) – For extra internal consistency checking. If Nside is present, it will be checked whether \(12\times N_{side}^2\) matches
data

Data in ObservablesDict-compatible shape

key

Standard key associated with this observable

NAME = 'fd'
class imagine.observables.dataset.SynchrotronHEALPixDataset(data, frequency, type, error=None, cov=None, Nside=None)[source]

Bases: imagine.observables.dataset.HEALPixDataset

Stores a synchrotron emission map into an IMAGINE-compatible dataset. This can be Stokes parameters, total and polarised intensity, and polarisation angle.

The parameter type and the units of the map in data must follow:

  • ‘I’ - total intensity (in unit K-cmb)
  • ‘Q’ - Stokes Q (in unit K-cmb, IAU convention)
  • ‘U’ - Stokes U (in unit K-cmb, IAU convention)
  • ‘PI’ - polarisation intensity (in unit K-cmb)
  • ‘PA’ - polarisation angle (in unit rad, IAU convention)
Parameters:
  • data (numpy.ndarray) – 1D-array containing the HEALPix map
  • frequency (astropy.units.Quantity) – Frequency of the measurement (if relevant)
  • Nside (int, optional) – For extra internal consistency checking. If Nside is present, it will be checked whether \(12\times N_{side}^2\) matches data.size
  • type (str) – The type of map being supplied in data.
data

Data in ObservablesDict-compatible shape

key

Standard key associated with this observable

NAME = 'sync'
class imagine.observables.dataset.DispersionMeasureHEALPixDataset(data, error=None, cov=None, Nside=None)[source]

Bases: imagine.observables.dataset.HEALPixDataset

Stores a dispersion measure map into an IMAGINE-compatible dataset

Parameters:
  • data (numpy.ndarray) – 1D-array containing the HEALPix map
  • Nside (int, optional) – For extra internal consistency checking. If Nside is present, it will be checked whether \(12\times N_{side}^2\) matches
data

Data in ObservablesDict-compatible shape

key

Standard key associated with this observable

NAME = 'dm'

imagine.observables.observable module

In the Observable class we define three data types, i.e., - ‘measured’ - ‘simulated’ - ‘covariance’ where ‘measured’ indicates the hosted data is from measurements, which has a single realization, ‘simulated’ indicates the hosted data is from simulations, which has multiple realizations, ‘covariance’ indicates the hosted data is a covariance matrix, which has a single realization but by default should not be stored/read/written by a single computing node.

‘measred’ data puts its identical copies on all computing nodes, which means each node has a full storage of ‘measured’ data.

‘simulated’ data puts different realizations on different nodes, which means each node has part of the full realizations, but at least a full version of one single realization.

‘covariance’ data distributes itself into all computing nodes, which means to have a full set of ‘covariance’ data, we have to collect pieces from all the computing nodes.

class imagine.observables.observable.Observable(data=None, dtype=None, coords=None)[source]

Bases: object

Observable class is designed for storing/manipulating distributed information. For the testing suits, please turn to “imagine/tests/observable_tests.py”.

Parameters:
  • data (numpy.ndarray) – distributed/copied data
  • dtype (str) – Data type, must be either: ‘measured’, ‘simulated’ or ‘covariance’
append(new_data)[source]

appending new data happens only to SIMULATED dtype the new data to be appended should also be distributed which makes the appending operation naturally in parallel

rewrite flag will be switched off once rewriten has been performed

data

Data stored in the LOCAL processor (numpy.ndarray, read-only).

dtype

‘measured’, ‘simulated’ or ‘covariance’

Type:Data type, can be either
ensemble_mean
global_data

Data gathered from ALL processors (numpy.ndarray, read-only). Note that only master node hosts the global data, while slave nodes hosts None.

rw_flag

Rewriting flag, if true, append method will perform rewriting

shape

Shape of the GLOBAL array, i.e. considering all processors (numpy.ndarray, read-only).

size

Local data size (int, read-only) this size means the dimension of input data not the sample size of realizations

imagine.observables.observable_dict module

For convenience we define dictionary of Observable objects as ObservableDict from which one can define define the classes Measurements, Covariances, Simulations and Masks, which can be used to store:

  • measured data sets
  • measured/simulated covariances
  • simulated ensemble sets
  • mask “maps” (but actually mask lists)
Conventions for observables entries
  • Faraday depth: (‘fd’,’nan’,str(size/Nside),’nan’)
  • Dispersion measure: (‘dm’,’nan’,str(pix),’nan’)
  • Synchrotron emission: (‘sync’,str(freq),str(pix),X)
    where X stands for:
    • ‘I’ - total intensity (in unit K-cmb)
    • ‘Q’ - Stokes Q (in unit K-cmb, IAU convention)
    • ‘U’ - Stokes U (in unit K-cmb, IAU convention)
    • ‘PI’ - polarisation intensity (in unit K-cmb)
    • ‘PA’ - polarisation angle (in unit rad, IAU convention)
Remarks:
  • str(freq), polarisation-related-flag are redundant for Faraday depth
    and dispersion measure so we put ‘nan’ instead
  • str(pix/nside) stores either Healpix Nside, or just number of
    pixels/points we do this for flexibility, in case users have non-HEALPix based in/output
Remarks on the observable tags

str(freq), polarisation-related-flag are redundant for Faraday depth and dispersion measure so we put ‘nan’ instead

str(pix/nside) stores either Healpix Nisde, or just number of pixels/points we do this for flexibility, in case users have non-HEALPix-map-like in/output

Masking convention
masked erea associated with pixel value 0, unmasked area with pixel value 1
Masking method
mask only applies to observables/covariances observable after masking will be re-recorded as plain data type

Note

Distribution with MPI

  • all data are either distributed or copied, where “copied” means each node stores the identical copy which is convenient for hosting measured data and mask map
  • Covariances has Field object with global shape “around” (data_size//mpisize, data_size) “around” means to distribute matrix correctly as described in “imagine/tools/mpi_helper.py”
class imagine.observables.observable_dict.ObservableDict[source]

Bases: imagine.tools.class_tools.BaseClass

Base class from which imagine.observables.observable_dict.Measurements, imagine.observables.observable_dict.Covariances, imagine.observables.observable_dict.Simulations and imagine.observables.observable_dict.Masks are derived.

See imagine.observables.observable_dict module documentation for further details.

append(name, new_data, plain=False)[source]

Adds/updates name and data

Parameters:
  • name (str tuple) – Should follow the convention: (data-name,str(data-freq),str(data-Nside/size),str(ext)). If data is independent from frequency, set ‘nan’. ext can be ‘I’,’Q’,’U’,’PI’,’PA’, ‘nan’ or other customized tags.
  • new_data – distributed/copied numpy.ndarray or Observable
  • plain (bool) – If True, means unstructured data. If False (default case), means HEALPix-like sky map.
apply_mask(mask_dict)[source]
Parameters:mask_dict (imagine.observables.observable_dict.Masks) – Masks object
keys()[source]
archive
class imagine.observables.observable_dict.Masks[source]

Bases: imagine.observables.observable_dict.ObservableDict

Stores HEALPix mask maps

See imagine.observables.observable_dict module documentation for further details.

append(name, new_data, plain=False)[source]

Adds/updates name and data

Parameters:
  • name (str tuple) – Should follow the convention: (data-name,str(data-freq),str(data-Nside/"tab"),str(ext)). If data is independent from frequency, set ‘nan’. ext can be ‘I’,’Q’,’U’,’PI’,’PA’, ‘nan’ or other customized tags.
  • new_data – distributed/copied numpy.ndarray or Observable
  • plain (bool) – If True, means unstructured data. If False (default case), means HEALPix-like sky map.
apply_mask(*args, **kwargs)[source]
Parameters:mask_dict (imagine.observables.observable_dict.Masks) – Masks object
class imagine.observables.observable_dict.Measurements[source]

Bases: imagine.observables.observable_dict.ObservableDict

Stores observational data sets

See imagine.observables.observable_dict module documentation for further details.

append(name=None, new_data=None, plain=False, dataset=None)[source]

Adds/updates name and data

Parameters:
  • dataset (imagine.observables.dataset.Dataset) – The IMAGINE dataset already adjusts the format of the data and sets the adequate key. If dataset is present, all other arguments will be ignored.
  • name (str tuple) – Should follow the convention: (data-name,str(data-freq),str(data-Nside)/"tab",str(ext)). If data is independent from frequency, set ‘nan’. ext can be ‘I’,’Q’,’U’,’PI’,’PA’, ‘nan’ or other customized tags.
  • new_data (numpy.ndarray or imagine.observables.observable.Observable) – distributed/copied numpy.ndarray or Observable
  • plain (bool) – If True, means unstructured/tabular data. If False (default case), means HEALPix-like sky map.
apply_mask(mask_dict=None)[source]
Parameters:mask_dict (imagine.observables.observable_dict.Masks) – Masks object
class imagine.observables.observable_dict.Simulations[source]

Bases: imagine.observables.observable_dict.ObservableDict

Stores simulated ensemble sets

See imagine.observables.observable_dict module documentation for further details.

append(name, new_data, plain=False)[source]

Adds/updates name and data

Parameters:
  • name (str tuple) – Should follow the convention: (data-name,str(data-freq),str(data-Nside)/"tab",str(ext)). If data is independent from frequency, set ‘nan’. ext can be ‘I’,’Q’,’U’,’PI’,’PA’, ‘nan’ or other customized tags.
  • new_data – distributed/copied numpy.ndarray or Observable
  • plain (bool) – If True, means unstructured data. If False (default case), means HEALPix-like sky map.
apply_mask(mask_dict)[source]
Parameters:mask_dict (imagine.observables.observable_dict.Masks) – Masks object
class imagine.observables.observable_dict.Covariances[source]

Bases: imagine.observables.observable_dict.ObservableDict

Stores observational covariances

See imagine.observables.observable_dict module documentation for further details.

append(name=None, new_data=None, plain=False, dataset=None)[source]

Adds/updates name and data

Parameters:
  • name (str tuple) – Should follow the convention: (data-name,str(data-freq),str(data-Nside)/"tab",str(ext)). If data is independent from frequency, set ‘nan’. ext can be ‘I’,’Q’,’U’,’PI’,’PA’, ‘nan’ or other customized tags.
  • data – distributed/copied ndarray/Observable
  • plain (bool) – If True, means unstructured data. If False (default case), means HEALPix-like sky map.
apply_mask(mask_dict)[source]
Parameters:mask_dict (imagine.observables.observable_dict.Masks) – Masks object

Module contents

class imagine.observables.Dataset[source]

Bases: imagine.tools.class_tools.BaseClass

Base class for writing helpers to convert arbitrary observational datasets into IMAGINE’s standardized format

REQ_ATTRS = ['NAME']
cov
data

Array in the shape (1, N)

frequency
key

Key used in the Observables_dictionary

name
class imagine.observables.TabularDataset(data, name, data_column=None, units=None, coordinates_type='galactic', lon_column='lon', lat_column='lat', x_column='x', y_column='y', z_column='z', error_column=None, frequency='nan', tag='nan')[source]

Bases: imagine.observables.dataset.Dataset

Base class for tabular datasets, where the data is input in either in a Python dictionary-like object (dict, astropy.table.Table, pandas.DataFrame, etc).

Parameters:
  • data (dict-like) – Can be a dict, astropy.table.Table, pandas.DataFrame, or similar object containing the data.

  • name (str) – Standard name of this type of observable. E.g. ‘fd’, ‘sync’, ‘dm’.

  • data_column (str) – Key used to access the relevant dataset from the provided data (i.e. data[data_column]).

  • units (astropy.units.Unit or str) – Units used for the data.

  • coordinates_type (str) – Type of coordinates used. Can be ‘galactic’ or ‘cartesian’.

  • lon_column (str) – Key used to access the Galactic longitudes (in deg) from data.

  • lat_column (str) – Key used to access the Galactic latitudes (in deg) from data.

  • lat_column (str) – Key used to access the Galactic latitudes (in deg) from data.

  • x_column, y_column, z_column (str) – Keys used to access the coordinates (in kpc) from data.

  • frequency (astropy.units.Quantity) – Frequency of the measurement (if relevant)

  • tag (str) –

    Extra information associated with the observable.
    • ‘I’ - total intensity (in unit K-cmb)
    • ‘Q’ - Stokes Q (in unit K-cmb, IAU convention)
    • ‘U’ - Stokes U (in unit K-cmb, IAU convention)
    • ‘PI’ - polarisation intensity (in unit K-cmb)
    • ‘PA’ - polarisation angle (in unit rad, IAU convention)
class imagine.observables.HEALPixDataset(data, error=None, cov=None, Nside=None)[source]

Bases: imagine.observables.dataset.Dataset

Base class for HEALPix datasets, which are input as a simple 1D-array without explicit coordinate information

class imagine.observables.FaradayDepthHEALPixDataset(data, error=None, cov=None, Nside=None)[source]

Bases: imagine.observables.dataset.HEALPixDataset

Stores a Faraday depth map into an IMAGINE-compatible dataset

Parameters:
  • data (numpy.ndarray) – 1D-array containing the HEALPix map
  • Nside (int, optional) – For extra internal consistency checking. If Nside is present, it will be checked whether \(12\times N_{side}^2\) matches
data

Data in ObservablesDict-compatible shape

key

Standard key associated with this observable

NAME = 'fd'
class imagine.observables.SynchrotronHEALPixDataset(data, frequency, type, error=None, cov=None, Nside=None)[source]

Bases: imagine.observables.dataset.HEALPixDataset

Stores a synchrotron emission map into an IMAGINE-compatible dataset. This can be Stokes parameters, total and polarised intensity, and polarisation angle.

The parameter type and the units of the map in data must follow:

  • ‘I’ - total intensity (in unit K-cmb)
  • ‘Q’ - Stokes Q (in unit K-cmb, IAU convention)
  • ‘U’ - Stokes U (in unit K-cmb, IAU convention)
  • ‘PI’ - polarisation intensity (in unit K-cmb)
  • ‘PA’ - polarisation angle (in unit rad, IAU convention)
Parameters:
  • data (numpy.ndarray) – 1D-array containing the HEALPix map
  • frequency (astropy.units.Quantity) – Frequency of the measurement (if relevant)
  • Nside (int, optional) – For extra internal consistency checking. If Nside is present, it will be checked whether \(12\times N_{side}^2\) matches data.size
  • type (str) – The type of map being supplied in data.
data

Data in ObservablesDict-compatible shape

key

Standard key associated with this observable

NAME = 'sync'
class imagine.observables.DispersionMeasureHEALPixDataset(data, error=None, cov=None, Nside=None)[source]

Bases: imagine.observables.dataset.HEALPixDataset

Stores a dispersion measure map into an IMAGINE-compatible dataset

Parameters:
  • data (numpy.ndarray) – 1D-array containing the HEALPix map
  • Nside (int, optional) – For extra internal consistency checking. If Nside is present, it will be checked whether \(12\times N_{side}^2\) matches
data

Data in ObservablesDict-compatible shape

key

Standard key associated with this observable

NAME = 'dm'
class imagine.observables.Observable(data=None, dtype=None, coords=None)[source]

Bases: object

Observable class is designed for storing/manipulating distributed information. For the testing suits, please turn to “imagine/tests/observable_tests.py”.

Parameters:
  • data (numpy.ndarray) – distributed/copied data
  • dtype (str) – Data type, must be either: ‘measured’, ‘simulated’ or ‘covariance’
append(new_data)[source]

appending new data happens only to SIMULATED dtype the new data to be appended should also be distributed which makes the appending operation naturally in parallel

rewrite flag will be switched off once rewriten has been performed

data

Data stored in the LOCAL processor (numpy.ndarray, read-only).

dtype

‘measured’, ‘simulated’ or ‘covariance’

Type:Data type, can be either
ensemble_mean
global_data

Data gathered from ALL processors (numpy.ndarray, read-only). Note that only master node hosts the global data, while slave nodes hosts None.

rw_flag

Rewriting flag, if true, append method will perform rewriting

shape

Shape of the GLOBAL array, i.e. considering all processors (numpy.ndarray, read-only).

size

Local data size (int, read-only) this size means the dimension of input data not the sample size of realizations

class imagine.observables.ObservableDict[source]

Bases: imagine.tools.class_tools.BaseClass

Base class from which imagine.observables.observable_dict.Measurements, imagine.observables.observable_dict.Covariances, imagine.observables.observable_dict.Simulations and imagine.observables.observable_dict.Masks are derived.

See imagine.observables.observable_dict module documentation for further details.

append(name, new_data, plain=False)[source]

Adds/updates name and data

Parameters:
  • name (str tuple) – Should follow the convention: (data-name,str(data-freq),str(data-Nside/size),str(ext)). If data is independent from frequency, set ‘nan’. ext can be ‘I’,’Q’,’U’,’PI’,’PA’, ‘nan’ or other customized tags.
  • new_data – distributed/copied numpy.ndarray or Observable
  • plain (bool) – If True, means unstructured data. If False (default case), means HEALPix-like sky map.
apply_mask(mask_dict)[source]
Parameters:mask_dict (imagine.observables.observable_dict.Masks) – Masks object
keys()[source]
archive
class imagine.observables.Masks[source]

Bases: imagine.observables.observable_dict.ObservableDict

Stores HEALPix mask maps

See imagine.observables.observable_dict module documentation for further details.

append(name, new_data, plain=False)[source]

Adds/updates name and data

Parameters:
  • name (str tuple) – Should follow the convention: (data-name,str(data-freq),str(data-Nside/"tab"),str(ext)). If data is independent from frequency, set ‘nan’. ext can be ‘I’,’Q’,’U’,’PI’,’PA’, ‘nan’ or other customized tags.
  • new_data – distributed/copied numpy.ndarray or Observable
  • plain (bool) – If True, means unstructured data. If False (default case), means HEALPix-like sky map.
apply_mask(*args, **kwargs)[source]
Parameters:mask_dict (imagine.observables.observable_dict.Masks) – Masks object
class imagine.observables.Measurements[source]

Bases: imagine.observables.observable_dict.ObservableDict

Stores observational data sets

See imagine.observables.observable_dict module documentation for further details.

append(name=None, new_data=None, plain=False, dataset=None)[source]

Adds/updates name and data

Parameters:
  • dataset (imagine.observables.dataset.Dataset) – The IMAGINE dataset already adjusts the format of the data and sets the adequate key. If dataset is present, all other arguments will be ignored.
  • name (str tuple) – Should follow the convention: (data-name,str(data-freq),str(data-Nside)/"tab",str(ext)). If data is independent from frequency, set ‘nan’. ext can be ‘I’,’Q’,’U’,’PI’,’PA’, ‘nan’ or other customized tags.
  • new_data (numpy.ndarray or imagine.observables.observable.Observable) – distributed/copied numpy.ndarray or Observable
  • plain (bool) – If True, means unstructured/tabular data. If False (default case), means HEALPix-like sky map.
apply_mask(mask_dict=None)[source]
Parameters:mask_dict (imagine.observables.observable_dict.Masks) – Masks object
class imagine.observables.Simulations[source]

Bases: imagine.observables.observable_dict.ObservableDict

Stores simulated ensemble sets

See imagine.observables.observable_dict module documentation for further details.

append(name, new_data, plain=False)[source]

Adds/updates name and data

Parameters:
  • name (str tuple) – Should follow the convention: (data-name,str(data-freq),str(data-Nside)/"tab",str(ext)). If data is independent from frequency, set ‘nan’. ext can be ‘I’,’Q’,’U’,’PI’,’PA’, ‘nan’ or other customized tags.
  • new_data – distributed/copied numpy.ndarray or Observable
  • plain (bool) – If True, means unstructured data. If False (default case), means HEALPix-like sky map.
apply_mask(mask_dict)[source]
Parameters:mask_dict (imagine.observables.observable_dict.Masks) – Masks object
class imagine.observables.Covariances[source]

Bases: imagine.observables.observable_dict.ObservableDict

Stores observational covariances

See imagine.observables.observable_dict module documentation for further details.

append(name=None, new_data=None, plain=False, dataset=None)[source]

Adds/updates name and data

Parameters:
  • name (str tuple) – Should follow the convention: (data-name,str(data-freq),str(data-Nside)/"tab",str(ext)). If data is independent from frequency, set ‘nan’. ext can be ‘I’,’Q’,’U’,’PI’,’PA’, ‘nan’ or other customized tags.
  • data – distributed/copied ndarray/Observable
  • plain (bool) – If True, means unstructured data. If False (default case), means HEALPix-like sky map.
apply_mask(mask_dict)[source]
Parameters:mask_dict (imagine.observables.observable_dict.Masks) – Masks object