radis.tools.database module

Implements a spectrum database SpecDatabase class to manage them all.

It basically manages a list of Spectrum JSON files, adding a Pandas dataframe structure on top to serve as an efficient index to visualize the spectra input conditions, and slice through the Dataframe with easy queries

Examples

See and get objects from database:

from radis.tools import SpecDatabase
db = SpecDatabase(r"path/to/database")     # create or loads database

db.update()  # in case something changed (like a file was added manually)
db.see(['Tvib', 'Trot'])   # nice print in console

s = db.get('Tvib==3000 & Trot==1500')[0]  # get all spectra that fit conditions
db.add(s)  # update database (and raise error because duplicate!)

Note that SpectrumFactory can be configured to automatically look-up and update a database when spectra are calculated.

An example of script to update all spectra conditions in a database (ex: when a condition was added afterwards to the Spectrum class):

# Example: add the 'medium' key in conditions
db = "database_CO"
for f in os.listdir(db):
   if not f.endswith('.spec'): continue
   s = load_spec(join(db, f))
   s.conditions['medium'] = 'vacuum'
   s.store(join(db,f), if_exists_then='replace')

You can see more examples on the Spectrum Database section of the website.


class SpecDatabase(path='.', filt='.spec', add_info=None, add_date='%Y%m%d', verbose=True, binary=True, nJobs=- 2, batch_size='auto', lazy_loading=True)[source]

Bases: radis.tools.database.SpecList

add(spectrum, store_name=None, if_exists_then='increment', **kwargs)[source]

Add Spectrum to database, whether it’s a Spectrum object or a file that stores one. Check it’s not in database already.

Parameters

spectrum (Spectrum object, or path to a .spec file (str)) – if a Spectrum object: stores it in the database (using the store() method), then adds the file to the database folder. if a path to a file (str): first copy the file to the database folder, then loads the copied file to the database.

Other Parameters
  • store_name (str, or None) – name of the file where the spectrum will be stored. If None, name is generated automatically from the Spectrum conditions (see add_info= and if_exists_then=)

  • if_exists_then ('increment', 'replace', 'error', 'ignore') – what to do if file already exists. If 'increment' an incremental digit is added. If 'replace' file is replaced (!). If 'ignore' the Spectrum is not added to the database and no file is created. If 'error' (or anything else) an error is raised. Default 'increment'.

  • **kwargs (**dict) – extra parameters used in the case where spectrum is a file and a .spec object has to be created (useless if spectrum is a file already). kwargs are forwarded to Spectrum.store() method. See the store() method for more information.

  • *Other :py:meth:`~radis.spectrum.spectrum.Spectrum.store` parameters can be given

  • as kwargs arguments (***)

  • compress (0, 1, 2) – if True or 1, save the spectrum in a compressed form

    if 2, removes all quantities that can be regenerated with update(), e.g, transmittance if abscoeff and path length are given, radiance if emisscoeff and abscoeff are given in non-optically thin case, etc. If not given, use the value of SpecDatabase.binary The performances are usually better if compress = 2. See https://github.com/radis/radis/issues/84.

  • add_info (list) – append these parameters and their values if they are in conditions example:

    nameafter = ['Tvib', 'Trot']
    
  • discard (list of str) – parameters to exclude. To save some memory for instance Default ['lines', 'populations']: retrieved Spectrum will loose the line_survey() and plot_populations() methods (but it saves a ton of memory!).

Examples

from radis.tools import SpecDatabase
db = SpecDatabase(r"path/to/database")     # create or loads database
db.add(s, discard=['populations'])

You can see more examples on the Spectrum Database section of the website.

See also

get(), get_unique(), get_closest(), update()

compress_to(new_folder, compress=True)[source]

Saves the Database in a new folder with all Spectrum objects under compressed (binary) format. Read/write is much faster. After the operation, a new database should be initialized in the new_folder to access the new Spectrum.

Parameters
  • new_folder (str) – folder where to store the compressed SpecDatabase. If doesn’t exist, it is created.

  • compress (boolean, or 2) – if True, saves under binary format. Faster and takes less space. If 2, additionaly remove all redundant quantities.

find_duplicates(columns=None)[source]

Find spectra with same conditions. The first duplicated spectrum will be 'False', the following will be 'True' (see .duplicated()).

Parameters

columns (list, or None) – columns to find duplicates on. If None, use all conditions.

Examples

>>> db.find_duplicates(columns={'x_e', 'x_N_II'})

Out[34]: file 20180710_101.spec True 20180710_103.spec True dtype: bool

You can see more examples on the Spectrum Database section of the website.

fit_spectrum(s_exp, residual=None, normalize=False, normalize_how='max', conditions='', **kwconditions)[source]

Returns the Spectrum in the database that has the lowest residual with s_exp.

Parameters

s_exp (Spectrum) – Spectrum to fit (typically: experimental spectrum)

Other Parameters
  • residual (func, or None) – which residual function to use. If None, use get_residual() with option ignore_nan=True and options normalize and normalize_how as defined by the user.

    get_residual should have the form:

    lambda s_exp, s, normalize: func(s_exp, s, normalize=normalize)
    

    where the output is a float. Default None

  • conditions, **kwconditions (str, **dict) – restrain fitting to only Spectrum that match the given conditions in the database. See get() for more information.

  • normalize (bool, or Tuple) – see get_residual()

  • normalize_how (‘max’, ‘area’) – see get_residual()

Returns

s_best – closest Spectrum to s_exp

Return type

Spectrum

Examples

Using a customized residual function (below: to get the transmittance):

from radis import get_residual
db = SpecDatabase('...')
db.fit_spectrum(s_exp, get_residual=lambda s_exp, s: get_residual(s_exp, s, var='transmittance'))

You can see more examples on the Spectrum Database section of the website.

print_index(file=None)[source]
to_dict()[source]

Returns all Spectra in database under a dictionary, indexed by file.

Returns

out – {path : Spectrum object} dictionary

Return type

dict

Note

SpecList.items().values() is equivalent to SpecList.get()

update(force_reload=False, filt='.spec')[source]

Reloads database, updates internal index structure and export it in <database>.csv.

Parameters
  • force_reload (boolean) – if True, reloads files already in database. Default False

  • filt (str) – only consider files ending with filt. Default .spec

Notes

Can be loaded in parallel using joblib by setting the nJobs and batch_size attributes of SpecDatabase. See joblib.parallel.Parallel for information on the arguments

update_conditions()[source]

Reloads conditions of all Spectrum in database.

class SpecList(*spectra, **kwargs)[source]

Bases: object

conditions()[source]

Show conditions in database.

get(conditions='', **kwconditions)[source]

Returns a list of spectra that match given conditions.

Parameters
  • database (list of Spectrum objects) – the database

  • conditions (str) –

    a list of conditions. Example:

    db.get('Tvib==3000 & Trot==1500')
    
  • kwconditions (dict) –

    an unfolded dict of conditions. Example:

    db.get(Tvib=3000, Trot=1500)
    
Other Parameters
  • inplace (bool) – if True, return the actual object in the database. Else, return copies. Default False

  • verbose (bool) – more blabla

  • scale_if_possible (bool) – if True, spectrum is scaled for parameters that can be computed directly from spectroscopic quantities (e.g: 'path_length', 'molar_fraction'). Default False

Returns

out

Return type

list of Spectrum

Examples

spec_list = db.get('Tvib==3000 & Trot==1300')

or:

spec_list = db.get(Tvib=3000, Trot=1300)
get_closest(scale_if_possible=True, **kwconditions)[source]

Returns the Spectra in the database that is the closest to the input conditions.

Note that for non-numeric values only equals should be given. To calculate the distance all numeric values are scaled by their mean value in the database

Parameters
  • kwconditions (named arguments) – i.e: Tgas=300, path_length=1.5

  • scale_if_possible (boolean) – if True, spectrum is scaled for parameters that can be computed directly from spectroscopic quantities (e.g: 'path_length', 'molar_fraction'). Default True

Other Parameters
  • verbose (boolean) – print messages. Default True

  • inplace (boolean) – if True, returns the actual object in database. Else, return a copy. Default False

See also

get(), get_unique()

get_items(condition)[source]

Returns all Spectra in database under a dictionary; indexed by condition

Requires that condition is unique

Parameters

condition (str) – condition. Ex: Trot

Returns

out – {condition:Spectrum}

Return type

dict

See also

to_dict(), get()

get_unique(conditions='', scale_if_possible=False, **kwconditions)[source]

Returns a spectrum that match given conditions.

Raises an error if the spectrum is not unique.

Parameters

args – see get() for more details

Returns

s

Return type

Spectrum

See also

get(), get_closest()

items()[source]

Iterate over all Spectrum in database.

Examples

Print name of all Spectra in dictionary:

db = SpecDatabase('.')
for path, s in db.items():
    print(path, s.name)

Update all spectra in current folder with a new condition (‘author’):

db = SpecDatabase('.')
for path, s in db.items():
    s.conditions['author'] = 'me'
    s.store(path, if_exists_then='replace')

See also

keys(), values(), to_dict()

keys()[source]

Iterate over all {path} in database.

See also

values(), items(), to_dict()

map(function)[source]

Apply function to all Spectra in database.

Examples

Add a missing parameter:

db = SpecDatabase('...')

def add_condition(s):
    s.conditions['exp_run'] = 1
    return s

db.map(add_condition)
plot(nfig=None, legend=True, **kwargs)[source]

Plot all spectra in database.

Parameters

nfig (str, or int, or None) – figure to plot on. Default None: creates one

Other Parameters
  • kwargs (dict) – parameters forwarded to the Spectrum plot() method

  • legend (bool) – if True, plot legend.

Returns

fig, ax – figure

Return type

matplotlib figure and ax

Examples

Plot all spectra in a folder:

db = SpecDatabase('my_folder')
db.plot(wunit='nm')

See also

Spectrum

py:meth:plot method

plot_cond(cond_x, cond_y, z_value=None, nfig=None)[source]

Plot database conditions available:

Parameters
  • cond_x, cond_y (str) – columns (conditions) of database.

  • z_value (array, or None) – if not None, colors the 2D map with z_value. z_value is ordered so that z_value[i] corresponds to row[i] in database.

Examples

::
>>> db.plot(Tvib, Trot)     # plot all points calculated
>>> db.plot(Tvib, Trot, residual)     # where residual is calculated by a fitting
                                      # procedure...
see(columns=None, *args)[source]

Shows Spectrum database with all conditions (columns=None) or specific conditions.

Parameters

columns (str, list of str, or None) – shows the conditions value for all cases in database. If None, all conditions are shown. Default None e.g.:

db.see(['Tvib', 'Trot'])

Notes

Makes the ‘file’ column the index, and also discard the ‘Spectrum’ column (that holds all the data) for readibility

values()[source]

Iterate over all {Spectrum} in database.

See also

keys(), items(), to_dict()

view(columns=None, *args)[source]

alias of see()

See also

see()

in_database(smatch, db='.', filt='.spec')[source]

Old function.

is_jsonable(x)[source]
load_spec(file, binary=True)[source]

Loads a .spec file into a Spectrum object. Adds file in the Spectrum file attribute.

Parameters
  • file (str) – .spec file to load

  • binary (boolean) – set to True if the file is encoded as binary. Default True. Will autodetect if it fails, but that may take longer.

Returns

s – a Spectrum object

Return type

Spectrum

Examples

See also

SpecDatabase, store()

plot_spec(file, what='radiance', title=True, **kwargs)[source]

Plot a .spec file. Uses the plot() method internally.

Parameters

file (str, or Spectrum object) – .spec file to load, or Spectrum object directly

Other Parameters

kwargs (dict) – arguments forwarded to plot()

Returns

fig – where the Spectrum has been plotted

Return type

matplotlib figure

See also

plot()

read_conditions_file(path)[source]

Read .csv file with calculation/measurement conditions of all spectra.

File must have at least the column “file”

Parameters

path (csv file) – summary of all spectra conditions.

Returns

Return type

None.

save(s, path, discard=[], compress=True, add_info=None, add_date=None, if_exists_then='increment', verbose=True, warnings=True)[source]

Save a Spectrum object in JSON format. Object can be recovered with load_spec(). If many Spectrum are saved in a same folder you can view their properties with the SpecDatabase structure.

Parameters
  • s (Spectrum) – to save

  • path (str) – filename to save. No extension needed. If filename already exists then a digit is added. If filename is a directory then a new file is created within this directory.

  • discard (list of str) – parameters to discard. To save some memory.

  • compress (boolean) – if False, save under text format, readable with any editor. if True, saves under binary format. Faster and takes less space. If 2, removes all quantities that can be regenerated with s.update(), e.g, transmittance if abscoeff and path length are given, radiance if emisscoeff and abscoeff are given in non-optically thin case, etc. Default False

  • add_info (list, or None/False) – append these parameters and their values if they are in conditions. e.g:

    add_info = ['Tvib', 'Trot']
    
  • add_date (str, or None/False) – adds date in strftime format to the beginning of the filename. e.g:

    add_date = '%Y%m%d'
    
  • if_exists_then ('increment', 'replace', 'error', 'ignore') – what to do if file already exists. If 'increment' an incremental digit is added. If 'replace' file is replaced (!). If 'ignore' the Spectrum is not added to the database and no file is created. If 'error' (or anything else) an error is raised. Default 'increment'.

Returns

fout – filename used (may be different from given path as new info or incremental identifiers are added)

Return type

str