straxen.storage package

Submodules

straxen.storage.online_monitor_frontend module

class straxen.storage.online_monitor_frontend.OnlineMonitor(uri=None, take_only=None, database=None, col_name='online_monitor', readonly=True, *args, **kwargs)[source]

Bases: MongoFrontend

Online monitor Frontend for Saving data temporarily to the database.

straxen.storage.online_monitor_frontend.get_mongo_uri(user_key='pymongo_user', pwd_key='pymongo_password', url_key='pymongo_url', header='RunDB')[source]

straxen.storage.rucio_local module

class straxen.storage.rucio_local.RucioLocalBackend(rucio_dir, *args, **kwargs)[source]

Bases: FileSytemBackend

Get data from local rucio RSE.

class straxen.storage.rucio_local.RucioLocalFrontend(path=None, *args, **kwargs)[source]

Bases: StorageFrontend

Storage that loads from rucio by assuming the rucio file naming convention without access to the rucio database.

Normally, you don’t need this StorageFrontend as it should return the same data as the RunDB frontend

determine_rse()[source]
did_is_local(did)[source]

Determines whether or not a given did is on a local RSE. If there is no local RSE, returns False.

Parameters:

did – Rucio DID string

Returns:

boolean for whether DID is local or not.

local_prefixes = {'SDSC_USERDISK': '/expanse/lustre/projects/chi135/shockley/rucio/', 'UC_DALI_USERDISK': '/dali/lgrandi/rucio/'}
local_rses = {'SDSC_USERDISK': '.sdsc.', 'UC_DALI_USERDISK': '^dali(\\w*).rcc.'}
storage_type = 1

straxen.storage.rucio_remote module

class straxen.storage.rucio_remote.RucioRemoteBackend(staging_dir, rses_only=(), download_heavy=False, remove_heavy=False, tries=3, num_threads=1, stage=False, **kwargs)[source]

Bases: FileSytemBackend

Get data from remote Rucio RSE.

dset_cache: Dict[str, str] = {}
heavy_types = ['raw_records', 'raw_records_nv', 'raw_records_he']
class straxen.storage.rucio_remote.RucioRemoteFrontend(staging_dir='./strax_data', rses_only=(), download_heavy=False, remove_heavy=False, tries=3, num_threads=1, stage=False, *args, **kwargs)[source]

Bases: StorageFrontend

Uses the rucio client for the data find.

find(key: DataKey, write=False, check_broken=False, **kwargs)[source]

Return (str: backend class name, backend-specific) key to get at / write data, or raise exception.

Parameters:

  • key – DataKey of data to load {data_type: (plugin_name, version, {config_option: value, …}, …}

  • write – Set to True if writing new data. The data is immediately registered, so you must follow up on the write!

  • check_broken – If True, raise DataNotAvailable if data has not been complete written, or writing terminated with an exception.

find_several(keys, **kwargs)[source]

Return list with backend keys or False for several data keys.

Options are as for find()

local_did_cache = None
path = None
storage_type = 4
class straxen.storage.rucio_remote.RucioSaver(*args, **kwargs)[source]

Bases: Saver

TODO: Saves data to rucio if you are the production user.

exception straxen.storage.rucio_remote.TooMuchDataError[source]

Bases: Exception

straxen.storage.rucio_remote.key_to_rucio_did(key: DataKey) → str[source]

Convert a strax.datakey to a rucio did field in rundoc.

straxen.storage.rucio_remote.parse_rucio_did(did: str) → tuple[source]

Parses a Rucio DID and returns a tuple of (number:int, dtype:str, hash:str)

straxen.storage.rundb module

class straxen.storage.rundb.RunDB(minimum_run_number=7157, maximum_run_number=None, runid_field='name', local_only=False, new_data_path=None, reader_ini_name_is_mode=False, rucio_path=None, mongo_url=None, mongo_user=None, mongo_password=None, mongo_database=None, *args, **kwargs)[source]

Bases: StorageFrontend

Frontend that searches RunDB MongoDB for data.

find_several(keys: List[DataKey], **kwargs)[source]

Return list with backend keys or False for several data keys.

Options are as for find()

hosts = {'dali': '^dali.*rcc.*|fried.rice.edu', 'midway': '^midway2.*rcc.*|^midway.*rcc.*'}
number_query()[source]
progress_bar = False
provide_run_metadata = True
run_metadata(run_id, projection=None)[source]

Return run metadata dictionary, or raise RunMetadataNotAvailable.

storage_type = 1
userdisks = {'UC_DALI_USERDISK': '^dali.*rcc.*|fried.rice.edu', 'UC_MIDWAY_USERDISK': '^midway2.*rcc.*|^midway.*rcc.*'}

Module contents