es_sfgtools.workflows.midprocess.mid_processing module

This module defines the IntermediateDataProcessor class, which is responsible for post-processing of data.

class es_sfgtools.workflows.midprocess.mid_processing.IntermediateDataProcessor(station_metadata: Site, directory_handler: DirectoryHandler)

Bases: WorkflowABC

A class to handle post-processing of data.

mid_process_workflow: bool = True
midprocess_sync_s3(overwrite: bool = False)

Uploads the current station directory to S3 for synchronization.

SFGMain/cascadia-gorda/NCC1/2025_A_1126 –> s3://<bucket_name>/cascadia-gorda/NCC1/2025_A_1126

parse_surveys(survey_id: str | None = None, override: bool = False, write_intermediate: bool = False)

Parses the surveys from the current campaign and adds them to the directory structure.

Parameters:

  • survey_id (Optional[str], optional) – The ID of the survey to parse. If None, all surveys are parsed, by default None.

  • override (bool, optional) – Whether to override existing files, by default False.

  • write_intermediate (bool, optional) – Whether to write intermediate files, by default False.

prepare_shotdata_garpos(campaign_id: str | None = None, survey_id: str | None = None, custom_filters: dict | None = None, overwrite: bool = False) → None

Prepares shotdata for GARPOS processing.

Parameters:

  • campaign_id (Optional[str], optional) – The ID of the campaign, by default None.

  • survey_id (Optional[str], optional) – The ID of the survey, by default None.

  • custom_filters (Optional[dict], optional) – Custom filters to apply, by default None.

  • overwrite (bool, optional) – Whether to overwrite existing files, by default False.

prepare_single_garpos_survey(survey: Survey, custom_filters: dict | None = None, overwrite: bool = False)

Prepares a single survey for GARPOS processing.

Parameters:

  • survey (Survey) – The survey metadata.

  • custom_filters (dict, optional) – Custom filters to apply, by default None.

  • overwrite (bool, optional) – Whether to overwrite existing files, by default False.