[Collector] Automation Potential

[katharinabassler]

This issue serves as a centralized collector for all ideas and proposals related to automating processes or workflows in all prosEO-related activities. The goal is to identify repetitive, manual, or error-prone tasks that could benefit from automation. Please take into account your experiences and existing routines with other processing software, especially PSM.

Scope:

• Repetitive manual tasks.
• Error-prone processes that could be standardised or validated automatically.
• Integration gaps between tools or systems.

How to Contribute:
Add a comment below with your automation idea, with the following points as applicable:

• Title: [Brief, descriptive title]
• Area: [e.g., Data Processing, Testing, Deployment, Monitoring]
• Current Process: [Describe the manual/existing process]
• Proposed Automation: [What should be automated and how?]
• Impact: [Expected benefits: time saved, error reduction, scalability, etc.]

Prioritisation:
All suggestions will be reviewed and prioritised based on impact, feasibility, and alignment with prosEO goals. Separate issues will be created and assigned as needed.

[katharinabassler]

Update on suggestion below:

Spring Boot and Grafana are now leveraged for basic monitoring. Further monitoring was decided to be delegated to external providers. Requirement specification meetings were held, candidates were proposed.

---

Title: Centralised Monitoring with Spring Boot Actuator + Grafana

Area: Monitoring

Current Process:
Currently, monitoring of prosEO services is somewhat fragmented. Developers and operators rely on:

• GUI monitoring
• CLI scripts
• Kubernetes dashboard
• Cloud dashboards

This makes it difficult to proactively detect issues, compare system behaviour over time, or quickly diagnose incidents.

Proposed Automation:
Introduce a standardised monitoring stack using Spring Boot Actuator, Prometheus, and Grafana:

• Enable Spring Boot Actuator metrics endpoints across all prosEO services
• Expose metrics in Prometheus format via /actuator/prometheus
• Automatically collect metrics using a central Prometheus instance
• Create shared Grafana dashboards and alerting rules according to operations needs and ESA requirements.

Impact:

• Early detection of operational issues
• Reduced manual effort for system checks and data aggregation
• Faster incident diagnosis and troubleshooting
• Standardised monitoring across all services
• Improved reliability and scalability of prosEO operations

Prioritisation Notes:
The implementation complexity is relatively low since Spring Boot Actuator is already enabled. The main effort lies in collecting requirements, to a lesser extent in setting up shared dashboards and alert rules.

[katharinabassler]

Suggestion below dismissed as wontfix.

Currently, a single validation environment is used to test new prosEO versions as well as specific solutions for different production environments. Changes are deployed as needed by the different projects or by development milestones, rather than always testing the newest code.

Introducing CI/CD would not hold significant benefits, as the manual deployment process is easy and fast. Several of the gains stated below such as easier joint troubleshooting and earlier detection of integration issues depend on the common use of the validation environment, not on the deployment process.

---

Title: Automated CI/CD Pipeline for Deployment to Validation Environment

Area: Testing

Current Process:
Deployment to the validation environment is currently a manual process. It typically involves:

• Building projects and images locally
• Manually updating configurations or versions
• Manually triggering deployments
• Limited traceability of which code version is deployed
• Risk of inconsistent deployment steps between services

Proposed Automation:
Introduce a standardised CI/CD pipeline that automatically builds, tests, and deploys services to the validation environment:

• Trigger pipeline on merge to dedicated branch
• Automatically build and version Docker images
• Run automated unit and integration tests
• Deploy automatically to the validation environment using infrastructure-as-code or deployment scripts (compatible with current Ansible format)
• Record deployment metadata (version, commit, timestamp)
• Optionally include approval gates before deployment

Impact:

• Faster and easier system testing with reduced manual effort
• Improved traceability and reproducibility for easier joint troubleshooting
• Earlier detection of integration issues or issues limited to distributed deployments
• Preparation for possible future CI/CD or GitOps pipelines based on the master branch

[Axel-Schmidt]

Title: Automatic JIRA ticket creation from Email requests

Area: Daily Operations

Current Process:

• Manual Email scanning; checking for existing JIRA ticket
• Manual creation of JIRA ticket
• At least double work filling JIRA issue with relevant information and responding to email (+ in case update CAMS and Redmine)

Proposed Automation:

• Usage of benefits from JIRA Cloud (role model: TerraByte)

Impact:

• Missing external email requests in the flooded email account improves
• Reduced tracking overhead
• No super-large time saving, but every little helps