FRA1 event storage cluster performance issues impacting search jobs

Incident Report for SEKOIA FRA1

Resolved

This incident has been resolved.
Posted Jun 13, 2025 - 21:06 UTC

Update

The performance of events research are back to normal. We are continuing to monitor the situation closely to ensure that this positive trend continues. We appreciate your patience during this time and will continue to provide updates.
Posted Jun 13, 2025 - 15:09 UTC

Update

We have identified that the performance issues with our event storage cluster in the FRA1 region were primarily due to a high volume of heavy specific search operations. We have now stopped these operations, and we have cancelled the remaining running tasks. We are already observing a significant improvement in system performance and search job duration. We are continuing to monitor the situation closely to ensure that this positive trend continues. We appreciate your patience during this time and will continue to provide updates.
Posted Jun 13, 2025 - 14:59 UTC

Update

We are still experiencing performance issues with our event storage cluster in the FRA1 region today. We have identified a high volume of specific search operations that are contributing to the load on the system. We are currently exploring options to manage these operations more efficiently to alleviate the load. We appreciate your continued patience and will provide further updates as we continue to work towards a resolution.
Posted Jun 13, 2025 - 12:05 UTC

Monitoring

Our team has successfully applied measures to reduce the load on the newly added nodes in our event storage cluster in the FRA1 region. However, we are still observing slower than normal search jobs. Our team continues to monitor the situation closely and further adjustments will be made as necessary. We apologize for any inconvenience caused and appreciate your patience as we work towards a resolution.
Posted Jun 12, 2025 - 16:08 UTC

Update

We have identified that the performance issues with our event storage cluster in the FRA1 region are likely due to high load on newly added nodes. These nodes are currently handling the majority of new traffic and most recent queries, resulting in slower search jobs. Our team is discussing options to alleviate the traffic to these nodes. We appreciate your patience as we work to resolve this issue.
Posted Jun 12, 2025 - 16:03 UTC

Identified

We are currently experiencing performance issues with our event storage cluster in the FRA1 region. This is impacting search jobs, resulting in slower event research. Our engineers are investigating this issue and we are working to restore normal functionality as soon as possible.
Posted Jun 12, 2025 - 16:03 UTC
This incident affected: Event storage.