Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for detecting tasks in a "paused" state #214

Open
atc0005 opened this issue Apr 29, 2024 · 1 comment
Open

Add support for detecting tasks in a "paused" state #214

atc0005 opened this issue Apr 29, 2024 · 1 comment
Assignees
Labels
enhancement New feature or request plugin/sync plans
Milestone

Comments

@atc0005
Copy link
Owner

atc0005 commented Apr 29, 2024

Ran into a situation where connections to the host (self-referencing) were refused and the affected task went into a "paused" state. Looking at the specific task I see the option to "Resume" the task.

E.g.,

image

Error text:

Failed to open TCP connection to rsat.example.com:443 (Connection refused - connect(2) for "rsat.example.com" port 443)

Ideally support would be available to detect tasks stuck in this state and surface them.

Even with these stuck tasks, scheduled sync plans continue to "run" therefore not triggering the existing check_rsat_sync_plans "there is a problem" logic.

Future note to self

// IsStuck indicates whether (after any applied grace time) the sync plan is
// considered to be in a "stuck" state (Next Sync state set to past date/time).
//
// Grace time is applied to help prevent flagging a sync plan that is
// "spinning up" or in a temporary pending status (e.g., on a busy system) as
// problematic.
//
// NOTE: Very busy systems keeping sync plans in a pending state for an
// extended duration are still likely to be flagged as non-OK by current
// logic.
func (sp SyncPlan) IsStuck() bool {
now := time.Now().UTC()
nextSync := time.Time(sp.NextSync).UTC()
switch {
case sp.Enabled && nextSync.Before(now):
diff := now.Sub(nextSync).Minutes()
if diff <= syncTimeGraceMinutes {
return false
}
return true
default:
return false
}
}

@atc0005 atc0005 added enhancement New feature or request plugin/sync plans labels Apr 29, 2024
@atc0005 atc0005 added this to the Future milestone Apr 29, 2024
@atc0005 atc0005 self-assigned this Apr 29, 2024
@atc0005
Copy link
Owner Author

atc0005 commented May 20, 2024

https://rsat.example.com/foreman_tasks/api/tasks?search=state=paused

Example "pretty" payload:

{
  "total": 657825,
  "subtotal": 0,
  "page": 1,
  "per_page": 20,
  "sort": {
    "by": "started_at",
    "order": "DESC"
  },
  "results": []
}

The bit that we're specifically focused on:

"subtotal": 0,

@atc0005 atc0005 pinned this issue May 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request plugin/sync plans
Projects
None yet
Development

No branches or pull requests

1 participant