Skip to content

Fail tasks stuck in PENDING#4983

Open
sethboyles wants to merge 1 commit intomainfrom
fail_pending_tasks
Open

Fail tasks stuck in PENDING#4983
sethboyles wants to merge 1 commit intomainfrom
fail_pending_tasks

Conversation

@sethboyles
Copy link
Copy Markdown
Member

@sethboyles sethboyles commented Mar 30, 2026

  • A short explanation of the proposed change:

This PR started counting PENDING tasks towards quotas: #4651

This has revealed a number of tasks stuck in PENDING that are incorrectly inflating orgs' memory usage.

I haven't traced the cause for all of the stuck tasks yet, but I believe it probably happened when trying to save the state right after a Diego submission--I think it could have happened if there was a DB availability issue, for example. Regardless, these PENDING tasks will never get updated if they hit this state.

This PR transitions them to FAILED after a short period.

  • I have reviewed the contributing guide

  • I have viewed, signed, and submitted the Contributor License Agreement

  • I have made this pull request to the main branch

  • I have run all the unit tests using bundle exec rake

  • I have run CF Acceptance Tests

module Diego
class TasksSync
BATCH_SIZE = 500
PENDING_TASK_EXPIRATION = 300
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add a unit to the constant name?
Also does it make sense to make this configurable with a default of 300s?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think there is a big benefit to making this configurable.

@sethboyles sethboyles force-pushed the fail_pending_tasks branch from 97adc6b to 9965116 Compare April 1, 2026 17:15
@sethboyles sethboyles requested a review from johha April 1, 2026 17:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants