Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the ability to trigger block reports on simulated DataNodes #37

Closed
xkrogen opened this issue Apr 23, 2018 · 1 comment
Closed

Add the ability to trigger block reports on simulated DataNodes #37

xkrogen opened this issue Apr 23, 2018 · 1 comment

Comments

@xkrogen
Copy link
Collaborator

xkrogen commented Apr 23, 2018

Due to the way blocks are injected into the simulated DataNodes in Dynamometer, sometimes it is possible for the DataNode to send its initial block report completely empty. This is because the blocks are injected after the DataNode is started, due to the setup of a MiniDFSCluster. If this happens, another block report will not be sent until the block report interval has passed, which can be a very long time. This can result in (a) test timeouts (b) long setup times.

We can add the ability for the ApplicationMaster to monitor which nodes have not reported full block reports, and trigger block reports on those nodes.

xkrogen added a commit to xkrogen/dynamometer that referenced this issue Apr 24, 2018
…taNodes that haven't reported everything yet.
@xkrogen
Copy link
Collaborator Author

xkrogen commented Apr 27, 2018

Example test failure: https://travis-ci.org/linkedin/dynamometer/jobs/372123344

xkrogen added a commit to xkrogen/dynamometer that referenced this issue May 3, 2018
…taNodes that haven't reported everything yet.
@xkrogen xkrogen closed this as completed in ea76696 May 4, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant