Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tensorboard_profiling_keras.ipynb fails even in colab: logs -> $logs, No step marker found #6954

Open
seanv507 opened this issue Nov 24, 2024 · 0 comments

Comments

@seanv507
Copy link

Environment information (required)

colab
/~https://github.com/tensorflow/tensorboard/blob/master/docs/tensorboard_profiling_keras.ipynb

Issue description

tensorboard profiler will only find profile information if pointed to the exact subdirectory where log is written ( ie single run)

therefore code needs to be changed to %tensorboard --logdir=$logs from %tensorboard --logdir=logs (to include the timestamp ie logs/ )

when this is done, profile tab is visible (and can select profile in the dropdown inactive/...

profile information is now visible, but on the overview page it warns

No step marker observed and hence the step time is unknown. This may happen if (1) training steps are not instrumented (e.g., if you are not using Keras) or (2) the profiling duration is shorter than the step time. For (1), you need to add step instrumentation; for (2), you may try to profile longer.

all timings are reported as zero on overview page
Screenshot 2024-11-24 at 10 58 09

@pritamdodeja
Copy link

This bug is caused by tensorflow.python.profiler.internal._pywrap_profiler.ProfilerSession().export_to_tb() putting profiler data in the wrong place. This bug was introduced into tensorflow/tensorflow/python/profiler/internal/profiler_wrapper.cc a long time ago (around v2.12 release). The invocation happens in tensorflow/python/profiler/profiler_v2.py around line 150 in tensorflow v2.15.1. Please fix this so that profiler starts working again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants