[DeepSeek] Enable checkpoint load from HF #908
Conversation
facebook-github-bot
added
the
CLA Signed
| # This source code is licensed under the BSD-style license found in the | ||
| # LICENSE file in the root directory of this source tree. | ||
|
|
||
| # torchrun --standalone --nproc-per-node 4 run.py |
There was a problem hiding this comment.
is this file named 'run' instead of 'train' because it's not supposed to train? It looks like it is using a training oriented pipeline-schedule (not fwd-only) but missing a loss function etc. Maybe its just a temporary file for bootstrapping...
There was a problem hiding this comment.
It is a temp file for bootstrapping. And it will diverge into generate.py and train.py later.
| @@ -0,0 +1,183 @@ | |||
| # Copyright (c) Meta Platforms, Inc. and affiliates. | |||
There was a problem hiding this comment.
nit: model_args.py? since ModelArgs
There was a problem hiding this comment.
Hmm, I think both work. My purpose of creating this file is to store configs of different DeepSeek flavors, like the DeepSeek-V2-Lite in it.
|
|
||
| if pp_rank == pp_size - 1: | ||
| print(y.shape) | ||
|
|
There was a problem hiding this comment.
maybe a simple "Success - forward completed" or similar print just to show the forward completed properly?
Enable loading weights from HF checkpoint.
download.pyto allow user download a HF checkpoint into local disk cache.python download.py {model_id}checkpoint.pyto load tensors from HF cache dir into a model.ModelArgsinto a separate file, adding DeepSeek config registrytorchrun --standalone --nproc-per-node 4 run.py