some confusion on causal model

[xjtulien]

Thank you very much for sharing such wonderful work. I have read the paper and the code in the related repository in detail. I have some questions about the code related to the causal model. These questions are specifically reflected in the following code:
in train.py line 403 - 442:

         if self.model_type == "causal":
            # The InstanceSplitter pads time series on the left to be equal to the
            # context_length. However, certain models (e.g., GPT2) with absolute
            # position embeddings should not be trained with left padding.
            # The following piece of code moves padding from left to right.

            assert input_ids.shape[-1] == entry["past_is_pad"].shape[0]

            # Find the index where padding starts
            pad_start_idx = np.searchsorted(1 - entry["past_is_pad"], 1)
            padded_input_ids, obs_input_ids = torch.tensor_split(
                input_ids, [pad_start_idx], dim=-1
            )
            padded_attention_mask, obs_attention_mask = torch.tensor_split(
                attention_mask, [pad_start_idx], dim=-1
            )

            # Move padding to the right
            input_ids = torch.cat(
                [
                    obs_input_ids,
                    labels,
                    padded_input_ids,
                ],
                axis=-1,
            )
            attention_mask = torch.cat(
                [
                    obs_attention_mask,
                    labels_mask,
                    padded_attention_mask,
                ],
                axis=-1,
            )

            # labels for causal models are same as the input_ids.
            # Internally transformers shifts the labels by one during training.
            labels = input_ids.clone()
            input_ids[~attention_mask] = self.tokenizer.config.pad_token_id
            labels[~attention_mask] = -100
            labels = input_ids.clone()

Why not concatenate future_target with input_ids and use them together as labels?
I would really appreciate it if you could find time to reply to my question.

[abdulfatir]

@xjtulien Your question has been answered in the comment in the code.

# The InstanceSplitter pads time series on the left to be equal to the
# context_length. However, certain models (e.g., GPT2) with absolute
# position embeddings should not be trained with left padding.
# The following piece of code moves padding from left to right.

[xjtulien]

It might be that I didn't describe it clearly or that I still don't understand. What I'm wondering is why the final labels = input_ids.clone() doesn't concatenate with future_target? It seems like future_target is not being used.

[abdulfatir]

labels is the scaled version of future_target. See:

chronos-forecasting/scripts/training/train.py

Line 400 in 39515ff

labels, labels_mask = self.tokenizer.label_input_transform(future_target, scale)