Fix bugs in Ghost BN in TabNet and typos in README

README.md

@@ -46,11 +46,11 @@ For more details, please refer to our paper ["Qlib: An AI-oriented Quantitative
 </div>
 
 
-At the module level, Qlib is a platform that consists of the above components. The components are designed as loose-coupled modules and each component could be used stand-alone.
+At the module level, Qlib is a platform that consists of the above components. The components are designed as loose-coupled modules, and each component could be used stand-alone.
 
 | Name                   | Description                                                                                                                                                                                                                                                                                                                                                             |
 | ------                 | -----                                                                                                                                                                                                                                                                                                                                                                   |
-| `Infrastructure` layer | `Infrastructure` layer provides underlying support for Quant research. `DataServer` provides high-performance infrastructure for users to manage and retrieve raw data. `Trainer` provides flexible interface to control the training process of models which enable algorithms controlling the training process.                                                       |
+| `Infrastructure` layer | `Infrastructure` layer provides underlying support for Quant research. `DataServer` provides a high-performance infrastructure for users to manage and retrieve raw data. `Trainer` provides a flexible interface to control the training process of models, which enable algorithms to control the training process.                                                       |
 | `Workflow` layer       | `Workflow` layer covers the whole workflow of quantitative investment. `Information Extractor` extracts data for models. `Forecast Model` focuses on producing all kinds of forecast signals (e.g. _alpha_, risk) for other modules. With these signals `Portfolio Generator` will generate the target portfolio and produce orders to be executed by `Order Executor`. |
 | `Interface` layer      | `Interface` layer tries to present a user-friendly interface for the underlying system. `Analyser` module will provide users detailed analysis reports of forecasting signals, portfolios and execution results                                                                                                                                                                 |
 
@@ -130,7 +130,8 @@ This dataset is created by public data collected by [crawler scripts](scripts/da
 the same repository.
 Users could create the same dataset with it. 
 
-*Please pay **ATTENTION** that the data is collected from [Yahoo Finance](https://finance.yahoo.com/lookup) and the data might not be perfect. We recommend users to prepare their own data if they have high-quality dataset. For more information, users can refer to the [related document](https://qlib.readthedocs.io/en/latest/component/data.html#converting-csv-format-into-qlib-format)*.
+*Please pay **ATTENTION** that the data is collected from [Yahoo Finance](https://finance.yahoo.com/lookup), and the data might not be perfect.
+We recommend users to prepare their own data if they have a high-quality dataset. For more information, users can refer to the [related document](https://qlib.readthedocs.io/en/latest/component/data.html#converting-csv-format-into-qlib-format)*.
 
 <!-- 
 - Run the initialization code and get stock data:
@@ -220,7 +221,7 @@ Qlib provides a tool named `qrun` to run the whole workflow automatically (inclu
       -->
 
 ## Building Customized Quant Research Workflow by Code
-The automatic workflow may not suite the research workflow of all Quant researchers. To support a flexible Quant research workflow, Qlib also provides a modularized interface to allow researchers to build their own workflow by code. [Here](examples/workflow_by_code.ipynb) is a demo for customized Quant research workflow by code.
+The automatic workflow may not suit the research workflow of all Quant researchers. To support a flexible Quant research workflow, Qlib also provides a modularized interface to allow researchers to build their own workflow by code. [Here](examples/workflow_by_code.ipynb) is a demo for customized Quant research workflow by code.
 
 
 # [Quant Model Zoo](examples/benchmarks)
@@ -313,7 +314,7 @@ which creates a dataset (14 features/factors) from the basic OHLCV daily data of
 * `+(-)E` indicates with (out) `ExpressionCache`
 * `+(-)D` indicates with (out) `DatasetCache`
 
-Most general-purpose databases take too much time on loading data. After looking into the underlying implementation, we find that data go through too many layers of interfaces and unnecessary format transformations in general-purpose database solutions.
+Most general-purpose databases take too much time to load data. After looking into the underlying implementation, we find that data go through too many layers of interfaces and unnecessary format transformations in general-purpose database solutions.
 Such overheads greatly slow down the data loading process.
 Qlib data are stored in a compact format, which is efficient to be combined into arrays for scientific computation.
 

examples/benchmarks/LSTM/workflow_config_lstm_Alpha158.yaml

@@ -29,7 +29,7 @@ data_handler_config: &data_handler_config
         - class: CSRankNorm
           kwargs:
               fields_group: label
-    label: ["Ref($close, -2) / Ref($close, -1) - 1"] 
+    label: ["Ref($close, -2) / Ref($close, -1) - 1"]
 
 port_analysis_config: &port_analysis_config
     strategy:

examples/benchmarks/TabNet/workflow_config_TabNet_Alpha158.yaml

@@ -44,6 +44,7 @@ task:
         class: TabnetModel
         module_path: qlib.contrib.model.pytorch_tabnet
         kwargs:
+            d_feat: 158
             pretrain: True
     dataset:
         class: DatasetH

examples/benchmarks/TabNet/workflow_config_TabNet_Alpha360.yaml

@@ -0,0 +1,75 @@
+qlib_init:
+    provider_uri: "~/.qlib/qlib_data/cn_data"
+    region: cn
+market: &market csi300
+benchmark: &benchmark SH000300
+data_handler_config: &data_handler_config
+    start_time: 2008-01-01
+    end_time: 2020-08-01
+    fit_start_time: 2008-01-01
+    fit_end_time: 2014-12-31
+    instruments: *market
+    infer_processors:
+        - class: RobustZScoreNorm
+          kwargs:
+              fields_group: feature
+              clip_outlier: true
+        - class: Fillna
+          kwargs:
+              fields_group: feature
+    learn_processors:
+        - class: DropnaLabel
+        - class: CSRankNorm
+          kwargs:
+              fields_group: label
+    label: ["Ref($close, -2) / Ref($close, -1) - 1"]
+port_analysis_config: &port_analysis_config
+    strategy:
+        class: TopkDropoutStrategy
+        module_path: qlib.contrib.strategy.strategy
+        kwargs:
+            topk: 50
+            n_drop: 5
+    backtest:
+        verbose: False
+        limit_threshold: 0.095
+        account: 100000000
+        benchmark: *benchmark
+        deal_price: close
+        open_cost: 0.0005
+        close_cost: 0.0015
+        min_cost: 5
+task:
+    model:
+        class: TabnetModel
+        module_path: qlib.contrib.model.pytorch_tabnet
+        kwargs:
+            d_feat: 360
+            pretrain: True
+    dataset:
+        class: DatasetH
+        module_path: qlib.data.dataset
+        kwargs:
+            handler:
+                class: Alpha360
+                module_path: qlib.contrib.data.handler
+                kwargs: *data_handler_config
+            segments:
+                pretrain: [2008-01-01, 2014-12-31]
+                pretrain_validation: [2015-01-01, 2016-12-31]
+                train: [2008-01-01, 2014-12-31]
+                valid: [2015-01-01, 2016-12-31]
+                test: [2017-01-01, 2020-08-01]
+    record: 
+        - class: SignalRecord
+          module_path: qlib.workflow.record_temp
+          kwargs: {}
+        - class: SigAnaRecord
+          module_path: qlib.workflow.record_temp
+          kwargs: 
+            ana_long_short: False
+            ann_scaler: 252
+        - class: PortAnaRecord
+          module_path: qlib.workflow.record_temp
+          kwargs: 
+            config: *port_analysis_config

qlib/contrib/model/pytorch_tabnet.py

@@ -93,12 +93,8 @@ def __init__(
         np.random.seed(self.seed)
         torch.manual_seed(self.seed)
 
-        self.tabnet_model = TabNet(
-            inp_dim=self.d_feat, out_dim=self.out_dim, vbs=vbs, relax=relax, device=self.device
-        ).to(self.device)
-        self.tabnet_decoder = TabNet_Decoder(self.out_dim, self.d_feat, n_shared, n_ind, vbs, n_steps, self.device).to(
-            self.device
-        )
+        self.tabnet_model = TabNet(inp_dim=self.d_feat, out_dim=self.out_dim, vbs=vbs, relax=relax).to(self.device)
+        self.tabnet_decoder = TabNet_Decoder(self.out_dim, self.d_feat, n_shared, n_ind, vbs, n_steps).to(self.device)
         self.logger.info("model:\n{:}\n{:}".format(self.tabnet_model, self.tabnet_decoder))
         self.logger.info("model size: {:.4f} MB".format(count_parameters([self.tabnet_model, self.tabnet_decoder])))
 
@@ -401,7 +397,7 @@ class FinetuneModel(nn.Module):
     """
 
     def __init__(self, input_dim, output_dim, trained_model):
-        super().__init__()
+        super(FinetuneModel, self).__init__()
         self.model = trained_model
         self.fc = nn.Linear(input_dim, output_dim)
 
@@ -410,9 +406,9 @@ def forward(self, x, priors):
 
 
 class DecoderStep(nn.Module):
-    def __init__(self, inp_dim, out_dim, shared, n_ind, vbs, device):
-        super().__init__()
-        self.fea_tran = FeatureTransformer(inp_dim, out_dim, shared, n_ind, vbs, device)
+    def __init__(self, inp_dim, out_dim, shared, n_ind, vbs):
+        super(DecoderStep, self).__init__()
+        self.fea_tran = FeatureTransformer(inp_dim, out_dim, shared, n_ind, vbs)
         self.fc = nn.Linear(out_dim, out_dim)
 
     def forward(self, x):
@@ -421,13 +417,13 @@ def forward(self, x):
 
 
 class TabNet_Decoder(nn.Module):
-    def __init__(self, inp_dim, out_dim, n_shared, n_ind, vbs, n_steps, device):
+    def __init__(self, inp_dim, out_dim, n_shared, n_ind, vbs, n_steps):
         """
         TabNet decoder that is used in pre-training
         """
         self.out_dim = out_dim
 
-        super().__init__()
+        super(TabNet_Decoder, self).__init__()
         if n_shared > 0:
             self.shared = nn.ModuleList()
             self.shared.append(nn.Linear(inp_dim, 2 * out_dim))
@@ -438,7 +434,7 @@ def __init__(self, inp_dim, out_dim, n_shared, n_ind, vbs, n_steps, device):
         self.n_steps = n_steps
         self.steps = nn.ModuleList()
         for x in range(n_steps):
-            self.steps.append(DecoderStep(inp_dim, out_dim, self.shared, n_ind, vbs, device))
+            self.steps.append(DecoderStep(inp_dim, out_dim, self.shared, n_ind, vbs))
 
     def forward(self, x):
         out = torch.zeros(x.size(0), self.out_dim).to(x.device)
@@ -448,9 +444,7 @@ def forward(self, x):
 
 
 class TabNet(nn.Module):
-    def __init__(
-        self, inp_dim=6, out_dim=6, n_d=64, n_a=64, n_shared=2, n_ind=2, n_steps=5, relax=1.2, vbs=1024, device="cpu"
-    ):
+    def __init__(self, inp_dim=6, out_dim=6, n_d=64, n_a=64, n_shared=2, n_ind=2, n_steps=5, relax=1.2, vbs=1024):
         """
         TabNet AKA the original encoder
 
@@ -463,7 +457,7 @@ def __init__(
             relax coefficient:
             virtual batch size:
         """
-        super().__init__()
+        super(TabNet, self).__init__()
 
         # set the number of shared step in feature transformer
         if n_shared > 0:
@@ -474,10 +468,10 @@ def __init__(
         else:
             self.shared = None
 
-        self.first_step = FeatureTransformer(inp_dim, n_d + n_a, self.shared, n_ind, vbs, device)
+        self.first_step = FeatureTransformer(inp_dim, n_d + n_a, self.shared, n_ind, vbs)
         self.steps = nn.ModuleList()
         for x in range(n_steps - 1):
-            self.steps.append(DecisionStep(inp_dim, n_d, n_a, self.shared, n_ind, relax, vbs, device))
+            self.steps.append(DecisionStep(inp_dim, n_d, n_a, self.shared, n_ind, relax, vbs))
         self.fc = nn.Linear(n_d, out_dim)
         self.bn = nn.BatchNorm1d(inp_dim, momentum=0.01)
         self.n_d = n_d
@@ -486,14 +480,14 @@ def forward(self, x, priors):
         assert not torch.isnan(x).any()
         x = self.bn(x)
         x_a = self.first_step(x)[:, self.n_d :]
-        sparse_loss = torch.zeros(1).to(x.device)
+        sparse_loss = []
         out = torch.zeros(x.size(0), self.n_d).to(x.device)
         for step in self.steps:
             x_te, l = step(x, x_a, priors)
             out += F.relu(x_te[:, : self.n_d])  # split the feautre from feat_transformer
             x_a = x_te[:, self.n_d :]
-            sparse_loss += l
-        return self.fc(out), sparse_loss
+            sparse_loss.append(l)
+        return self.fc(out), sum(sparse_loss)
 
 
 class GBN(nn.Module):
@@ -506,14 +500,17 @@ class GBN(nn.Module):
     """
 
     def __init__(self, inp, vbs=1024, momentum=0.01):
-        super().__init__()
+        super(GBN, self).__init__()
         self.bn = nn.BatchNorm1d(inp, momentum=momentum)
         self.vbs = vbs
 
     def forward(self, x):
-        chunk = torch.chunk(x, x.size(0) // self.vbs, 0)
-        res = [self.bn(y) for y in chunk]
-        return torch.cat(res, 0)
+        if x.size(0) <= self.vbs:  # can not be chunked
+            return self.bn(x)
+        else:
+            chunk = torch.chunk(x, x.size(0) // self.vbs, 0)
+            res = [self.bn(y) for y in chunk]
+            return torch.cat(res, 0)
 
 
 class GLU(nn.Module):
@@ -525,7 +522,7 @@ class GLU(nn.Module):
     """
 
     def __init__(self, inp_dim, out_dim, fc=None, vbs=1024):
-        super().__init__()
+        super(GLU, self).__init__()
         if fc:
             self.fc = fc
         else:
@@ -561,8 +558,8 @@ def forward(self, a, priors):
 
 
 class FeatureTransformer(nn.Module):
-    def __init__(self, inp_dim, out_dim, shared, n_ind, vbs, device):
-        super().__init__()
+    def __init__(self, inp_dim, out_dim, shared, n_ind, vbs):
+        super(FeatureTransformer, self).__init__()
         first = True
         self.shared = nn.ModuleList()
         if shared:
@@ -577,7 +574,7 @@ def __init__(self, inp_dim, out_dim, shared, n_ind, vbs, device):
             self.independ.append(GLU(inp, out_dim, vbs=vbs))
         for x in range(first, n_ind):
             self.independ.append(GLU(out_dim, out_dim, vbs=vbs))
-        self.scale = torch.sqrt(torch.tensor([0.5], device=device))
+        self.scale = float(np.sqrt(0.5))
 
     def forward(self, x):
         if self.shared:
@@ -596,10 +593,10 @@ class DecisionStep(nn.Module):
     One step for the TabNet
     """
 
-    def __init__(self, inp_dim, n_d, n_a, shared, n_ind, relax, vbs, device):
+    def __init__(self, inp_dim, n_d, n_a, shared, n_ind, relax, vbs):
         super().__init__()
         self.atten_tran = AttentionTransformer(n_a, inp_dim, relax, vbs)
-        self.fea_tran = FeatureTransformer(inp_dim, n_d + n_a, shared, n_ind, vbs, device)
+        self.fea_tran = FeatureTransformer(inp_dim, n_d + n_a, shared, n_ind, vbs)
 
     def forward(self, x, a, priors):
         mask = self.atten_tran(a, priors)