LSTM Trend Classification: Scaling, Regularization, and Label Design

# LSTM for trend prediction


# LSTM for trend prediction












Been wanting to get my hands dirty with ML for a while now and since I'm interested in finance and trading as well, I figured this would be a good project to get started after reading Deep LSTM with Reinforcement Learning Layer for Financial Trend Prediction in FX High Frequency Trading Systems (https://www.mdpi.com/2076-3417/9/20/4460/htm) by Francesco Rundo.

I'm working on the first step, a LSTM with 3 timeseries as input and a categorical output (0, 1, 2).

After working on it all day I got something to work, but with my results being far off of what Francescos model apparently achieves (43% vs 70%), I'm not really sure if it is just a matter of throwing more training data and time at it, or if I made a basic mistake due to being new to ML in general.

```
def create_model():
  model = Sequential()
  model.add(LSTM(HIDDEN_CELL_NUMBER, input_shape=(100, 3), return_sequences=True))
  model.add(LSTM(HIDDEN_CELL_NUMBER, input_shape=(100, 3)))
  model.add(Dense(3, activation=activations.softmax))
  return model

HIDDEN_CELL_NUMBER = 300
EPOCHS = 10

class_weights = class_weight.compute_class_weight('balanced', np.unique(y_train), y_train)
y_train = to_categorical(y_train)
y_TEST = to_categorical(y_TEST)

model = create_model()
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['categorical_accuracy'])

model.fit(x_train, y_train, class_weight=class_weights, epochs=EPOCHS, verbose=1)
```

Are optimizer, loss and metrics the correct choices for 3 categorical outputs? Is there anything else obvious that I did wrong?

## Answer by Stéphane (score 7, accepted)

https://quant.stackexchange.com/a/54385

The first thing you can do to help a neural network learn more rapidly is to normalize all inputs between 0 and 1. The library sklearn has a preprocess.scale() function that does just that -- make sure to do it separately for training and testing data (or training, validation and testing data if you use three separate sets). This alone can make a huge difference. I saw a deep reinforcement learning tutorial where it massively accelerated the learning. Just to be on the safe side, tensorflow.keras.layers has a BatchNormalization() function that you can add in between each LSTM laters as

```
model.add( BatchNormalization() )
```

Another very important detail here is that you never introduced dropouts between your layers. It's essentially designed to randomnly get rid of some information and it is typically used in all types of recurrent networks. As a default, you could add it between each layer with something like a 20% dropout rate:

```
model.add( Dropout(0.2) )
```

This one can also be found in the layers module of tensorflow.keras. A last thing is that people routinely stack at least one dense layer on top of LSTM layers before the output layer is introduced. Cook and Hall (2015) at the Federal Reserve found it was working well on macroeconomic data and it seems to be the standard everywhere. So, these are the first simple things I'd personally try first.

At the level of finance, you do have to think carefully about how you label the data. For example, if you can go short on an asset, you have to make sure that you're not labeling all down moves as opportunities to take a short position -- the fall must be large enough to overcome fees, slippage and price impact. Likewise with price increases. This will generally mean that you have a lot of "do nothing" labels in your data and that means there is a local optimum where saying do nothing 100% of the time is actually hard to beat. I see that you're using weights, but if you have a lot of data, a simple way to deal with it is to just randomnly select a balanced (or closer to being balanced) dataset. If that fails, weighing observations like you do would be the last simple option before considering oversampling.

I've seen some systems that use very few features do relatively well on cyptocurrencies, for example. But then again it's a tutorial and they didn't bother doing more than identifying the sign of the growth rate over 3 minutes interval... Depending on how smart you are about labeling or your goal, accuracy might be hard to get. Ask Jim Simons. It took him 40 years and a lot of very smart PhDs to get big in quantitative finance.

EDIT You can, for example, use

```
model = Sequential()
for i in range(N_LSTM_CELLS-1):
    model.add( LSTM(LSTM(HIDDEN_CELL_NUMBER, 
                         input_shape=(100, 3), 
                         return_sequences=True)) )
    model.add( Dropout(0.2) )
    model.add( BatchNormalization() )

model.add( LSTM(LSTM(HIDDEN_CELL_NUMBER, 
                         input_shape=(100, 3)) )
model.add( Dropout(0.2) )
model.add( BatchNormalization() )

model.add( Dense(N_UNITS, activation="relu") )
model.add( Dense(3,       activation="softmax") )
```

And you can visit pythonprogramming.net, the Deep Learning Course.