machine learning in fintech?

The pain of lazy evaluation

Recently, I have spent less and less time designing model architectures and tuning hyperparameters. In comparison, the time spent writing feature-processing code has multiplied. Spark is certainly efficient at processing large datasets, but its lazy evaluation is also its most frustrating trait. After carefully writing a data-processing job, I can wait five hours for it to fail while saving to HDFS—not with a Python stack trace, but with a pile of errors produced during Java translation. It leaves me feeling deeply powerless.

Every cell execution feels like Russian roulette: will the next few hours yield anything, or end in nothing?

Data preprocessing at work

This is completely different from competitions. I used to think data preprocessing meant dealing with missing values, outliers, and categorical labels. In real work, that would already count as clean data. More often there are variable-length sequences full of missing values and strings; sparse labels and miscellaneous embedding vectors are concatenated together. On one side are enormous numbers of zeros, on the other hundreds of floating-point dimensions. Sometimes you cannot explain why one vector in an embedding ranks so highly in feature importance.

The trade-off between high dimensionality and models

Once dimensionality reaches several thousand or more, there are not many model choices left. Particularly when distributed training is required, the practical options are tree models, linear models, and DNNs. If we still want the elusive goal of interpretability, only the first two remain.

Given the cost of each training run, so-called hyperparameter tuning disappears. I can only hope that this tree model finds sufficiently important split points, or that the randomly selected negative samples have sufficiently complete features to form a clear separating plane—or hyperplane—from positive samples.

Negative-sample selection is a kind of dark art. That is my conclusion. Amitabha.

Reality: small samples and class imbalance

The counterpart to high dimensionality is a small number of samples. With nearly ten thousand features and barely more than ten thousand positive examples, clever architectural tricks at the model level no longer matter much. You would not believe that simply stacking more layers could improve metrics in this situation; that is a reality one has to accept. Positive and negative samples can also differ noticeably in sequence-feature length. If sequence length alone can predict the result, why not just use a rule to match them directly?

Marketing algorithms and recommendation algorithms differ greatly at the sample level, because the former’s samples are literally money. Small samples and class imbalance create many challenges, and they further constrain the model choices mentioned above.

Closing thoughts

Written nearly a year into this career, I know that the machine learning I have encountered is only a drop in the ocean. Someone once said that marketing algorithms in fintech cannot really be called algorithm engineering, only data analysis. Based on what I have learned so far, I cannot really deny that. Still, out of a small bit of vanity, I call myself a Machine Learning Engineer on LinkedIn. Hahaha.

May 25, 2025, Suzhou