add blogs

Author: TimePi

blog/2019-05-28-first-blog-post.md

@@ -0,0 +1,124 @@
+# Application of DeepFM Model in AppS
+
+In the field of recommendation systems, efficiently combining low-order and high-order feature interactions to improve prediction accuracy has always been a key challenge. The DeepFM model offers a solution that combines memory capacity and generalization ability by integrating Factorization Machines (FM) with Deep Neural Networks (DNN). This article will introduce the application and effectiveness of DeepFM in the AppS business.
+
+<img src="../static/images/deepfm.svg" title="" alt="" width="522" data-align="center">
+
+## Introduction
+
+DeepFM (Deep Factorization Machine) is a recommendation system model that combines factorization machines (FM) with deep learning. It aims to capture both low-order and high-order feature interactions simultaneously. The architecture of DeepFM consists of two components: the FM component and the Deep component. The FM component is used to capture low-order feature interactions, while the Deep component learns high-order feature interactions through a multi-layer perceptron (MLP).
+
+### FM Module
+
+- **Function**: The FM module focuses on capturing second-order interactions between features. It leverages feature embeddings to compute interaction terms and efficiently represent relationships between sparse features.
+- **Advantage**: By modeling low-order interactions, the FM module effectively handles sparse data, making it particularly suitable for scenarios with a large number of sparse features.
+
+### DNN Module
+
+- **Function**: The DNN module is used to learn high-order feature combinations. Through a multi-layer neural network, DNN can capture complex nonlinear feature interactions.
+- **Customization Capability**: Users can design the network structure of the DNN according to specific needs, including the number of layers, the number of neurons in each layer, activation functions, and regularization strategies.
+- **Advantage**: With a flexible structure design, the DNN module can generalize to new feature combinations and improve the model's adaptability to different data distributions.
+
+### Benefits of DeepFM over FM
+
+1. **Comprehensive Feature Interaction Capability**: Traditional FM models mainly focus on second-order interactions between features, whereas DeepFM can effectively capture high-order feature interactions by introducing a deep learning component, thus improving recommendation accuracy.
+
+2. **No Need for Manual Feature Engineering**: DeepFM can automatically learn feature interactions, reducing the reliance on manual feature engineering, which is particularly useful for handling complex, large-scale datasets.
+
+3. **Shared Feature Embeddings**: The feature embedding layer in DeepFM is shared between the FM and Deep components, making the model more efficient in capturing feature interactions while reducing the number of model parameters.
+
+### Advantages of DeepFM
+
+- **Comprehensive Capability**: DeepFM combines the strengths of FM and DNN, allowing it to learn both low-order and high-order feature interactions without the need for feature engineering.
+- **Model Simplicity**: Compared to training FM and DNN separately and then combining them, DeepFM maintains model compactness and efficiency by sharing the feature embedding layer.
+- **Wide Applicability**: Due to its flexibility and strong expressive power, DeepFM is widely used in fields such as ad click-through rate prediction and recommendation systems.
+
+### Example Code for Developing DeepFM with PyTorch
+
+In the following example, we will develop the FM and DNN modules separately and then combine them into a complete DeepFM model.
+
+```python
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+class DeepFM(nn.Module):
+    def __init__(self, field_dims, embed_dim, mlp_dims):
+        super(DeepFM, self).__init__()
+        self.embeddings = nn.ModuleList([
+            nn.Embedding(dim, embed_dim) for dim in field_dims
+        ])
+        self.linear = nn.Linear(sum(field_dims), 1)
+        self.fm = FM(embed_dim)
+        self.dnn = DNN(sum(field_dims) * embed_dim, mlp_dims)
+
+    def forward(self, x):
+        x_emb = [emb(x[:, i]) for i, emb in enumerate(self.embeddings)]
+        x_emb = torch.cat(x_emb, dim=1)
+        x_linear = self.linear(x)
+        x_fm = self.fm(x_emb)
+        x_dnn = self.dnn(x_emb.view(x_emb.size(0), -1))
+        return x_linear + x_fm + x_dnn
+
+class FM(nn.Module):
+    def __init__(self, embed_dim):
+        super(FM, self).__init__()
+        self.embed_dim = embed_dim
+
+    def forward(self, x):
+        square_of_sum = torch.sum(x, dim=1) ** 2
+        sum_of_square = torch.sum(x ** 2, dim=1)
+        return 0.5 * torch.sum(square_of_sum - sum_of_square, dim=1, keepdim=True)
+
+class DNN(nn.Module):
+    def __init__(self, input_dim, dims):
+        super(DNN, self).__init__()
+        layers = []
+        for dim in dims:
+            layers.append(nn.Linear(input_dim, dim))
+            layers.append(nn.ReLU())
+            input_dim = dim
+        self.layers = nn.Sequential(*layers)
+
+    def forward(self, x):
+        return self.layers(x)
+
+# Example usage:
+field_dims = [10, 10, 10]  # Example field dimensions
+embed_dim = 10
+mlp_dims = [64, 32]
+model = DeepFM(field_dims, embed_dim, mlp_dims)
+
+# Dummy input
+x = torch.randint(0, 10, (4, len(field_dims)))  # Batch size 4
+output = model(x)
+print(output)
+```
+
+## Application
+
+### 1. Feature Embedding Configuration
+
+In our DeepFM model, the embedding dimension for each feature is set to 10. This configuration effectively captures low-order feature interactions and provides a solid foundation for subsequent high-order feature learning through the deep neural network.
+
+### 2. Model Training and Optimization
+
+Building on our experience with FM model training, the DeepFM model excels in combining memory and generalization. The FM component captures low-order feature interactions, while the DNN component learns high-order feature combinations. This combination achieves excellent results in the current business scenario.
+
+- Memory Capability: DeepFM uses the FM component's low-order interactions to capture known, stable feature combinations.
+
+- Generalization Capability: Through the DNN component, DeepFM can discover new, potential high-order feature combinations, enhancing the prediction of user behavior.
+
+### 3. AB Testing Results
+
+In the "Guess You Like" module, deploying the DeepFM model led to a **4.66%** increase in average distribution per user. This result indicates that DeepFM significantly enhances the quality of personalized recommendations for users.
+
+<img src="../static/images/DeepFM-AB.png" title="" alt="" width="522" data-align="center">
+
+## Further Reading
+
+- [A Factorization-Machine based Neural Network for CTR Prediction - arXiv](https://arxiv.org/abs/1703.04247)
+
+- [Deep Factorization Machines — Dive into Deep Learning](https://d2l.ai/chapter_recommender-systems/deepfm.html)
+
+- [DeepFM for recommendation systems explained with codes](https://medium.com/data-science-in-your-pocket/deepfm-for-recommendation-systems-explained-with-codes-c200063990f7)

blog/2019-05-29-long-blog-post.md

@@ -0,0 +1,73 @@
+# Application of the ESMM Model in AppS
+
+In modern recommendation systems, particularly within the AppS business environment, predicting user behaviors such as Click-Through Rate (CTR) and Conversion Rate (CVR) is crucial for enhancing user satisfaction and driving business growth. The ESMM model, with its unique architecture and efficient multi-task learning capability, offers an outstanding solution for the AppS business.
+
+<img title="" src="../static/images/ESMM-origin.webp" alt="" width="522" data-align="center">
+
+## Introduction
+
+ESMM, short for Entire Space Multi-task Model, is a multi-task learning model specifically designed to tackle problems related to ad recommendations and user behavior prediction. The core idea of ESMM is to enhance the overall performance of the model by simultaneously learning multiple related tasks. This approach not only shares potential information between different tasks but also effectively alleviates the issue of data sparsity.
+
+The ESMM model is typically applied to predict CTR and CVR. Traditional methods often train two separate models to predict CTR and CVR, whereas ESMM simultaneously performs these two prediction tasks within a unified framework, thereby capturing the correlation between them more effectively.
+
+To learn more about the foundational concepts of ESMM, you can read this [academic paper on ESMM](https://arxiv.org/abs/1804.07931).
+
+### Major Advantages of ESMM
+
+- **Data Efficiency**: By sharing the feature space, ESMM can better utilize data, especially in sparse data scenarios.
+- **Performance Enhancement**: By jointly learning multiple tasks, ESMM can better capture the mutual influences between related tasks, improving the accuracy of predictions.
+- **Simplified Architecture**: Compared to training multiple models independently, ESMM provides a more streamlined and efficient solution.
+
+### Differences Between ESMM and MMOE
+
+In multi-task learning, besides ESMM, there is another popular model known as MMOE (Multi-gate Mixture-of-Experts). Both MMOE and ESMM aim to enhance the performance of multiple tasks by sharing information, but they exhibit significant differences in architecture and application scenarios:
+
+#### Architectural Differences
+
+- **ESMM**: ESMM conducts multi-task learning by sharing the entire feature space. It primarily uses a unified network structure to simultaneously predict multiple tasks (such as CTR and CVR) and enhances overall performance by sharing underlying features.
+
+- **MMOE**: MMOE employs a more complex structure by introducing multiple expert networks and gating mechanisms to dynamically select suitable features and model paths for each task. Each task has its own gating network to select the most relevant information from multiple experts.
+
+#### Application Scenarios
+
+- **ESMM**: Suitable for scenarios where tasks are highly related and require extensive information sharing, particularly when data is sparse and efficient information utilization is needed.
+
+- **MMOE**: More flexible and applicable to scenarios where task correlations are weaker or personalized feature selection is required. Due to its complex selection mechanism, MMOE performs better in situations with conflicting task requirements.
+
+#### Performance Aspects
+
+- **ESMM**: Provides stable performance improvements between related tasks through its simplified network architecture and efficient feature sharing.
+
+- **MMOE**: Capable of offering higher prediction accuracy in complex task environments through flexible expert selection mechanisms, especially when task requirements are diverse.
+
+## Application
+
+### Similarity in Basic Structure Between ESMM and MMOE
+
+The ESMM model shares many structural similarities with the traditional MMOE model. Both employ a multi-task learning framework to enhance the performance of different tasks by sharing information. However, the ESMM model adopts a different approach in the final conversion rate prediction: it calculates the predicted Conversion Rate (pCVR) as the product of two towers, a design aimed at fully capturing the interaction between CTR and CVR.
+
+### Key Components of the ESMM Model
+
+<img title="" src="../static/images/ESMM.webp" alt="" width="522" data-align="center">
+
+#### Two Expert Networks
+
+When applied to the AppS business, ESMM uses two expert networks. These expert networks are responsible for handling features related to CTR and CVR tasks, respectively. Through specialized network structures, ESMM can better extract and utilize task-specific information, thereby enhancing prediction accuracy.
+
+#### Two Gating Mechanisms
+
+In addition to expert networks, ESMM also employs two gating mechanisms to control the CTR and CVR tasks separately. These gating mechanisms dynamically adjust the selection and utilization of features for each task, ensuring that each task receives the most suitable information flow. Through optimization of gating mechanisms, ESMM provides more precise results in complex user behavior predictions.
+
+## Experimental Results and Effects
+
+In practical applications within the AppS business, the ESMM model has demonstrated significant results through A/B testing. In the "Guess You Like" module, the ESMM model successfully achieved a 6.45% increase in average distribution per user.
+
+<img title="" src="../static/images/ESMM-AB.png" alt="" width="522" data-align="center">
+
+## Further Reading
+
+- [Entire Space Multi-Task Model: An Effective Approach for Estimating ... - arXiv](https://arxiv.org/abs/1804.07931)
+
+- [ESMM &mdash; easy_rec 0.8.5 documentation](https://easyrec.readthedocs.io/en/latest/models/esmm.html)
+
+- [GitHub - dai08srhg/ESMM: PyTorch implementation of Entire Space Multitask Model (ESMM)](https://github.com/dai08srhg/ESMM)