Academia.eduAcademia.edu

Figure 16 – uploaded by Tiong Yew Tang

See full PDFdownloadDownload figure
The macro F1-scores and accuracy for several classifier-augmenter combinations in average scores

Figure 16 The macro F1-scores and accuracy for several classifier-augmenter combinations in average scores

Related Figures (19)

FIGURE 1. Current invoice classification manual process.
FIGURE 1. Current invoice classification manual process.
INVOICE DATA DESCRIPTION
INVOICE DATA DESCRIPTION
FIGURE 2. Invoice categories class imbalance
FIGURE 2. Invoice categories class imbalance
FIGURE 3. Overall methodology from data preparation to evaluation phase.
FIGURE 3. Overall methodology from data preparation to evaluation phase.
TRAIN-TEST PAIRING DATASET DESCRIPTION In the text cleaning sub-process, the invoice descriptions are cleaned. This step aims to ensure only words with useful semantics are kept and to ensure the learning efficiency of the model. The process starts with the loading of the ‘D01’ dataset and ends with the output of the “‘D02’ dataset. This process sequence is visualized in Fig. 5. First, a text-cleaning function is used to clean the text. The function includes removal of
TRAIN-TEST PAIRING DATASET DESCRIPTION In the text cleaning sub-process, the invoice descriptions are cleaned. This step aims to ensure only words with useful semantics are kept and to ensure the learning efficiency of the model. The process starts with the loading of the ‘D01’ dataset and ends with the output of the “‘D02’ dataset. This process sequence is visualized in Fig. 5. First, a text-cleaning function is used to clean the text. The function includes removal of
digits, lowercasing, removing symbols, removing single characters, removing multiple spaces, removing stop words, and removing month names. Then, the cleaned text will be subjected to checking against the WordNet lexicon database to retain only English words. As short texts potentially contain words which may be acronyms, jargon or incomprehensible to the language model, a conservative approach is to only retain English words. Due to the cleaning steps, there will be entirely blank rows. Those rows will be removed from the dataset. Next, instances tagged to more than one class were also removed from the dataset to reduce overlapping of classes. There is an exception to this, where within a multi-labe scenario there are two labels for a single description, but one label was the minority with only one sample. In this case, the instances were not discarded. Instead, the minority sample wil be reclassed to the majority label to resolve the issue. Lastly instances where the class is tagged to only one instance were removed from the dataset to ensure at least one instance wil be in the train and test set.
digits, lowercasing, removing symbols, removing single characters, removing multiple spaces, removing stop words, and removing month names. Then, the cleaned text will be subjected to checking against the WordNet lexicon database to retain only English words. As short texts potentially contain words which may be acronyms, jargon or incomprehensible to the language model, a conservative approach is to only retain English words. Due to the cleaning steps, there will be entirely blank rows. Those rows will be removed from the dataset. Next, instances tagged to more than one class were also removed from the dataset to reduce overlapping of classes. There is an exception to this, where within a multi-labe scenario there are two labels for a single description, but one label was the minority with only one sample. In this case, the instances were not discarded. Instead, the minority sample wil be reclassed to the majority label to resolve the issue. Lastly instances where the class is tagged to only one instance were removed from the dataset to ensure at least one instance wil be in the train and test set.
FIGURE 5. Text cleaning sub-process. steps undertaken for the train test split are listed in Table III. Before splitting the train and test set, any class labels with only 1 instance are removed from the dataset because it cannot split. This is to ensure both the training and testing datasets have representatives of all classes. The dataset is then split into the training and testing sets to the ratio of 85:15. This is to ensure there is sufficient training for effective learning and enough testing data for accurate evaluation. The outputs of this process are ‘D1 train’, ‘D1_test’, ‘D2_train’, and ‘D2 test’.
FIGURE 5. Text cleaning sub-process. steps undertaken for the train test split are listed in Table III. Before splitting the train and test set, any class labels with only 1 instance are removed from the dataset because it cannot split. This is to ensure both the training and testing datasets have representatives of all classes. The dataset is then split into the training and testing sets to the ratio of 85:15. This is to ensure there is sufficient training for effective learning and enough testing data for accurate evaluation. The outputs of this process are ‘D1 train’, ‘D1_test’, ‘D2_train’, and ‘D2 test’.
label column with augmented texts is then converted to string format. The new label column is then tokenized. Only one word from the tokens will be selected randomly for all instances. Then, a new column is generated containing the original description text concatenated with the newly selected word for each row. This method creates semantically enriched format. The new label column is then tokenized. Only one
label column with augmented texts is then converted to string format. The new label column is then tokenized. Only one word from the tokens will be selected randomly for all instances. Then, a new column is generated containing the original description text concatenated with the newly selected word for each row. This method creates semantically enriched format. The new label column is then tokenized. Only one
“X_test’, ‘y train’, and ‘y test’ variables are assigned accordingly. Then, the train set is tokenized using the NLTK library, and a word index is created containing all the words in the train set descriptions. The NLTK library is used due to its popularity in working with NLP in Python. Next, the “X_ train’ and ‘X_test’ word tokens are converted into integer sequences and padded to ensure the same vector sequence length for each instance. The padding step is needed as the model requires input of the same sequence size for each instance. Subsequently, the Bi-LSTM model is built and compiled. The dropout rate is configured to a standard 0.5 across all the Bi- LSTM experiments in this study. This rate is selected to reduce overfitting, especially for the smaller train datasets. A learning rate of 0.001, although relatively small, is chosen to ensure stable learning and improve the chances of arriving at the optimal model, as opposed to a higher learning rate. A slow learning decay of le-6 is also chosen to ensure stable training. After compilation, the Bi-LSTM model is fitted to the vector text representation of the “X_train’ and ‘y_ train’ variables for model training. A batch size of 128 was chosen. Larger batch size settings were attempted but not pursued due to larger GPU memory requirements for larger batch sizes. A train-validation
“X_test’, ‘y train’, and ‘y test’ variables are assigned accordingly. Then, the train set is tokenized using the NLTK library, and a word index is created containing all the words in the train set descriptions. The NLTK library is used due to its popularity in working with NLP in Python. Next, the “X_ train’ and ‘X_test’ word tokens are converted into integer sequences and padded to ensure the same vector sequence length for each instance. The padding step is needed as the model requires input of the same sequence size for each instance. Subsequently, the Bi-LSTM model is built and compiled. The dropout rate is configured to a standard 0.5 across all the Bi- LSTM experiments in this study. This rate is selected to reduce overfitting, especially for the smaller train datasets. A learning rate of 0.001, although relatively small, is chosen to ensure stable learning and improve the chances of arriving at the optimal model, as opposed to a higher learning rate. A slow learning decay of le-6 is also chosen to ensure stable training. After compilation, the Bi-LSTM model is fitted to the vector text representation of the “X_train’ and ‘y_ train’ variables for model training. A batch size of 128 was chosen. Larger batch size settings were attempted but not pursued due to larger GPU memory requirements for larger batch sizes. A train-validation
FIGURE 9. Bi-LSTM modelling sub-process. split of 85-15 was set considering the large dataset volume used in this study where a 15% volume is sufficient for model performance reliability. Early stopping was set at 3 epochs past the epoch with the best validation loss. After training, the *X_test’ is applied to the model for label prediction. Lastly, the predicted outputs are compared against
FIGURE 9. Bi-LSTM modelling sub-process. split of 85-15 was set considering the large dataset volume used in this study where a 15% volume is sufficient for model performance reliability. Early stopping was set at 3 epochs past the epoch with the best validation loss. After training, the *X_test’ is applied to the model for label prediction. Lastly, the predicted outputs are compared against
datasets, which may provide biased accuracy results. This is caused by high prediction accuracy for the majority classes and low prediction accuracy for the minority classes. As such, the macro Fl-score is utilized as the next classification evaluation metric. Each class, regardless of size or prevalence in the dataset, is considered equally in the macro Fl-score
datasets, which may provide biased accuracy results. This is caused by high prediction accuracy for the majority classes and low prediction accuracy for the minority classes. As such, the macro Fl-score is utilized as the next classification evaluation metric. Each class, regardless of size or prevalence in the dataset, is considered equally in the macro Fl-score
DATASET VOLUMES POST DATA PREPARATION afAwid VY TEXT CLASSIFICATION PERFORMANCE
DATASET VOLUMES POST DATA PREPARATION afAwid VY TEXT CLASSIFICATION PERFORMANCE
Post cleaning, the accuracy of all three classifiers improved to the range of 83%-84% accuracy, with LSVM achieving the best performance at 84.4%. This confirms that the raw text has a lot of noise, containing many specific industry terms and named entities [32]. After retaining only English words, the performance improved by up to around 7 ppts from the baseline accuracy scores. The Macro Fl-scores improved up to a maximum of 8 ppts with measures up to 75% achieved by LSVM while deep learning classifiers lag at below the 70% mark, indicating that deep learning classifiers still do not have enough information to detect good patterns for better classification performance.
Post cleaning, the accuracy of all three classifiers improved to the range of 83%-84% accuracy, with LSVM achieving the best performance at 84.4%. This confirms that the raw text has a lot of noise, containing many specific industry terms and named entities [32]. After retaining only English words, the performance improved by up to around 7 ppts from the baseline accuracy scores. The Macro Fl-scores improved up to a maximum of 8 ppts with measures up to 75% achieved by LSVM while deep learning classifiers lag at below the 70% mark, indicating that deep learning classifiers still do not have enough information to detect good patterns for better classification performance.
accuracy, as supported by prior studies [34], [35]. The classification results improved cleaning. This confirms that the and a lot of industry terms [32] perform poorly before cleaning. macro F1 scores improved by wu to more than 80% after raw text carries a lot of noise , which caused the model to After semantic enrichment, p to 10 ppts on average. This supports the hypothesis that obtaining auxiliary semantic information from labels [61] is sparsity and improving text c beneficial for reducing data assification performance in general. The findings also support BERT as the best performer, which is consistent [105], [106]. with previous research [96],
accuracy, as supported by prior studies [34], [35]. The classification results improved cleaning. This confirms that the and a lot of industry terms [32] perform poorly before cleaning. macro F1 scores improved by wu to more than 80% after raw text carries a lot of noise , which caused the model to After semantic enrichment, p to 10 ppts on average. This supports the hypothesis that obtaining auxiliary semantic information from labels [61] is sparsity and improving text c beneficial for reducing data assification performance in general. The findings also support BERT as the best performer, which is consistent [105], [106]. with previous research [96],
Related topics:
Artificial IntelligenceMachine LearningAccountingArtficial Neural Networksrobotics process automation(Robotic process Automation) RPARobotic Process Automation TechnologyLarge language models
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved