Objective.

Folk literature is an important carrier for describing the social life and cultural thoughts of the general public. Due to natural

historical

or human factors

the words in folk literature texts are ambiguous

difficult to identify or even completely missing. For better research and dissemination

it is necessary to repair the incomplete folk literature texts. There is a significant difference between the folk literary text data and the pre-training data used in the pre-training phase of pre-trained language models. For example

there are differences such as the presence of special voca

bularies and structural features. Such differences lead to catastrophic forgetting when directly fine-tuning pre-trained language models

in which the model needs to make a large number of parameter adjustments and forget the universal language knowledge learned. Avoiding catastrophic forgetting in pre-trained language models for this repair task and making the restored sentences conform to the characteristics of folk literature are the two main challenges. a knowledge distillation-based method for folk literature text restoration is proposed.

Methods.

Considering the characteristics of limited annotated data

the presence of special vocabularies and the structural nature

and adopt the pre-trained language model to expand the knowledge distillation and train the student network

enabling the automatic restoration of incomplete folk literary sentences. First

utilize pre-trained language models and student networks to extract basic feature vectors of characters from folk literary texts. These basic feature vectors are then used to construct semantic feature matrices

which undergo intermediate feature knowledge distillation. This process involves computing the SmoothL1 loss between the semantic feature matrices of each layer in the pre-trained language model and the student network

ensuring minimal distribution differences between the output features of the student network and the teacher network. By leveraging the teacher network's understanding of character-level general knowledge

the student network's comprehension of the overall semantic meaning of sentences is enhanced. Subsequently

the structural relationships among the basic feature vectors in the semantic feature matrix are treated as structural knowledge of folk literary text sentences. A structural feature matrix is constructed and subjected to structural feature knowledge distillation to reinforce the constraints of structural knowledge during the parameter update process of the student network

enhancing the structural regularity of re

paired sentences.

Results and Discussions.

For the three typical genres of folk literature

the corresponding datasets were constructed and experimental studies were conducted. In the comparative experiments

BERT on the constructed folk literary text datasets showed improvements in BLEUAvg values by 0.12%

0.8%

and 0.29%

and reductions in PPL values by 146.07

168.8

and 72.52

respectively. GPT on the constructed folk literary text datasets showed improvements in BLEUAvg values by 1%

1.28%

and 0.66%

and reductions in PPL values by 233.25

303.39

and 144.96

respectively. BART on the constructed folk literary text datasets demonstrated improvements in BLEUAvg values by 6.19%

6.41%

and 11.67%

and reductions in PPL values by 38.75

7.48

and 14.82

proving the effectiveness of the proposed method. In the ablation experiments

the BLEUAvg index of the w/S model was on average 0.3% higher than that of the w/F model

indicating that structural feature knowledge distillation has a better effect on improving the accuracy of folk literary text sentences compared to intermediate feature knowledge distillation. The PPL index was on average 22 higher

indicating that intermediate feature knowledge distillation has a better effect on improving the fluency of folk literary text sentences. The results of the ablation experiments also indicate that combining these two distillation methods can further improve the BLEUAvg index and reduce the PPL index compared to the w/S model and w/F model. In the mask rate experiment

the combined knowledge distillation method showed improvements in BLEUAvg indices and reductions in PPL indices relative to traditional fine-tuning methods

demonstrating the robustness of the combined knowledge distillation method. Furthermore

when the mask ratio is set at 15% or 20%

the BLEUAvg and PPL metrics typically demonstrate the most optimal improvement

indicating that the combined knowledge distillation method is more effective in capturing crucial linguistic information from i

ncomplete folk literary text sentences when the number of missing characters is moderate

providing the student network with rich and accurate semantic and structural knowledge. The case study intuitively demonstrates the execution results of the combined knowledge distillation method

indicating that the method can generate coherent

complete

and well-formatted sentences.

Conclusions.

According to the special vocabulary and structural features of folk literature

and the catastrophic forgetting phenomenon faced by existing controllable text generation methods

along with the insufficient generalization in handling data from the vertical domain of folk literature

a combined knowledge distillation method is proposed. This method constructs semantic feature matrices and structural feature matrices

and conducts knowledge distillation on these two matrices. Experimental results demonstrate that our method can effectively prevent catastrophic forgetting in pre-trained language models and produce sentences with more accurate semantics

comprehensive content

and sentences that better align with the formatting requirements of sentences in folk literature texts.