Text Generation using Long Short Term Memory to Generate a LinkedIn Post

Muhammad Rizqi Assabil, Novi Yusliani, Annisa Darmawahyuni

Abstract

LinkedIn is one of the most popular sites out there to advertise oneself to potential employer. This study aims to create a good enough text generation model that it can generate a text as if it were made by someone who posts on LinkedIn. This study will use a Neural Network layer called Long Short Term Memory (LSTM) as the main algorithm and the train data consists of actual posts made by users in LinkedIn. LSTM is an algorithm that is created to reduce vanishing and exploding gradient problem in Neural Network. From the result, final accuracy and loss varies. Increasing learning rate from its default value of 0.001, to 0.01, or even 0.1 creates worse model. Meanwhile, increasing dimensions of LSTM will sometimes increases training time or decreases it while not really increasing model performance. In the end, models chosen at the end are models with around 97% of accuracy. From this study, it can be concluded that it is possible to use LSTM to create a text generation model. However, the result might not be too satisfying. For future work, it is advised to instead use a newer model, such as the Transformer model.

Full Text:

PDF

References

E. Reiter and R. Dale, “Building applied natural language generation systems,” Nat. Lang. Eng., vol. 3, no. 1, pp. 57–87, Mar. 1997, doi: 10.1017/S1351324997001502.

E. Goldberg, N. Driedger, and R. I. Kittredge, “Using natural-language processing to produce weather forecasts,” IEEE Expert, vol. 9, no. 2, pp. 45–53, Apr. 1994, doi: 10.1109/64.294135.

F. Portet et al., “Automatic generation of textual summaries from neonatal intensive care data,” Artif. Intell., vol. 173, no. 7–8, pp. 789–816, May 2009, doi: 10.1016/j.artint.2008.12.002.

D. Hutchison et al., “Every Picture Tells a Story: Generating Sentences from Images,” in Computer Vision – ECCV 2010, vol. 6314, K. Daniilidis, P. Maragos, and N. Paragios, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2010, pp. 15–29. doi: 10.1007/978-3-642-15561-1_2.

C. Ireland, “Alan Turing at 100,” Harvard Gazette, Sep. 13, 2022. https://news.harvard.edu/gazette/story/2012/09/alan-turing-at-100/ (accessed Oct. 04, 2022).

P. A. Gagniuc, Markov Chains: From Theory to Implementation and Experimentation. Hoboken, NJ, USA: John Wiley & Sons, Inc., 2017. doi: 10.1002/9781119387596.

B. Hanin, “Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?,” 2018, doi: 10.48550/ARXIV.1801.03744.

A. Graves, “Long Short-Term Memory,” in Supervised Sequence Labelling with Recurrent Neural Networks, vol. 385, Berlin, Heidelberg: Springer Berlin Heidelberg, 2012, pp. 37–45. doi: 10.1007/978-3-642-24797-2_4.

P. J. Werbos, “Backpropagation through time: what it does and how to do it,” Proc. IEEE, vol. 78, no. 10, pp. 1550–1560, Oct. 1990, doi: 10.1109/5.58337.

G. Van Houdt, C. Mosquera, and G. Nápoles, “A review on the long short-term memory model,” Artif. Intell. Rev., vol. 53, no. 8, pp. 5929–5955, Dec. 2020, doi: 10.1007/s10462-020-09838-1.

S. Santhanam, “Context based Text-generation using LSTM networks,” 2020, doi: 10.48550/ARXIV.2005.00048.

D. Pawade, A. Sakhapara, M. Jain, N. Jain, and K. Gada, “Story Scrambler - Automatic Text Generation Using Word Level RNN-LSTM,” Int. J. Inf. Technol. Comput. Sci., vol. 10, no. 6, pp. 44–53, Jun. 2018, doi: 10.5815/ijitcs.2018.06.05.

J. Guillory and J. T. Hancock, “The Effect of Linkedin on Deception in Resumes,” Cyberpsychology Behav. Soc. Netw., vol. 15, no. 3, pp. 135–140, Mar. 2012, doi: 10.1089/cyber.2011.0389.

M. Peixeiro, “Introduction to LSTM Units While Playing Jazz,” Introduction to LSTM Units While Playing Jazz, Apr. 17, 2019. https://towardsdatascience.com/introduction-to-lstm-units-while-playing-jazz-fa0175b59012 (accessed Jan. 24, 2023).

D. Rosenberg, “Stop, Words,” Representations, vol. 127, no. 1, pp. 83–92, Aug. 2014, doi: 10.1525/rep.2014.127.1.83.

Y. A. Gerhana, A. R. Atmadja, W. B. Zulfikar, and N. Ashanti, “The implementation of K-nearest neighbor algorithm in case-based reasoning model for forming automatic answer identity and searching answer similarity of algorithm case,” in 2017 5th International Conference on Cyber and IT Service Management (CITSM), Denpasar, Bali, Indonesia, Aug. 2017, pp. 1–5. doi: 10.1109/CITSM.2017.8089233.

G. Grefenstette, “Tokenization,” in Syntactic Wordclass Tagging, vol. 9, H. van Halteren, Ed. Dordrecht: Springer Netherlands, 1999, pp. 117–133. doi: 10.1007/978-94-015-9273-4_9.

Zhong Su, Qiang Yang, Ye Lu, and Hongjiang Zhang, “WhatNext: a prediction system for Web requests using n-gram sequence models,” in Proceedings of the First International Conference on Web Information Systems Engineering, Hong Kong, China, 2000, vol. 1, pp. 214–221. doi: 10.1109/WISE.2000.882395.

M. Dwarampudi and N. V. S. Reddy, “Effects of padding on LSTMs and CNNs,” 2019, doi: 10.48550/ARXIV.1903.07288.

A. C. H. Choong and N. K. Lee, “Evaluation of convolutionary neural networks modeling of DNA sequences using ordinal versus one-hot encoding method,” in 2017 International Conference on Computer and Drone Applications (IConDA), Kuching, Nov. 2017, pp. 60–65. doi: 10.1109/ICONDA.2017.8270400.

A. Vaswani et al., “Attention Is All You Need,” 2017, doi: 10.48550/ARXIV.1706.03762.

DOI: https://doi.org/10.36706/sjia.v4i2.64

Refbacks

  • There are currently no refbacks.