HomePhabricator
Diffusion Comm 4ddc00224434

[keyserver] Process new messages for search

Description

[keyserver] Process new messages for search

Summary:
issues: https://linear.app/comm/issue/ENG-3314/implement-a-function-for-processing-messages-for-search-and-storing,
https://linear.app/comm/issue/ENG-3315/implement-adding-processed-messages-to-db-when-new-messages-are
tokenizeAndStem splits the text into words, removes stopwords, and stemms the remaining words. It returns an array. The second paramether is a boolean tellinng whether to keep the stopwords. source
code:https://github.com/NaturalNode/natural/blob/master/lib/natural/stemmers/stemmer.js

Test Plan:
Tested that when a new message is created, a proper field appears in the search table. Tested that when a message is edited (by passing an artificial edit message to processMessagesForSearch
function) a proper field is edited in the search table.
run yarn jest search-utils

Reviewers: kamil, bartek, kuba, ashoat

Reviewed By: ashoat

Subscribers: ashoat, tomek, atul

Differential Revision: https://phab.comm.dev/D7077

Details

Provenance
InkaSokolowskaAuthored on Apr 7 2023, 6:08 AM
Reviewer
ashoat
Differential Revision
D7077: [keyserver] Process new messages for search
Parents
rCOMM24f517176988: [keyserver] Add type workaround for Intl.Segmenter
Branches
Unknown
Tags
Unknown