Answering to comments on D7076:

natural.PorterStemmer.tokenizeAndStem uses AggressiveTokenizer by default (PorterStemmer gets this function from Stemmer that it extendes, and Stemmer uses AggressiveTokenizer).

From the other tokanizers the only one that looks potentially useful for us is RegexpTokenizer, that allows to define own regex.

const tokenizer = new natural.RegexpTokenizer({
      pattern: /([A-Za-zÀ-ÿ-'"]+|[0-9._]+|.|!|\?|:|;|,|-)/iu,
    });
    console.log(tokenizer.tokenize(text));

produces

[ 'Hello', 'test', "it's", 'working', '😄', 'test' ]

But it would be a bit difficult to have it parse it's as it's and 'hello' as hello. Generally we would have to come up with a good regex.

The disadvantage of Intl.Segmenter is that there seems to be some problem with types: Intl is typed in flow builtin definitions, but it does not contain Segmenter https://github.com/facebook/flow/blob/main/lib/intl.js
But it works, so if I work around the flow problem it should be fine.

Although it's worth noting that the stemmer will remove the s from it's anyway, since it's treating it as a modifying suffix.
So after applying Intl.Segmenter.segment and natural.PorterStemmer.stem we get

['hello', 'test', "it'", 'work', '😄', 'test']

Intl.Segmenter seems like the best bet! We can figure out the Flow types... worst case we might need to do something like const OurIntl: OurIntlType = (Intl: any); or something.

In D7077#214454, @inka wrote:

Although it's worth noting that the stemmer will remove the s from it's anyway, since it's treating it as a modifying suffix.
So after applying Intl.Segmenter.segment and natural.PorterStemmer.stem we get

['hello', 'test', "it'", 'work', '😄', 'test']

That seems good to me! It makes sense that it would stem "it's" into "it". One thing that's weird is that it's not stripping the apostrophe... ideally it would just be "it", which will likely be filtered because it's a stop word. I'm not sure if "it'" (with apostrophe) counts as a stop word.

One easy solution would be to add some manual code that strips punctuation (non-letter characters) after the stemmer runs.

I don't think we want to strip all non-letter characters, because then we loose the emojis again.
Here is punctuation regex defined by wikipedia:

[][!"#$%&'()*+,./:;<=>?@\^_`{|}~-]

maybe we could use that?

I'm also wondering if we should remove the punctuation from inside of words. Maybe only strip the punctuation at the beginning and at the end? Since it makes sense to leave o'clock and 100,000 and such.

I run a test, and for hello{item}bye the segmenter parsed it to hello {item} bye for all symbols apart from ., _ , ', @, which were left as hello{item}bye . And 100{item}000 it parsed to 100 {item} 000 for all symbols apart from ., ,, ;, _, ', @. So the separated punctuation symbols would be removed, and only those special cases I listed would be left.

I added D7238 to type Intl to have the Segmenter field.

It's at the bottom of this diff, I think you maybe didn't see it?

Oops!! Not sure how I missed that, sorry about it.

I think it would be good to have unit tests of the exact result of segmentAndStem, rather than just checking the result with a RegExp. That way the reader can tell what segmentAndStem is doing exactly, and we can test it end-to-end.

Regarding the punctuation, are we are deciding between "hello+goodbye" being treated as one word "hellogoodbye", versus being split into two words "hello goodbye"? If so, I think it makes sense to split into two words in that case. (Whereas I would prefer for cases like "Ashoat's" to be stemmed into "Ashoat" if possible.)

(If we had a unit test of hello+goodbye and Ashoat's showing what we expect the exact result to be, that would answer my question!)