Realized tonight that one of the stripping functions I'm using seems to be randomly skipping words.
<?php
function wordstrip($document){
//I truncated the list here for brevity
$wordlist = array(
"it39s",
"039",
"the",
"while",
"message");
//convert all uppercase to lower so matches work correctly
$document = strtolower($document);
foreach($wordlist as $word)
$document = preg_replace("/\s". $word ."\s/", " ", $document);
//echo $word;
//echo $document;
$nopunc = preg_replace('/[^a-z0-9]+/i', ' ', $document);
$trimmed = trim($nopunc);
return $trimmed;
}
?>
It's skipping the word "the" and I have no clue why. The list is something like 200 words long and I know its working since it strips out most other words..
I fed it "The Last Letter A Message to George W Bush and Dick Cheney From a Dying Veteran" and got back "the last letter a to george w bush and dick cheney from a dying veteran"
I think its due to the "/\s" since the "the" is at the beginning of the string. I tried "/\s?" but that didnt work. I think I just need to make the spaces optional right?
Thank you
You could use \b
to represent a word boundary and not fiddle with the spaces or periods or whatever else might surround a word:
$document = strtolower($document);
foreach($wordlist as $word)
$document = preg_replace("/\b". $word ."\b/", " ", $document);
$nopunc = preg_replace('/[^a-z0-9]+/i', ' ', $document);
$trimmed = trim($nopunc);
return $trimmed;