strtok的一些问题
用strtok对以下文章进行分词：
President Barack Obama delivers remarks to troops at Fort Campbell, Ky., May 6, 2011. Vice President Joe Biden
stands at left. (Official White House Photo by Pete Souza)
It was an emotional week, from the relief of the President's announcement that Osama bin Laden was dead to the
somber rememberance of the victims in New York City. It ended 日本 on a note of gratitude, as the President and .
The Vice President:
They’re America’s “quiet professionals” -- because success demands secrecy. But I will say this. Like all of
将strtok里面的风格字符设为
char seps[] = ".?!';: _,\"-–\\\\\\/\t\n[]<>—)(“’”…\r";
日本这个中文单词就会显示出乱码，但是将那个char seps[]里面的”去掉，就不会显示乱码，但是quiet professionals”的”又无法识别，这是怎么回事？
[解决办法]
strtok是不支持多字节字符集的。在多字节环境下，分词变得非常复杂，根本不是strtok能考虑的。
strtok仅仅处理英文，单字节。
我当年上学的时候，专门写了一个两字节的strchr用来定位汉字
