# gse
**Repository Path**: xingnan/gse
## Basic Information
- **Project Name**: gse
- **Description**: No description available
- **Primary Language**: Unknown
- **License**: Apache-2.0
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No
## Statistics
- **Stars**: 0
- **Forks**: 0
- **Created**: 2020-08-28
- **Last Updated**: 2021-05-24
## Categories & Tags
**Categories**: Uncategorized
**Tags**: None
## README
# [gse](https://github.com/go-ego/gse)
Go 语言高效分词, 支持英文、中文、日文等
[](https://github.com/go-ego/gse/commits/master)
[](https://circleci.com/gh/go-ego/gse)
[](https://codecov.io/gh/go-ego/gse)
[](https://travis-ci.org/go-ego/gse)
[](https://goreportcard.com/report/github.com/go-ego/gse)
[](https://godoc.org/github.com/go-ego/gse)
[](https://github.com/go-ego/gse/releases/latest)
[](https://gitter.im/go-ego/ego?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
词典用双数组 trie（Double-Array Trie）实现，
分词器算法为基于词频的最短路径加动态规划, 以及 DAG 和 HMM 算法分词.
支持 HMM 分词, 使用 viterbi 算法.
支持普通、搜索引擎、全模式、精确模式和 HMM 模式多种分词模式，支持用户词典、词性标注，可运行 JSON RPC 服务。
分词速度单线程 9.2MB/s，goroutines 并发 26.8MB/s. HMM 模式单线程分词速度 3.2MB/s.（ 双核 4 线程 Macbook Pro）。
## Binding:
[gse-bind](https://github.com/vcaesar/gse-bind), binding JavaScript and other, support more language.
## 安装/更新
```
go get -u github.com/go-ego/gse
```
## [Build-tools](https://github.com/go-ego/re)
```
go get -u github.com/go-ego/re
```
### re gse
创建一个新的 gse 程序
```
$ re gse my-gse
```
### re run
运行我们刚刚创建的应用程序, CD 到程序文件夹并执行:
```
$ cd my-gse && re run
```
## 使用
```go
package main
import (
"fmt"
"github.com/go-ego/gse"
"github.com/go-ego/gse/hmm/pos"
)
var (
seg gse.Segmenter
posSeg pos.Segmenter
new = gse.New("zh,testdata/test_dict3.txt", "alpha")
text = "你好世界, Hello world, Helloworld."
)
func cut() {
hmm := new.Cut(text, true)
fmt.Println("cut use hmm: ", hmm)
hmm = new.CutSearch(text, true)
fmt.Println("cut search use hmm: ", hmm)
hmm = new.CutAll(text)
fmt.Println("cut all: ", hmm)
}
func main() {
cut()
segCut()
}
func posAndTrim(cut []string) {
cut = seg.Trim(cut)
fmt.Println("cut all: ", cut)
posSeg.WithGse(seg)
po := posSeg.Cut(text, true)
fmt.Println("pos: ", po)
po = posSeg.TrimWithPos(po, "zg")
fmt.Println("trim pos: ", po)
}
func cutPos() {
fmt.Println(seg.String(text, true))
fmt.Println(seg.Slice(text, true))
po := seg.Pos(text, true)
fmt.Println("pos: ", po)
po = seg.TrimPos(po)
fmt.Println("trim pos: ", po)
}
func segCut() {
// 加载默认字典
seg.LoadDict()
// 载入词典
// seg.LoadDict("your gopath"+"/src/github.com/go-ego/gse/data/dict/dictionary.txt")
// 分词文本
tb := []byte("山达尔星联邦共和国联邦政府")
// 处理分词结果
// 支持普通模式和搜索模式两种分词，见代码中 ToString 函数的注释。
// 搜索模式主要用于给搜索引擎提供尽可能多的关键字
fmt.Println("输出分词结果, 类型为字符串, 使用搜索模式: ", seg.String(tb, true))
fmt.Println("输出分词结果, 类型为 slice: ", seg.Slice(tb))
segments := seg.Segment(tb)
// 处理分词结果
fmt.Println(gse.ToString(segments))
segments1 := seg.Segment([]byte(text))
fmt.Println(gse.ToString(segments1, true))
}
```
[自定义词典分词示例](/examples/dict/main.go)
```Go
package main
import (
"fmt"
"github.com/go-ego/gse"
)
func main() {
var seg gse.Segmenter
seg.LoadDict("zh,testdata/test_dict.txt,testdata/test_dict1.txt")
seg.LoadStop()
// seg.LoadDictEmbed()
// seg.LoadStopEmbed()
text1 := "所以, 你好, 再见"
fmt.Println(seg.String(text1, true))
segments := seg.Segment([]byte(text1))
fmt.Println(gse.ToString(segments))
}
```
[中文分词示例](/examples/main.go)
[日文分词示例](/examples/jp/main.go)
## Authors
- [Maintainers](https://github.com/orgs/go-ego/people)
- [Contributors](https://github.com/go-ego/gse/graphs/contributors)
## License
Gse is primarily distributed under the terms of both the MIT license and the Apache License (Version 2.0), thanks for [sego](https://github.com/huichen/sego) and [jieba](https://github.com/fxsjy/jieba)([jiebago](https://github.com/wangbin/jiebago)).