# data-export

**Repository Path**: tbdp/data-export

## Basic Information

- **Project Name**: data-export
- **Description**: 数据同步，目前支持从hbase同步到hdfs 
- **Primary Language**: Java
- **License**: Apache-2.0
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 1
- **Forks**: 0
- **Created**: 2017-06-17
- **Last Updated**: 2020-12-19

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

#data-export

通用数据导出程序， 支持从hbase scan部分数据导出到不同的source

### 配置文件格式
```
{
    "preProcessors":[
      {
        "type":"sql",
        "name":"loadStockInfo",
        "config":{
            "sql":"select scode, marketType from stock_info where ? ",
            "datasource":"xxx" //数据库来源
        }
      }
    ],
    "reader": {
        "type": "hbase",/*目前只有hbase*/
        "tableName":"H_MinsLine",/*表名称*/
        "schema":"tbdp",/*schema*/
        "config": {  /**通用配置**/
          "reverseCode":true,
          "startRange":"${scode}_${marketType}_${tdate}_${ttime}",
          "endRange":"${scode}_${marketType}_${tdate}_${ttime}",
          "columnSchemaPath":"aa.json" /***列的类型描述信息**/
        }
    },
    "writer": {
        "type":"hdfs",
        "overwrite":true,
        "config":{
          "format":"parquet",
          "filePathPattern":"/tbdp/H_MinsLine/${tdate}/${ttime}/$Scode.dat"
        }
    },
    "postProcessors":[]
}


```