智能网关>> go-data-routing>> 返回
项目作者： Zensey

项目描述 ：
Pipelined data routing/trasnformation engine
高级语言： Go
项目地址: git://github.com/Zensey/go-data-routing.git
创建时间： 2020-07-28T11:29:56Z
项目社区：https://github.com/Zensey/go-data-routing
开源协议：
下载


go-data-routing

Test

The library provides a pipeline engine for stream processing of data.
It is based on the concept of EIP (enterprise integration patterns).

The motivation for this project is simple: to get an easy and clear way of coding ETL-like programs for parallel processing of data. In my case it was a BFS crawler tuned for extraction of specific metadata, (see a basic version in example folder).

Features

The library provides the following primitives:

  • route (chain of nodes processing messages)
  • node:
    • filter — filter a stream of exchanges
    • processor — processes a stream of exchanges (tasks) in parallel
    • wire tap: sends a copy of msg to another route (referenced by name)
    • to: enrich msg on another route (request-reply / enrichment pattern)

All the primitives are accessible through DSL.

Route builder DSL (methods):

Method Signature Args
Source f func(n *Node) function f, used for generation of exchanges
Filter f func(e Exchange, n *Node) function f, intercepts exchanges
Process workers int Number of workers, beeing used to process exchanges
To route string Name of the route, where to redirect an exchange for execution of request-reply
WireTap route string Name of the route, where to copy an exchange
Sink f func(e Exchange) error Function f, used for consumption exchanges

Examples:

  • a simplistic bfs crawler: go run examples/crawler/main.go

Design of node:

  • each node is connected with the next one (if exists) only with 1 channel
  • node owns an input channel
  • output is just a reference to the input of next node
  • node does not close the output channel, instead it just sends a Stop msg to a next node
  • if a node is the last in a chain than an output message being sent is discarded unless it’s not a RequestReply

Code coverage

make code-coverage and see the coverage.html for details.

  1. github.com/Zensey/go-data-routing/node.go (100.0%)
  2. github.com/Zensey/go-data-routing/nodetype_string.go (75.0%)
  3. github.com/Zensey/go-data-routing/pool.go (100.0%)
  4. github.com/Zensey/go-data-routing/route_builder.go (71.8%)
  5. github.com/Zensey/go-data-routing/router_context.go (100.0%)
  6. github.com/Zensey/go-data-routing/worker.go (81.8%)