Kafka>> flinkctl>> 返回
项目作者： magnusfurugard

项目描述 ：
CLI for managing Flink clusters and jobs.
高级语言： Go
项目地址: git://github.com/magnusfurugard/flinkctl.git
创建时间： 2020-09-05T20:54:37Z
项目社区：https://github.com/magnusfurugard/flinkctl
开源协议：Apache License 2.0
下载


flinkctl

A command line utility for managing remote Flink clusters and applications with a familiar api.

Why not use existing binary?

In all honesty you probably should. I wanted to learn what’s available from the Flink REST API so I figured I’d build my own client. In addition, flinkctl enables you to connect to remote clusters which is something I find useful.

Status of project

Lots of work to be done still, but currently it can do rudementary tasks. Use this in production at your own risk, it’s probably not stable.

  • Add and manage multiple different (remote/local) clusters (config add-cluster <url>, config use-cluster <existing url>)
  • Respect basic auth from config (some code is there, haven’t really tested anything)
  • Describe the current cluster (describe cluster)
  • Describe a single job (describe job <job id>)
  • List running jobs (get jobs, get jobs -d)
  • Submit fat jars (with args) (submit jar -f /path/to/fat.jar --parallelism 2)
  • List uploaded jars (get jars)
  • Submit an uploaded jar to cluster (submit job <uploaded file id>)
  • Stop a running job (stop job <job id>)
  • Stop running cluster jobmanager (stop cluster)
  • Trigger job rescale (but it’s disabled in newer Flink versions :sadpanda:) (scale job <job id> --parallelism=4)
  • Generate autocomplete for binary (generate-autocomplete <shell>)
  • Json or table output on all prints / consistent printing (some deviations exists)
  • Timestamps in output
  • Create job savepoint
  • Monitor misc triggerids
  • List job vertexes
  • Measure backpressure of different vertexes
  • Metrics
  • Clean up .flinkctl.yaml-management (at the moment it’s pretty garbage)
  • Tests… Tests everywhere.

Development / Examples

There’s a dummy app that runs forever with minimum utilization in flink-cluster. In addition, the Makefile bootstraps a Flink cluster to your kubectl config current-context.

Build the binary

Since this is a Cobra application (Golang), you’ll need Go to build it. Once that’s out of your way, you can use make build to generate the binary.

Using make install will also move that binary to ~/bin/.

  1. # Start Kubernetes locally
  2. minikube start 
  4. # Make sure your current context is minikube
  5. kubectl config current-context #if it's not minikube, change.
  7. # Deploy a Flink cluster
  8. make deploy 
  10. # Port forward the jobmanager to your machine
  11. minikube service flink-jobmanager # Opens up 3 browser tabs, find the Jobmanager master (it's the one with the regular Flink dashboard).
  13. # Configure flinkctl to use your cluster
  14. flinkctl config add-cluster <url to jobmanager, e.g. https://localhost:54321>
  16. # Validate flinkctl can see cluster
  17. flinkctl describe cluster

Build a fat jar

If you have a Flink job packaged already, skip this.

  1. # Go to the dummy app
  2. cd flink-cluster/DummyApp
  4. # Use Maven to build your app
  5. mvn clean package

Start using flinkctl features

Output from a freshly deployed (example) cluster.

  1. flinkctl describe cluster -o json
  2. # Outputs:
  3. # {
  4. #     "flink-commit": "d04872d",
  5. #     "flink-version": "1.11.0",
  6. #     "jobs-cancelled": 0,
  7. #     "jobs-failed": 0,
  8. #     "jobs-finished": 0,
  9. #     "jobs-running": 0,
  10. #     "slots-available": 4,
  11. #     "slots-total": 4,
  12. #     "taskmanagers": 2
  13. # }
  15. flinkctl submit jar -f flink-cluster/DummyApp/target/DummyApp-1.0-SNAPSHOT.jar
  16. # Outputs:
  17. #  FILENAME                                                         STATUS   
  18. # ---------------------------------------------------------------- --------- 
  19. #  066d0a1e-6e9d-44b6-af36-4a72727bc7b6_DummyApp-1.0-SNAPSHOT.jar   success  
  21. flinkctl submit job 066d0a1e-6e9d-44b6-af36-4a72727bc7b6_DummyApp-1.0-SNAPSHOT.jar
  22. # Outputs:
  23. #  JOBID                             
  24. # ---------------------------------- 
  25. #  b68454ec5d7ca9e13669687871cea629  
  27. flinkctl get jobs
  28. # Outputs:
  29. #  ID                                 STATUS   
  30. # ---------------------------------- --------- 
  31. #  b68454ec5d7ca9e13669687871cea629   RUNNING  
  33. flinkctl describe job b68454ec5d7ca9e13669687871cea629
  34. # Outputs:
  35. #  DURATION   END-TIME   ISSTOPPABLE   JID                                NAME                             NOW    START-TIME   STATE     
  36. # ---------- ---------- ------------- ---------------------------------- -------------------------------- ------ ------------ ---------- 
  37. #    109.3K       1.6T   No            b68454ec5d7ca9e13669687871cea629   Example app - Infinite counter   1.6T         1.6T   RUNNING  
  39. flinkctl stop job b68454ec5d7ca9e13669687871cea629
  40. # Outputs:
  41. # Successfully cancelled job b68454ec5d7ca9e13669687871cea629
  43. flinkctl get jobs
  44. # Outputs: 
  45. #  ID                                 STATUS    
  46. # ---------------------------------- ---------- 
  47. #  b68454ec5d7ca9e13669687871cea629   CANCELED

Contributing

Feel free to contribute as much as you want. I’m very open to suggestions and improvements. There’s a lot of unattended TODO’s that needs work, and that’s just the tip of the ice berg.