kafka-to-hbase-reconciliation

Reconciliation tool to confirm that messages written from Kafka have been successfully written to HBase

This repo contains Makefile, and Dockerfile to fit the standard pattern.
This repo is a base to create new Docker image repos, adding the githooks submodule, making the repo ready for use.

After cloning this repo, please run:

make bootstrap

Self signed certificates

The integration tests standup various services as docker containers and the
export and dks containers require keystores and truststores so that they can
communicate over 2-way https. To generate these:

make certificates

Local runs of build

Use this command to build and unit test and make a local distribution;

make local-all

Local runs of integration tests in docker compose stack

NOTE depends on the local build above. This is built into the makefile commands above.

Use this command to build and unit test and make a local distribution, then run all the integration tests;

make integration-all

Local runs of specific integration tests in docker compose stack

If you wish to isolate the integration test you’d like to run. First ensure you’ve done

make services

And then run either

make partitioned-integration-test

OR

make trim-reconciled-integration-test

Both of these will run a populate script to populate MySQL and Hbase (if required) to set the conditions for the test.
Then run a docker container for this respective test.
Then run behave tests to test the conditions.
NOTE The test reconciliation container starts daemonised, the make file will stop the container once all tests are complete.

Local running in an IDE

You can do this for both the application, and the integration tests.

• Make a local run configuration in i.e. Intellij from the gradle menu (see images below).
• For the app:
  • 
  • or
  • 
• For the integration tests:
  • 

Either, with local application.properties

• Verify the that the local application.properties matches all current docker compose settings

Or, with env vars

• Make a list duplicating the docker compose settings

Then, for either

• Edit it.
  • Verify you have all the same settings that are used for the local docker stack.
  • Change the addresses of all hosts to localhost or 127.0.0.1.
  • HBASE_ZOOKEEPER_QUORUM=localhost
  • METADATASTORE_ENDPOINT=localhost
• The ports exposed by the containers should already match.

Then for both,

• Add env vars for the logger - in intellij you can paste in this csv:

  CONTAINER_VERSION=latest
  ENVIRONMENT=local-ids
  APPLICATION_NAME=reconciliation
  APP_VERSION=test
  LOG_LEVEL=DEBUG

• Run the application/test target you just made, it should start and connect up.

Environment variables needed by the ECS container

These translate into the spring properties the application needs to run