Data Compare Tool using Pandas, Flask with MongoDB and Docker + AWS ECR — Part 4

Data comparison between MySQL tables using Python Pandas and Flask. MongoDB for logging the statistics and CI/CD using Docker with AWS ECR.

Architecture Diagram

Previous part : https://medium.com/@subham-sahoo/data-compare-tool-using-pandas-flask-with-mongodb-and-docker-aws-ecr-part-3-a7337362b416

Let’s run the application!!

Disclaimer : If you get stuck just do not get demotivated and leave stuffs halfway. Believe me the answers to your problems lie in just one search on internet (maybe more than one 😉).

GitHub repository : https://github.com/sksgit7/Data-compare-docker

Let’s say we have developed and tested our docker images. Now how to share it with other users?

For that we can build our images and store it on remote private repositories and then other users can just use a similar docker-compose yaml file to pull and run those images on their system.

Step 7 : Creating ECR repositories and pushing our images

Log in to your AWS account and choose ECR service. On the ECR dashboard, click on “create repository”. Choose “private” and give a name to the repository (same as the image name).

Then we will create another repository for mongo-express-db-compare in same way. In ECR we need to have one repository per image which can contain different versions of an image.

Here we will push our already created/built docker images which are db-compare and mongo-express-db-compare to ECR repository. If images are not built, we can use below command to build it:

docker build -t <image name>:<tag> <path to Dockerfile>

Then select the db-compare repository and click on “view push commands”. Even if we are on Windows still use/follow the Mac/Linux commands.

Authenticate : Use the first command to authenticate from local terminal to the AWS ECR.

Tag : Use third command to tag your Docker image.

docker tag db-compare:4.0 <AWS account id>.dkr.ecr.ap-south-1.amazonaws.com/db-compare:4.0

Generally it will come with latest tag for an image but as our image (already built) has 4.0 tag, I have used the same.

Then use the 4th command to push the image to ECR repository.

docker push <AWS account id>.dkr.ecr.ap-south-1.amazonaws.com/db-compare:4.0

Then click on the repository and we will see an image with name 4.0 there along with the image URI.

Similarly tag and push the mongo-express-db-compare image with 1.0 tag to the second ECR repository.

Note — We can also push the mongo image to ECR but as there are no changes required there, we will just pull the image from DockerHub directly.

Step 8 : Pull and run the images from ECR

To pull any image from ECR we need to authenticate first using the same command that we used to push.

We will use a slightly different compose-ecr.yaml file to pull and run the docker containers. Here will will not have the build commands and the image url will also point to ECR urls for Flask app and Mongo-Express.

version: ‘3’
services:
 db-compare-app:
 image: <AWS account id>.dkr.ecr.ap-south-1.amazonaws.com/db-compare:4.0
 ports:
 — 3000:3000
 volumes:
 — H:\DevOps\docker-techworld\db-compare\tgt-docker:/app/tgt
 
 mongodb:
 image: mongo
 ports:
 — 27018:27017
 environment:
 — MONGO_INITDB_ROOT_USERNAME=admin
 — MONGO_INITDB_ROOT_PASSWORD=password
 volumes:
 — mongo-data:/data/db
 healthcheck:
 test: echo ‘db.runCommand(“ping”).ok’ | mongosh localhost:27017/test — quiet
 interval: 15s
 timeout: 5s
 retries: 4
 start_period: 10s

mongo-express:
 image: <AWS account id>.dkr.ecr.ap-south-1.amazonaws.com/mongo-express-db-compare:1.0
 depends_on:
 mongodb:
 condition: service_healthy
 restart: always # fixes MongoNetworkError when mongodb is not ready when mongo-express starts
 ports:
 — 8080:8081
 environment:
 — ME_CONFIG_MONGODB_ADMINUSERNAME=admin
 — ME_CONFIG_MONGODB_ADMINPASSWORD=password
 — ME_CONFIG_MONGODB_SERVER=mongodb
volumes:
 mongo-data:
 driver: local

The image uri (url) we can get from ECR repository.

Have this file in db-compare directory and use the below command to pull and start the containers:

docker-compose -f compose-ecr.yaml up

We can see the images and containers on Docker Desktop too.

On browser, go to “http:localhost:3000” and provide the compare details (I have kept same only) and Submit.

Then go to “http:localhost:8080” and we can see one extra record for this run. Along with that we can see the previous entry even we removed the containers, because we have used named volumes to persist the data.

Also we can see a new folder inside tgt-docker (on local) with result files.

Note — The timestamp is in UTC time.

After playing around feel free to stop and remove the containers.

If you liked this project, kindly do clap. Feel free to share it with others.

👉Follow me on medium for more such interesting contents and projects.

Check out my other projects here : https://medium.com/@subham-sahoo/

Connect with me at LinkedIn. ✨

Thank you!!

References :

All references can be found here : https://github.com/sksgit7/Data-compare-docker/blob/main/references.txt