Building a Wildlife Recognition App with Tensorflow and AWS Lambda

I installed Tensorflow via these instructions to keep everything nice and encapsulated.

I also chose to use the make_image_classifier package. I've followed enough tutorials to understand what's going on under the hood, and all I really want to do is quickly get a model up and running. As such, this is not an ML tutorial! If you want to build your own model, I highly recommend fastai's Practical Deep Learning Course.

This is my first foray into a decentralized serverless project, and is less of a tutorial and more of an overview of how I went about architecting this particular project.

The architecture:

The task can be broken down into a few discrete parts. The model's functions (training and classifying), the app to allow users to interact with the model, and storing/outputting the results.

I also chose to add a Discord integration to allow users to correct mistakes.

For data I used the, starting with the Missouri set to get a good baseline.

I know I want to store the results in a database. For this I have chosen Mongo Atlas because of its generous free tier, easy setup, and my blithe confidence that it looks pretty easy to self-host as well if necessary.

So this app's web portion will need to:

  • Fetch data from the database and display it in fun and informative ways
  • Fetch data from the database and display as a list so that humans can provide feedback
  • Allow users to upload images for the user to classify

The database entries should have:

  • timestamp image was taken
  • classification
  • confidence interval
  • human-updated label, if corrected
  • timestamp of corrected, if corrected
  • geolocation if provided
  • link to the image in the S3 bucket

The lamdba functions should:

  • Call the classification function and return the necessary data on image upload to the S3 bucket.
  • If the classification confidence interval is low, invoke a webhook to ping the Discord server and ask for feedback on the site, with a link. (Discord's button interactions are not available in webhooks, sadly)
  • Populate the database with the new entry
  • In the future, call the ReoLink API at a regular interval (the camera used in this project is a ReoLink camera) and automatically parse/upload images and video to the bucket

The first thing I did was train the model on the Missouri dataset. Once I was satisfied with the results when invoking locally (manually feeding in an image from the command line and inspecting the printed output), I added it to AWS.

Next, I scaffolded a basic NEXTjs app. I chose to use the Auth0 example because it has everything I need out of the box. I hooked up my app to an Auth0 app and that was out of the way already.

Side note, how great is Next? Where was Next when I was learning React? I had so many projects utilizing 3rd party APIs out there, and I couldn't publish ANY of them because I didn't yet know how to write a whole express API and host them both.

Getting everything play nicely with serverless and AWS lambda was a bit of a headache.

First off, PIL doesn't really work, and you have to specify a layer

Second, if you're using a mac with an M1 chip, you'll likely get a mismatch between the architecture of the requested Docker image and the host's architecture.

Two ways to solve this, from the docs. One, update your serverless.yml to configure the Docker image, and include an platform: linux/amd64. This would be nifty if I was configuring my containers already, but I'm wasn't.

You can also just specify architecture: arm64 at the provider level, or on a per-function basis. I hope that I will never need to run different functions on different architectures, to be honest.

Fixing PIL errors: https://stackoverflow.com/questions/57197283/aws-lambda-cannot-import-name-imaging-from-pil https://aws.plainenglish.io/easiest-lambda-layers-for-python-functions-357d6dde8c4a with Klayers https://github.com/keithrozario/Klayers/tree/master/deployments/python3.8

Make sure to remove anything mac specific from the requirements.txt, such as tensorflow-macos and tensorflow-metal

This project was built with:

NextJS python tensorflow AWS lambda Docker auth0

This project was focused less heavily on coding specific solutions, and more on finding, debugging, and implementing solved solutions.

For deployment, I messed around with the serverless framework for way too long. Ultimately I ended up following these steps, replacing the model functions with my own model, which is hosted in an S3 bucket.

https://aws.amazon.com/blogs/machine-learning/using-container-images-to-run-tensorflow-models-in-aws-lambda/

Consider saving this as a project instead, and not writing a thing about it, instead putting it on your projects page.