Please enable JavaScript to use CodeHS

  • AI

Train Karel with AI

In this project, students will explore the use of a hand pose detection AI model to control Karel's movements.

Medium

1 Hour

High School

Middle School

Project Description

In this project, students will explore the use of a hand pose detection AI model in Tensorflow to control Karel’s movements.


Background

Tensorflow is an open-source library for machine learning tasks, allowing you to build and train artificial intelligence models for various applications. In simpler terms, it’s a toolkit for creating powerful AI programs.

MediaPipe’s hand pose detection model, available through TensorFlow, uses machine learning to identify hands in images and videos, pinpointing 21 keypoints on each hand to track its pose in real-time. You can try out the model in the online demo.

Hand pose demo with mapped keypoints on a hand holding an egg


Your Task

In this project, you will…

  1. Explore the hand pose detection model by reviewing MediaPipe’s documentation of keypoints.
  2. Try out MediaPipe’s demo program to better understand visual recognition models.
  3. Try out the provided program by using different hand poses to move Karel the dog from left to right on the screen.
  4. Modify the JavaScript code to use different hand pose keypoints that make Karel move left or right.
  5. Reflect on your exploration and code adjustments.

Project Overview

Here is an outline of the project activities: