How to Use ChatGPT to Generate Math Homework From Photos of My Daughter’s Assignments: Part 1
Turning photos of math worksheets into additional practice problems using gpt-4-vision-preview and Python
Last year, OpenAI’s ChatGPT introduced gpt-4-vision-preview, a model that allows you to ask questions about an image you upload. For example, you can ask it to describe a scene, a room, or even the contents of your refrigerator. I’ve considered using it as an interior designer, but I digress.
Today, we’re going to use ChatGPT, in Python, to take an image of a 6th grade math worksheet and generate a similar problem set. Why? Once in a while, I come up with practice problems to supplement my daughter’s math homework. It takes time to write good ones, and although I really enjoy personalizing them, I wouldn’t mind saving some time.
This blog post will cover:
- Using
gpt-4-vision-previewvia the OpenAI Python SDK to generate a worksheet from an uploaded photo. - Iterating until we are happy with the results.
In subsequent blog posts, we will use the generated assignment to:
- Ask
gpt-4to spit out LaTeX code so we can generate a PDF of the new assignment. - Experiment with generating a very simple, pure HTML/JS based flash card example.
Let’s get the elephant out of the room. Many parents and educators are rightfully concerned about students using ChatGPT and other GenAI tools to cheat or do homework for them. I am more interested in how we can use it to enhance education.
Please note: The examples in this series use GPT-4, requiring you to be a “Pay-As-You-Go” customer. As of this writing, that means you need at least $1 loaded into your account. Pricing for the various OpenAI models, based on tokens (or pixels), can be found here.
Iteration 1: Generating new problems from an image
We will use the following Python packages:
openai(for more information, see the ChatGPT developer quickstart)base64dotenv(to load an.envfile containing an OpenAI API key)
First, you’ll create an OpenAI client with your API key. For local development, it’s common practice to stick API keys and passwords in a .env file. (but make sure you only commit a sample file without real values into version control: if you’re using Git, add .env to your .gitignore).
from openai import OpenAI
from dotenv import load_dotenv, find_dotenv
import base64
# Load the .env file that contains the OPENAI_API_KEY
# OPENAI_API_KEY=<your key>
load_dotenv(find_dotenv())
# Create an OpenAI client, using the OPENAI_KEY environment variable
# client = OpenAI(api_key=<insert your key>)
client = OpenAI()The client “Chat Completions” API takes many parameters. We’ll use:
messages: an array of dictionaries representing custom instructions and a prompt that you are sending to ChatGPTmodel: the model you are usingmax_tokens: the maximum number of tokens you want in the response
Below is the structure of messages for requests to thegpt-4-vision model. You can submit a prompt and image via the "user” role. The image_url can be a link to a publicly accessible URL that returns an image, or a Base64 encoded image included inline in the request. We’ll encode the image and send it in the request. (For more details on how to use the Vision capability, please see the documentation here.)
messages = [
{
"role": "user",
"content": [
{
"type": "text",
"text": f"{prompt}",
},
{
"type": "image_url",
"image_url": {"url": f"data:image/jpeg;base64,{base64_image}"},
},
],
}
]In order to encode the image, let’s follow their recommendations to base64 encode the image, and then convert the bytes to a utf-8 encoded string for sending over HTTP.
# Images can be passed to ChatGPT either via a publicly accessible link, or
# inline as a base64 encoded string
# https://platform.openai.com/docs/guides/vision
def encode_image(image_path):
with open(image_path, "rb") as image_file:
return base64.b64encode(image_file.read()).decode("utf-8")In our first attempt, let’s ask ChatGPT “Using the first page of this 6th grade math test, please generate 10 more problems”.
# Question we are asking based on the image
prompt = "Using the first page of this 6th grade math test, please generate 10 more problems."
# Encodes a photo taken of a page from a 6th grade math assessment test.
base64_image = encode_image("data/unit-2-assessment.jpg")
# Set the messages to send to model
messages = [
{
"role": "user",
"content": [
{
"type": "text",
"text": f"{prompt}",
},
{
"type": "image_url",
"image_url": {"url": f"data:image/jpeg;base64,{base64_image}"},
},
],
}
]
# Send request to model
response = client.chat.completions.create(
model="gpt-4-vision-preview",
messages=messages,
max_tokens=4096)
# Print out the response from ChatGPT
print(response.choices[0].message.content)
worksheet_str = response.choices[0].message.contentBefore I could fully contemplate the thought of AI taking over parenting completely, this is what it came back with:
Sure! Here are 10 additional math problems suitable for a 6th-grade assessment based on the style of problems on the test page:
**Greatest Common Factor (GCF) and Least Common Multiple (LCM):**
5. a. Find the GCF of (24, 36).
b. Find the LCM of (6, 14).
6. a. Find the GCF of (21, 49).
b. Find the LCM of (15, 20).
**Real-World Application of GCF and LCM:**
7. Two lights start blinking together at 12:00 PM. One light blinks every 6 seconds and the other every 8 seconds. When will they blink together again?
8. A gardener wants to plant flowers that bloom every 7 days and bushes that bloom every 10 days. If she plants them both today, in how many days will they bloom on the same day?
**Fraction Multiplication and Area Models:**
9. Mike has a rope that is 3/4 of a yard long. He cuts it into 2 equal parts. What is the length of each part? Draw a model to represent the problem.
10. A recipe calls for 2/5 cup of sugar. If Sarah is making half of the recipe, how much sugar does she need? Draw a model to represent the problem.
**Word Problems Involving Fractions:**
11. A school has 2/3 of a pizza left over from a party. If they divide this evenly among 3 teachers, how much pizza will each teacher get?
12. A bag contains 3/5 lb of apples. If each apple weighs 1/5 lb, how many apples are in the bag?
**Miscellaneous:**
13. In a class of 30 students, 4/5 of the students brought their homework. How many students brought their homework?
14. If one batch of cookies requires 3/4 cup of chocolate chips, how many cups of chocolate chips are required to make 4 batches?
These problems are designed to reinforce and assess skills in finding GCF and LCM, understanding real-world applications of these concepts, using fractions in context, drawing area models, and solving word problems related to multiplication and division of fractions.Not bad ChatGPT! You passed the first round of my interview. However, when I tried it again, it didn’t generate headings like “Real-World Application of GCF and LCM” for each section.
Iteration 2: Adding more to the prompt
So in the hopes of achieving more consistency, I’d like to provide more instructions in the prompt. ChatGPT tends to explain what it’s trying to do, so maybe we can take advantage of that? Notice the first response starts with:
Sure! Here are 10 additional math problems suitable for a 6th-grade assessment based on the style of problems on the test page:Let’s modify the prompt to be: “Using the first page of this 6th grade math test, please generate 10 more problems based on the style of problems on the test page”. With this, we get a response closer to the original style. Also, I was wrong — there were never separate section headings in the original worksheet, but ChatGPT decided to include that in the first response.
Sure, I can create additional problems in the style presented in the math test you've shown. Below are 10 new problems across the various types seen in the image:
1. Find the greatest common factor for each pair of numbers.
a. GCF (16, 24) = ___
b. GCF (30, 45) = ___
2. Find the least common multiple for each pair of numbers.
a. LCM (7, 5) = ___
b. LCM (9, 6) = ___
3. The Pencil Factory produces two types of pencils: standard pencils every 5 minutes and colored pencils every 8 minutes. They start operations at 9:00 A.M.
How long until both types of pencils are produced together again? ___ minutes (unit)
4. Ben's family plans to grow flowers on 1/4 of their garden space. Ben and his brother will plant 1/2 of the flower portion of the garden.
How much of the family garden will Ben and his brother plant? ___
Draw an area model and write a number sentence to represent the problem.
5. Find the greatest common factor for each pair of numbers.
a. GCF (14, 21) = ___
b. GCF (35, 50) = ___
6. Find the least common multiple for each pair of numbers.
a. LCM (4, 10) = ___
b. LCM (8, 10) = ___
7. The Gadget Workshop produces two gadgets: a widget every 6 minutes and a gizmo every 9 minutes. Production starts at 7:30 A.M.
When will both gadgets next be produced at the same time? ___ minutes (unit)
8. Emma's family allocated 3/5 of their backyard for a vegetable garden. Emma and her cousin will cultivate 3/4 of the vegetable garden together.
What fraction of the backyard will Emma and her cousin cultivate? ___
Illustrate the situation with an area model and write a number sentence.
9. Find the greatest common factor for each pair of numbers.
a. GCF (18, 27) = ___
b. GCF (20, 30) = ___
10. Find the least common multiple for each pair of numbers.
a. LCM (3, 11) = ___
b. LCM (7, 8) = ___
These problems maintain the format and cover similar math concepts as the original test page, including greatest common factors (GCF), least common multiples (LCM), and fractional parts of a whole, along with real-world applications.Iteration 3: Adding a “system” message
You can give a “persona” of sorts to ChatGPT. According to the documentation:
“The system message helps set the behavior of the assistant. For example, you can modify the personality of the assistant or provide specific instructions about how it should behave throughout the conversation.”
Let’s update the messages list and send a request again. The initial system message needs to be the first element.
messages[:0] = {
"role": "system",
"content": "You are a helpful 6th grade math school teacher that creates homework assignments for your students."
}
response = client.chat.completions.create(
model="gpt-4-vision-preview",
messages=messages,
max_tokens=4096)
# Print out the response from ChatGPT
print(response.choices[0].message.content)
worksheet_str = response.choices[0].message.contentWe get a response similar to the previous one. Hmm, it doesn’t appear to have made a difference, other than ending with “Remember to show your work for full points!”:
Certainly! Based on the style of problems on the test page, here are 10 more problems for your 6th grade math homework assignment:
1. Find the greatest common factor for each pair of numbers.
a. GCF (24, 36) = ____
b. GCF (9, 27) = ____
2. Find the least common multiple for each pair of numbers.
a. LCM (7, 14) = ____
b. LCM (8, 10) = ____
3. The Happy Hat Shop makes two types of hats. They produce a sun hat every 6 minutes and a baseball cap every 8 minutes. If they start at 10:00 A.M., how long until the sun hat and baseball cap are made together again?
Answer: ________ minutes
4. Mike's family divided up their yard such that 1/4 of the yard will have flowers. Mike and his brother will plant 1/2 of the flower portion of the yard. How much of the yard will Mike and his brother plant with flowers?
Draw an area model and write a number sentence to represent the problem.
Number sentence: ____________
5. Find the greatest common factor for each pair of numbers.
a. GCF (35, 50) = ____
b. GCF (20, 45) = ____
6. Find the least common multiple for each pair of numbers.
a. LCM (3, 11) = ____
b. LCM (6, 14) = ____
7. The Fizzy Soda Company makes two flavors of soda. They produce a cola every 5 minutes and a lemon-lime soda every 7 minutes. They start at 9:00 A.M. How long until both flavors are produced together again?
Answer: ________ minutes
8. Laura's family is installing new carpet in their house such that 3/5 of the floors will be carpeted. Laura and her uncle will carpet 2/3 of the carpeted portion of the floors. How much of the house's floors will Laura and her uncle carpet?
Draw an area model and write a number sentence to represent the problem.
Number sentence: ____________
9. Find the greatest common factor for each pair of numbers.
a. GCF (21, 28) = ____
b. GCF (18, 54) = ____
10. Find the least common multiple for each pair of numbers.
a. LCM (9, 15) = ____
b. LCM (4, 16) = ____
These questions continue the concepts of GCF and LCM as well as real-world application problems involving fractions and repeated scheduling. Remember to show your work for full points!Iteration 3: Adding more instructions to the prompt
I’m fairly happy with the format being generated, but descriptive headings would be helpful to guide the student, at least when they’re first learning. Let’s try to get those back! I made a few attempts (not shown here), and each time, it unexpectedly seemed to impact the types of problems that were generated. Either a problem type present in the original worksheet was omitted, or something not present was included. The following prompt (changes in bold) gave me good results for the headings, but I wasn’t entirely happy: it still impacted the types of problems.
“Using the first page of this 6th grade math test, please generate 10 more problems based on the style of problems on the test page. Provide headings for each section.”
Based on the style of problems on the test page, here are 10 more problems that could serve as additional practice or homework for 6th-grade students:
**Greatest Common Factor (GCF)**
1. Find the greatest common factor for the following pair of numbers.
a. GCF (24, 36) = _______
b. GCF (30, 45) = _______
**Least Common Multiple (LCM)**
2. Find the least common multiple for the following pair of numbers.
a. LCM (7, 5) = _______
b. LCM (9, 6) = _______
**Real-World LCM Problem**
3. The community center offers a yoga class every 9 days and a pilates class every 6 days. If both classes are held today, in how many days will they both be offered on the same day again?
**Fraction of a Group Problem**
4. Pedro's bookshelf has 4 shelves, and \( \frac{3}{4} \) of them are filled with books. If each shelf can hold 16 books, how many books are on Pedro's bookshelf?
5. A recipe calls for \( \frac{1}{2} \) a cup of sugar. If Simon wants to make half a batch, how much sugar will he need?
**Fraction Multiplication Problem**
6. Miranda has \( \frac{3}{5} \) of a yard of fabric. She cuts off \( \frac{2}{3} \) of what she has for a project. How much fabric did she cut off?
**Area Model for Fractions**
7. Create an area model to represent the multiplication of \( \frac{3}{6} \times \frac{1}{4} \). Write the number sentence that represents this problem.
**Mixed Numbers**
8. Convert the mixed number to an improper fraction:
a. \( 3\frac{1}{4} \) = _______
b. \( 2\frac{2}{5} \) = _______
**Comparing Fractions**
9. Which is greater, \( \frac{5}{6} \) or \( \frac{2}{3} \)? Show how you know by either finding a common denominator or converting to decimals.
**Real-World Fraction Problem**
10. A school's soccer team won \( \frac{3}{5} \) of their games last season. If they played 25 games, how many games did they win?
Please make sure to show all your work for each problem and check your answers. Remember to simplify your fractions where possible. Good luck!I decided to try one more prompt (changes in bold):
“Using the first page of this 6th grade math test, please generate 10 more problems based on the style of problems on the test page. Please include descriptive titles for each section.”
### Greatest Common Factor (GCF)
5. **GCF Practice**
a. GCF (24, 36) = ______
b. GCF (40, 60) = ______
### Least Common Multiple (LCM)
6. **LCM Challenge**
a. LCM (6, 9) = ______
b. LCM (7, 14) = ______
### Real-world Applications of LCM and GCF
7. **Factory Schedule Problem**
The Pencil Factory produces red pencils every 5 hours and blue pencils every 8 hours. They start production at 7:00 A.M. When will both red and blue pencils be produced at the same time next?
### Fraction Multiplication in Real-world Context
8. **Gardening Fractions**
Daniel is planning a rectangular flower bed. He wants 1/4 of the flower bed to be roses and 1/2 of the roses section to consist of red roses. What fraction of the whole flower bed will be red roses?
Draw a rectangular model and write a number sentence to represent the problem.
### Additional Common Factor and Multiple Problems
9. **Common Factors and Multiples**
a. Name all the common factors of 14 and 21.
b. What is the smallest common multiple of 11 and 12?
### Practical Fraction Problem
10. **Baking Fractions**
Maria has 3/4 of a cup of sugar. She uses 2/3 of it to make a batch of cookies. How much sugar does she have left?
Draw a visual model and write a number sentence to represent the problem.
### Rational Number Operations
11. **Fraction Subtraction**
a. Subtract the fraction: 5/8 - 1/4 = ______
b. Subtract the fraction: 7/10 - 2/5 = ______
### Time Interval Problem Using LCM
12. **Meeting Time**
Two friends want to meet for yoga. One friend goes every 6 days, and the other goes every 9 days. If they both went today, after how many days will they both go to yoga on the same day next?
### Area and Fraction Review
13. **Camping Plot**
A family is camping on a rectangular plot that is 1/3 acre. They use 3/5 of this plot to set up their tents. What fraction of an acre did the tents cover?
Sketch a visual representation of the camping plot subdivided accordingly.
### Additional Calculation Practice
14. **Quick Calculations**
a. Multiply the fraction: 3/8 * 2/5 = ______
b. Divide the fraction: 6/7 ÷ 3/4 = ______
### Problem-Solving with Unit Rates
15. **Speedy Snail Trail**
A snail moves at a constant rate of 1/12 of a mile per hour. How far will the snail travel in 4 hours?
Provide a number sentence to calculate the distance traveled.
Please remember to write out your work for each problem and label your answers clearly. Good luck!Conclusions
It can certainly produce problems similar to the original worksheet, but it’s not easy getting consistent results! I need to experiment with adding more instructions (i.e. further specifying requirements).
- Trying out different prompts to generate headings seemed to impact what types of problems it was generating, which was totally unexpected.
- Sometimes it included problem types that were not in the original worksheet.
- Sometimes it would leave out some problem types (“area models”) that were in the original worksheet.
- It periodically changes the format of how it was outputting fractions: “1/4”, “¼”, “\(\frac{1}{4}\)” (the last one is Markdown format).
- Be ready to be told you’re not a cool dad. ¯\_(ツ)_/¯
Recommendations
- Use LangChain to orchestrate calls to ChatGPT (or other LLMs), create reusable “prompt templates”, and chain together a sequence of operations. (For the sake of brevity, I didn’t use LangChain for this example.)
- Use Jupyter notebooks for rapid feedback, reproducibility, and traceability: run one experiment per notebook and save your results in appropriately named folders and files. (This is how I worked on this.)
- Don’t give up! It may take many iterations to find something that you like. I’m okay with being somewhat forgiving here since I can easily edit the output.
- Try different worksheets to see how it performs.
