What Are the Expenses of Data Annotation Using an Internal Team?

In the first part of this series, we emphasized the necessity of meticulously annotating images and videos, which is crucial for developing AI products capable of precise analysis, predictions, and reliable results. We also highlighted the extensive time and financial investment required for individual annotation efforts.

This article delves into the costs and resources needed to sustain an in-house data annotation team.

But first, a quick recap of our scenario: A lead robotics scientist is designing a smart home assistant robot that distinguishes between dirt and valuable items in a home environment. The chaos of daily life often entails toys strewn about, misplaced eyewear, pet hair, and other miscellaneous items. The envisioned robot aims to clean effectively while assisting in locating misplaced items, thereby adding functionality beyond typical cleaning devices — potentially aiding the elderly in keeping tabs on their possessions.

As the chief scientist, your pivotal role involves working with a compact research team to gather a dataset comprising 100,000 images depicting various room settings with items scattered across floors. This dataset size aligns with the typical range for robotics projects, which can encompass thousands to millions of images.

Each image features, on average, 23 items, translating into the task of annotating roughly 2.3 million objects. This series of articles outlines various strategies to address this substantial annotation workload, including DIY methods, establishing an in-house team, outsourcing, and crowdsourcing.

Welcome to the second part of our series, focusing on data annotation costs. This article explores the financial implications of recruiting annotators and forming your annotation team.

• Case 1: You manage the task alone or with minimal colleague assistance.
• Case 2: You recruit annotators and collaborate with your own team.
• Case 3: You delegate the task to external professionals.
• Case 4: You opt for crowdsourcing.

Case 2: You Hire Annotators and Annotate with In-House Data Annotation Team

Just like any endeavor, forming an annotation team has advantages and challenges. Let’s examine this approach’s benefits, time commitments, and cost-efficiency.

Here, we focus solely on the monthly expenses and costs. The minimal team needed to annotate 2.3 million objects consists of 35 annotators, complemented by managerial staff responsible for onboarding, offboarding, and upskilling.

For these 35 annotators, one manager and three to four senior annotators are essential to lead the team.

Data Annotator Contracts and Team Size

Data annotation teams can range from small (up to five members) to extensive groups, with larger teams necessitating heightened coordination and management. Recruiting is straightforward for smaller teams but becomes more complex for larger assemblies. Annotators might be full-time employees with fixed salaries or contractors, with contractors presenting challenges in retention and engagement due to their involvement in multiple projects and expectations for workload-aligned compensation. When working with contractors, extra effort is necessary to ensure availability. For instance, if you require 35 annotators, consider recruiting between 60 to 70 to accommodate potential unavailability.

Data Annotation Project Time and Costs

Based on our experience, the recruitment process might take:

• Time to find a data annotation manager: At least 1 month
• Time to find one annotator: Up to 1 month
• Time to onboard one annotator: Up to 1 month

Job interviews and onboarding can occur concurrently. If fortunate, you might hire between 5 to 10 annotators monthly. However, assembling and training a large data annotation team typically requires 3–4 months.

Expense-wise, consider:

• Manager salary (per month): Up to $6,000 (data from Indeed, June 2024)
• Annotator salary (per hour): This varies — if hiring abroad, starting from $1/h and up to $40 for hires in the U.S. or those requiring a high level of qualification.

Where to recruit? Platforms like Upwork, Indeed, and LinkedIn are your go-to options. With recruitment agency assistance, job posting costs can range from $0 to $500 in rare cases.

Platforms like CVAT.ai, popular in the data annotation space, can significantly reduce both time and costs. Annotators tend to respond quickly to advertised vacancies.

Data Preparation and Annotation Speed

The next step is to prepare data: the dataset is foundational to any robotics project.

For this project, the scientist must review a vast collection of video footage to select pertinent frames and then devise a comprehensive data annotation specification. In our scenario, this specification plans to cover 40 different classes, each annotated with polygons individually.

On average, the complete guideline spans 30–50 pages. It will include detailed instructions for annotating each class, examples of correct and incorrect annotations, and edge cases. Drafting this detailed specification is time-intensive; it could take several weeks. The data annotation specification will be updated throughout the project, as it’s challenging to foresee all potential edge cases from the outset.

The time required to annotate each object with polygons will later be calculated, considering factors like the object’s complexity and size, the image’s clarity, and the annotator’s skill level.

• Simple Object (e.g., a rectangular object): 5–10 seconds
• Moderately Complex Object (e.g., a car): 30–60 seconds
• Highly Complex Object (e.g., a human with detailed limb annotations): 1–3 minutes or more

Operational Costs

In addition to onboarding and training expenses, the costs for data annotation projects also encompass licenses and instance costs per annotator. Each annotator may require a license for the annotation software used, which can vary significantly in price depending on the software’s complexity and capabilities.

CVAT costs $33 per seat, but you can also opt for free, open-source tools.

Remember, even “free” tools necessitate time and resources for setup and support; time is indeed money. While “free” means downloading and installing the open-source tool, the rest depends on your time, expertise, and effort — and how much of your paid time will be dedicated to this. Operational costs include accounting and contract management expenses, which are company-specific and cannot be precisely estimated.

Data Annotation Project Final Calculations

To calculate the total time required for 35 professional annotators to annotate 2,300,000 objects, where each object takes approximately 40 seconds on average to annotate, follow these steps:

Calculate the Total Time for All Objects:

• Total time = 2,300,000 objects × 40 seconds per object = 92,000,000 seconds, or 25,555.56 hours

Divide by the Number of Annotators to Find Time per Annotator:

• Time per annotator = 25,555.56 hours / 35 annotators = 730.16 hours

Thus, if all annotators work simultaneously and efficiently, each annotator will need about 18.25 work weeks, approximately 4.2 months, to complete the annotation of all 2,300,000 objects.

To calculate the costs for the scenario described, let’s break it down into its components and sum them up for the 4.2 months required for the project. We’ll assume each annotator earns $550 per month and that licenses cost vary from free to $33 per month. Additionally, management and validation cost $6,000 + 20% per month from the total cost of annotators.

Total Salary Costs for Annotators (4.2 months):

• Total annotator costs = $2,310 per annotator × 35 annotators = $80,850

Management and validation Fees (for 4.2 Months):

• Total cost for a data annotation manager = $25,200
• Management and validation Fees = $80,850 * 20% = $16,170

Conclusion: Annotating 100,000 images, that is, 2,300,000 objects, will take 4.2 months and $122,220. The costs of the software licenses must be added to this number.

Hidden and One Time Costs

When calculating the cost of an annotation team, it’s prudent to consider one-time costs like hiring time and effort. As we’ve mentioned, assembling a data annotation team begins with recruiting, a crucial step that influences the team’s development and effectiveness. Organizations typically choose between outsourcing recruitment or managing it internally.

Outsourcing Recruitment

• Time: Recruitment agencies can expedite the process, typically taking 2 to 6 weeks to secure a position.
• Cost: Agencies charge a fee based on the position’s annual salary, usually 15% to 30%.

Internal Recruitment

• Time: This method can take 4 to 8 weeks, depending on HR process efficiency and candidate availability.
• Cost: Costs include job posting fees ($0 to $500) and internal HR labor (approximately $55,000 annually or $26 per hour).

The figures provided are approximate and based on data from Indeed and LinkedIn; actual costs may vary and should be aligned with the company’s internal processes. For example, at CVAT.ai, we have automated our hiring process, allowing us to recruit top-notch annotators at competitive rates. We use Remote.com to onboard candidates and are satisfied with this HR platform. Our annotators hail from various countries, including Kenya, India, Nigeria, Ghana, Nepal, and Indonesia.

Considerations for Hiring Relatives

Smaller teams might consider hiring relatives for data annotation tasks. While this can add value in terms of trust and loyalty, it often leads to challenges such as a lack of professionalism and cost. Performance might only meet professional standards if the hiring criteria are aligned with the job’s technical demands.

Management Overhead

Post-recruitment, managing a data annotation team involves handling administrative tasks essential for maintaining AI development standards:

• Paperwork and Compliance: Managing contracts and compliance with labor laws.
• Financial Management: Overseeing accounts and payment systems.
• Work Environment Management: Providing training, managing workloads, and fostering a supportive work atmosphere.

Additional Considerations

• Technology and Tools: Investments in data management and annotation tools can enhance efficiency.
• Team Dynamics: The interaction between team members and management style significantly impacts productivity.
• Market Conditions: Economic factors and labor availability can influence recruitment and operational costs. These elements are often seen as “hidden costs” and vary significantly by organization, affecting overall expenses. They should be included in the final budget considerations due to their potential to significantly impact costs.

Conclusion

What will be the total duration and costs of the entire project? Here we discuss the baseline minimal price, excluding hiring and hidden costs:

• Total Duration: 18.25 work weeks, which is approximately 4.2 months.
• Cost Range: The costs vary from about $122,000 depending on team capacity and other factors like where you are located, whether you hire locally or worldwide, etc.

What else should you consider when reading this article? The calculation for the hiring process assumes linear and consistent recruitment and onboarding, which might not reflect real-world variations. Realistic scenarios may need buffer times for unexpected delays and additional costs for unplanned issues. The provided time and costs assume maximum efficiency. They may not account for variables such as sick leave, training efficacy, and turnover rates, which could significantly impact time and cost estimates. Empirical data from similar past projects could further refine estimating the time taken to annotate objects and onboarding costs. Overall, the presented figures are reasonable but should be treated as approximations with potential for variation based on real-world execution.

That’s all for today. We’ll see you in the next article, where we will discuss the cost of outsourcing data annotation to professionals.