Movebot data migrations are fast-but how fast is fast?
Speed is one of the most important parts of a data migration. Data movement needs to be fast to be worthwhile. And when it comes to speed, Movebot’s one of the fastest cloud data migration tools around. So how fast is fast?
Movebot can push through 100TB per day on a single migration.
There are a lot of factors that can limit that. Max speeds are important, but what you really want to know is how fast will MY data migration be?
Depending on what systems you’re migrating to and from, you can expect Movebot to transfer between 2–5TB per day. We’ll go over some scenarios to help you know when you can expect to be at the lower end, or when you can expect to push 5TB or more per day.
Understanding Speed
When you’re driving, you stick to the speed limit so you don’t get pulled over and come to a complete stop. Those speed limits are put in place to keep you safe… But sometimes they’re unnecessary.
Movebot on its own is like the Autobahn. And like the Autobahn, having no speed caps isn’t an accident waiting to happen . There aren’t any limiters in Movebot, because we want you to come in, do your migration, and be done with it. Our goal is for Movebot to be the fastest cloud data migration tool around so your transfers are quick and easy with no hassle.
So if Movebot isn’t limiting speed, why aren’t all migrations going at maximum speed? The main factors that limit speed right now are provider limits, hardware limits, and bandwidth uplink.
Provider limits
One of the most common hang-ups for data transfer speed is provider limits. If Moveboot is the Autobahn, then provider limits are like taking an exit and being slowed down to a crawl through traffic.
Types of provider limits
When we speak about provider limits, there are normally five things at play that will ultimately limit your migration speed. These limits are outside the control of Movebot normally.
More about Rate Limiting
Rate limiting is an interesting case, because on the surface it might seem like it’s simply limiting users to deter them from leaving. But its primary use is for protection against bots and DDOS attacks. Rate limiting helps protect websites or applications from increases in load (sometimes with malicious intent) that could degrade and impact the platform as well as brute force attacks.
When it comes to API calls, rate limiting is sometimes used to stop an API from being overwhelmed. If too many calls are made at once, the API may disconnect completely and stall services that rely on the API. Some providers include rate limiting to throttle the connection so that it doesn’t disconnect completely, but slows down instead.
Other times, rate limiting is for cost and resource controls. It can help companies estimate costs better and have them less likely to spiral out of control. This reasoning doesn’t benefit the consumer, however, and instead protects the developer from unexpected cost spikes.
Hardware limits for On-Prem Migrations
While Movebot has some ways to work around some provider limits, hardware limits are a different story. Since Movebot simply adds more machines to your job if the transfer requires it, compute power isn’t an issue with the software.
However, if you have a 15-year-old server with data you want to move to the cloud, that machine needs to process everything on its end which can take time. It’s similar to moving with a truck that has a speed limiter on it. Even on the Autobahn, a truck that physically can’t go faster than 60 MPH is stuck at that speed, regardless of the speed limit.
To help alleviate this, we normally recommend using a Virtual machine to run the migration of on-prem servers.
Recommended specs for migration virtual machine
But ultimately, if the server is smoking, the migration is going to crawl.
Bandwidth Uplink
More important than server performance, is having a reliable internet connection, and not just for watching Netflix. If the connection is slow because you don’t want to lose your $10/month dial-up plan grandfathered in from 1994, your migration will be slow going.
If you’re planning a transfer with a lot of data and have a server connected with anything under 100mb/s, it might be worth upgrading the plan if possible to speed up the job. If you aren’t able to do that, keep in mind that your connection speed is going to limit your transfer speed.
Based on our experience, transfer speed in Movebot tends to average about half of the theoretical uplink. The table below shows how different speeds affect how long it should take to move 10TB of data.
If you are unsure what your current bandwidth is, you can run a free speed test .
Block storage vs File storage
When it comes to speed, provider limits can greatly affect how fast your transfer is. But all of those are assuming that you are using traditional file storage. Block storage is an exception to the rule and can be significantly faster.
By “block storage”, we’re referring to S3 based storage. When Amazon developed S3 storage, they changed the game. Now, cloud platforms are following the trend and building block storage engines that conform with the S3 standard or come pretty close. Many have already made the transition.
The most noticeable feature with block storage is performance. In this case, speed. It’s fast. Blazingly fast (most of the time). Because it’s so fast, migrating into and out of block storage is blazingly fast too.
Block storage typically does not rate limit and performance can be scaled linearly. This allows the potential for incredible speeds. It is not unheard of for Movebot to transfer over 100–200TB of data per day on an S3 to S3 (or similar) migration.
Getting up to speed
If you want to get 5TB/day or more from your migration, there are a few settings you can optimize in Movebot that emphasize speed. One thing to take note of when setting up the migration is the geographic region; make sure to pick the one closest to where your data is stored for max speed.
As for optimization during a migration, as a next-gen cloud data migration tool, Movebot handles all of that for you. If it notices an opportunity to speed up your transfer, the systems will take advantage of the optimizations automatically. Once you’ve hopped on the Autobahn, you can let the driver handle the rest for you!
Stats, Metrics and The Performance Advisor
After you’ve connected the platforms and begun your data migration, you can see stats about the transfer right there in Movebot. The system collects stats like amount of data transferred, errors and failures, and much more. Summary stats give you an overall picture of the migration, as shown below.
For checking and debugging performance issues, the Performance Advisor is your best friend. Available under your migration project, it provides real-time stats as well as recommendations and an assessment on what is limiting your migration.
Along with the summary stats and Performance Advisor, Movebot has graphs with real-time stats on the following:
- Rate limiting errors/warnings
- Average upload and download time
- Data transfer rates
- Errors and error rates
- vCPU usage for the moveworkers
This means you can pop your head in at any time to check your stats and see how much time you can expect the rest of the job to take-often less time than it says.
And, if you’re in the midst of a migration and aren’t sure where to look for the information you need, or transfer speeds aren’t as fast as expected, feel free to reach out to us . Our team are experts and have a few tricks up our sleeves.
Ready to try a faster data migration solution?
At Movebot, when we say our data migrations are fast, we mean it. You can expect to move 2–5TB per day, but it could be much higher moving between certain platforms with an optimal setup.
Ready to move with a faster data migration tool? You can make an account and connect your storage for free, including running a pre-migration scan to see how much data you have and learn about potential concerns before the transfer. If you’re planning a data migration or are considering one somewhere down the line, give Movebot a try today.
Originally published at https://www.movebot.io.
