Tailing Text File with Non Blocking IO
Episode 1
What is Tailing text file?
If you ever played any stealth game, there must be an objective which you have to follow a person without exposing yourself. Actually we don’t mind where the person comes from or how far moved for this current position. We only care about his current position and whether he is moving or not. This is the exact game scenario of tailing a text file.
Tailing text file means you have to read the characters/lines which add after the current position and whenever there is new characters added to the file you have read them orderly.
Which File class need for this purpose ?
I am going to use Java for this topic because it has inbuilt classes and methods for this purpose. If you want to write a Java program to tail a text file, it is easy to open it as “RandomAccessFile” object. The reason is if you want to jump over a known position, you can easily move there without reading previous characters in that file.
How to tail the file ? (Overall Idea)
First we need a component (mostly a class) which used to tail the file. This class should have a method which need to run along with the application. Because we don’t know when new lines will be added to the file, But whenever it is added we need to read it. There are various methods to read from a file. we can classify them into two major topics, Blocking IO and Nonblocking IO.
Tailing with Blocking IO
If you google about reading files, most results Google provides come under this category. The thread which invokes the write and read methods will be blocked until there is data available for write or read. The thread cannot do other tasks for blocking period of time.
If you remember the game scenario after you start tailing, if that person stops you need to watch him frequently and wait for his next move. You were unable to move to another place even if you found a side quest because you may lose the person. Yes! you experienced tailing with blocking IO.
Let’s look following code snapshot.
The method Thread.sleep(1000) which used wait current processing thread for 1 second. I used this method to let the program wait for 1 second and check the file whether it has any new lines added or not. If there exist some new lines, length of the file should be increased so I can read the lines by seeking the last read position.
As said in earlier game scenario the program needs to check the file frequently and wait for some new lines. It can’t do other tasks because the thread is blocked.
But there is another problem too. What if a lot bunch of lines come within that 1 second. The program needs to deal with sudden inputs in huge amount. It needs quite amount of memory and ram to process the inputs. It will end up with considerable memory footprint for your program.
Hence my previous code makes my program to use CPU at 99 % (I have 8 core CPU, so each core usage is 12.5%). But my requirement is my program’s CPU usage must be below than 3%. This is the part where program needs Non Blocking IO.
Above method is not recommended due to mentioned problems. Please feel free to share the thoughts about the method. Let us discuss what are the lines caused high considerable memory footprint and Tailer class(Apache commons IO). Finally think about how that game scenario could improved for ease of tailing, because this idea is the key part of Nonblocking IO.
