Tuesdays at Virginia Tech
Randy Marchany, Twitter: @randymarchany
Years ago, a fellow sysadmin and I learned how to juggle. We started out with lacrosse balls (heavy, good bounce properties) then to clubs and torches. We didn’t do chainsaws but we did a frying pan, torch and an egg. It was fun and relaxing at the same time. A couple of us formed a small group called the Let ’em Drop Jugglers. There’s a certain rhythm to juggling and when everything works, it’s smooth as silk. Of course, there are lots of drops and one learns how to recover quickly from those mistakes. Sometimes those mistakes happened when no one was looking but sometimes all eyes were on us when things went south.
I also played in a band for a long time and we toured the US, UK and Europe. We retired a couple of years ago but those guys are my brothers. Playing in a band requires public speaking skills, practice, planning, execution, luck and most important of all, teamwork. People would come to us after concerts and say you guys sound like one unit and make it look easy. We thanked them and explained it took a lot of work to make something look easy to do. I was a grad assistant volleyball coach and a club volleyball coach. I got management training coaching the players and developed political techniques dealing with their parents.
I never realized how those skills would help me in my current role as the IT Security Officer (ITSO aka CISO) of Virginia Tech. I became the CISO in 2010 when my former boss, Wayne Donald, retired. He’s a great friend and still a mentor to me. I’m a “lifer” in the EDU IT world. I’ve been at Virginia Tech for 45 years starting as an IBM Systems Programmer, then doing microcomputer programming (the 80s version of IoT) automating data acquisition for research experiments, became a VAX system manager and finally a Unix (Solaris, AIX) sysadmin. I got into cybersecurity in 1991 when one of my servers got hacked. I discovered a small startup called the SANS Institute and wound up joining them in 1992. Virginia Tech created the IT Security Office (ITSO) in 1998 and I joined that office in 2001. “Office” is probably an exaggeration since the ITSO members were Wayne and myself at that time. Today, the ITSO has 8 full time analysts, 4 graduate research assistants and 3–5 undergraduate student workers.
So, what’s my job? Our former CIO told me my job was to “run the cyber defense of the University and change the culture of the user community by making them aware of good IT security practices.” All of the other CISOs who contribute to this series have the same task. I’m glad I can ask them for advice on the wide variety of topics we need to know in order to do our job — policy/standards, risk management, technical controls, public speaking skills, building teams, fostering trust. I’ve learned a lot from my peers and for that I’m thankful. It’s a challenge for all of us in this position. We have to be able communicate at the management, technical, end user, research, instructional level AND be able to translate an idea from one level to another. It doesn’t get more exciting or challenging than this.
My role is a little unique here at Virginia Tech. Part 1 of me is the CISO. Part 2 of me is the Director of the IT Security Lab (ITSL). Part 3 of me is teaching as an Associate Professor of Practice for the Electrical and Computer Engineering (ECE) department. The ITSL was created in 2003 as part of the ITSO to provide a teaching and research environment for undergraduate and graduate students interested in cybersecurity. I didn’t care what major they were in as long as they were interested in cybersecurity. We enlisted a full professor in the ECE department to be the committee chair for graduate students in the lab. We have 4 fully funded graduate research assistants (GRA) working in the lab. My goal was to provide a way for students to get hands-on experience working with the ITSO analysts to solve real time problems. The ITSL follows the “teaching hospital” model. The ITSL has graduated 14 PhD, 14 Masters students and obtained 3 cybersecurity patents so far. Our CIO is very supportive of establishing this partnership between the Administrative, Academic and Research wings of the University. We gain insight into the issues confronting IT and these processes. I can talk to faculty because they know I teach classes. I can talk to researchers because they know about our lab. I can talk to Administration units because we are one of them. These interactions help us design a security program that fits our culture.
Morning
I use the term “morning” loosely. I smile when I read about my peers who get up at dawn to exercise and get ready for the day. Folks, I’m not a morning person. If I see 5am, it’s from the midnight side of the fence and the end of my day not the beginning. I became a night owl when I was a student and the mainframe (remember, I’ve been doing this for 45 years) was available to get results within 2–3 hours and not 1 day. Being a musician reinforced that life schedule. I get up between 0815 and 0830, decide what type of Hawaiian shirt and shorts to wear and usually arrive at my office by 0915 (pre-COVID). I’ve been working from home since March so I’m usually online by 0900 and start to read my emails. It’s a weeding process where I do mass deletes of cold call vendor emails. I have a notepad next to my keyboard and I start a list of things to do based on the emails. These tasks range from answering a policy or technical question from a departmental IT manager, responding to a request from my CIO, doing a security awareness presentation for a University group or authorizing the disconnection of a compromised host. Once I have my “task list” ready, I place a silent bet with myself to see if I’ll actually get to complete everything on the list by the end of the day. My biggest enemy is back-to-back Zoom sessions. There are days when I start at 9 and end at 6 doing nothing but Zoom sessions. Talk about a draining situation!
The Red, Blue and Risk team leaders will let me know if there’s something that requires my attention. I usually check some of our sensor dashboards looking for trends. I check the inbound/outbound-connections-by-country to see if we’re being heavily scanned by hosts from foreign countries. I also check our connections-by-port display to see if there’s any correlation with the connections by country. The next dashboard I check is our “DNS Firewall” display. It’s not a firewall in the technical sense. It’s a DNS-RPZ setup. For me, this is one of those “clue” sites that helps us find compromised machines in our network. So far, it’s a normal day. “Normal” in the sense that we’re seeing heavy scanning from mostly from Russia. Our internal Red Team headed by Brad found an internal web site with an XSS issue and submitted a trouble ticket with the owner. All in all, it’s a typical day in the neighborhood. I try to check in with our Blue Team and Risk group via our internal Slack channels.
Tuesdays are my light day as far as meetings go but I taught a SANS Institute class (SEC 566–20 Critical Security Controls) last week so I have some catching up to do. Fall semester starts next week and I’ll be blocking out some time to do class prep for a graduate class I’m teaching this semester. One of the reasons I enjoy teaching these classes is that it keeps my technical chops up to date. This week I’m reviewing my Canvas class site with one of my graduate students. We’ve been updating all of the lecture slides, preparing the homeworks, and getting the Virginia Cyber Range exercise areas created for the class. Fortunately, I don’t have to change the delivery mechanisms for the class in today’s pandemic world. This class has been 100% online since its creation 10 years ago. I have 85 students registered from all around the country. Even though the class is completely online, I like to schedule a weekly evening zoom session where I can take questions and do some in-class exercises with the class.
Lunch Time
Lunch time is no time to do anything serious for a while. Pre-pandemic routines included walking to downtown restaurants and running some errands. I’ll stop by our office area once a week just to make sure everything is in one piece. I make it a point to get out of the house by driving to the downtown area, grab some takeout lunch and sit in one of the outside areas and get some fresh air if the weather is good. My staff knows this routine and on occasion, they’ll drive downtown and meet me for lunch observing social distancing. It’s nice to actually see someone and not through a Zoom screen. After lunch, I hop on my bike and ride a couple of laps around the main campus drillfield. It’s a short 1.75 mile loop but it’s good to get some exercise. Once that’s done, it’s time to head home.
Afternoon
Back in the home office, I have to review all of our lab graduate research assistant’s (GRA) contracts, get the paperwork signed (hurrah for digital signatures!) for a new GRA who’s joining us this week. My Risk Manager, Amy, contacts me to update me on a vendor security assessment. It’s not looking good for that vendor. I need to setup some Zoom sessions with the ITSL GRAs to catch up with them. When the students come back, my office usually does a couple of orientation sessions for graduate teaching assistants (GTA) reminding them of their responsibilities to protect the student data they’ll be handling. There are over 250 GTAs and they work in just about every department on campus. This year, our presentations will be virtual. I’m not worried about this transition because the analyst in my group that does this presentation is a really good and engaging speaker. I know Steve will do well in the online format. I finish reviewing 3 short videos I recorded as part of the New Department Head orientation process. My talks were on 6 university IT policies and accompanying standards.
Speaking of standards and policies, we’ve been updating our IT policies and standards to address new technology and operational changes in our world. I work with our Executive Director of Policy and Strategic Engagement on these updates. We bounced policy edits back and forth working on the wording until we think we have something ready for the review process. This week we updated our Acceptable Use Policy and Standard. We clarified the “reporting a violation” section of the policy.
I have a Virginia Cyber Range Executive Committee meeting in the afternoon. These meetings are informative and it’s encouraging to see the wide acceptance of the Cyber Range in the k-12, community college and 4 year universities in the state. While the VA Cyber Range is for Virginia schools, its twin, the US Cyber Range, offers the same services to anyone in the US. Virginiacyberrange.org and uscyberrange.org are the sites you should check out sometime. After this meeting, I have to create some presentations for upcoming webinars I’ll be doing in September. I’ve got 4 in the works. I’ll get one down and then start the practicing and refining loop until it feels smooth.
I try to reserve a block of time to read some of my favorite bloggers’ latest entries. Lenny Zeltser has great info in his blogs about CISO and other issues. I try to check his site weekly and John Strand offers great content at his company www site. Twitter feeds also provide some good information about vendor products and the latest trends.
Evening
The work day for me stops around 6pm when my wife gets home from her job. In the pre-pandemic days, Tuesdays were Geek Nite where a bunch of sysadmins, netadmins and security guys would go to a local pub after work and have a few beers and dinner. We’ve been doing that for 20 years now. The bartenders like to tell people to turn off their cell phones if they walk by our table. We’d argue about the most mundane geek stuff or draw up some new thing on a bunch on napkins. For now, Geek Nite is a Zoom session. It’s just not the same.
My wife and I will watch the evening news, catch up on the day’s events, walk the dogs and then I’ll go either on a bicycle ride or motorcycle ride just to get out of the house and clear my head. There’s a nice 11 mile bike trail near campus that is always a good ride. After the ride, it’s dinner time and then I’ll try to get some practice time on the piano, hammer dulcimer and fiddle. Later in the evening I’ll read history, action thrillers, or watch dumb movies on TV depending on my mood. We’ll take the dogs out for a late evening walk down our street and back and call it a day.
Some thoughts
I know that this job goes from a normal pace to hurricanes and tornados hitting at the same time. We have to juggle lots of things, be prepared to deal with drops, be a good team leader and member. Never be afraid to say “yes” but more importantly know when to say “no”. Every day I learn something new and hopefully from my mistakes. The most important task is to keep a good perspective on things. For that, I rely on my staff, my peers in the EDU world. I learn from them constantly. Stay healthy and safe.
