How we built from zero to one our lab IT infra-structure

Practical advice and discussion on software & hardware options for a genomics healthcare lab

Marcel Caraciolo
genomics-healthcare-systems
29 min readFeb 8, 2021

--

I believe that at your mind you imagined a complex IT environment with a huge datacenter housing several hardware components and bleeding edge technologies at our earlier days as a healthcare laboratory startup.

But the real life can be really challenging or difficult based on your financial conditions and priorities. This means that you have to find the quickest and cheapest setup that you could aim for your beginning lab. We sacrifice redundancy in all areas and select consumer-grade hardwares.

Our first data center — rack mount server at the pile of boxes and one UPS.

In this article, I’ll discuss the components you need to pay attention when you are starting to build your IT infra-structure laboratory, and suggest different implementations options.

The basic components of a laboratory IT infra-structure

Before we discuss the different configurations and options, let's define the main components that we must check when setup our laboratory datacenter.

Computing power and Memory
This is the main horsepower of your lab and consists of the physical server and CPU(s) you intend to use. The more physical processors the better, as well as the more CPU cores too. The focus is to host the intranet solutions from your lab which may include LMS, Firewall, Windows Server with Active Directory, etc.
The amount of memory (RAM) your labs makes available for use is an intrinsic component of our lab. In our scenario, we have a lot of bioinformatics tools and pipelines processing raw data from sequencers, which consumes a large amount of RAM from your servers.

Storage
Your lab needs storage for its Virtual Machine (VM) images , client OSs, shared files from the teams and backups from the applications and databases.

In our scenario we store the sequencing data after processing the pipelines, which will be high demanding, since human genome output can achieve about 180GB (per person).

Networking
This is what connects all your devices in our lab together, so it’s important to be sized correctly in terms of ports availability and speed. It is also important to mention the internet service providers (ISP) , responsible for the bandwidth outside your lab. Depending on your servers capacity and processing power, your samples data might be uploaded to the high performance cloud servers in order to be processed, requiring high speed and slow latency from your internet bandwidth.

Uninterruptible Power Supply (UPS)
A UPS allows your lab to run without power for a short amount of time when the input power source fails. This can be critical for preserving the health and longevity of your lab’s data and keep your critical laboratory equipment keep working until the power source returns or to shut down properly the equipment.

Backup
You should also plan for a backup in case things go wrong for any critical failures (trouble in physical servers, HDs, storages or due to external factors: fires, short circuits, etc.)

Internal phone systems
Your lab needs a private telephone network so your internal company can communicate internally (within their lab) and externally (with the the outside world). You have different communications channels to choose such as Voice over IP, analog or IDSN. Having a PBX phone system at your lab allows you to have more phones than physical phone lines and allows free calls between users.

Firewall
Security is a critical component for all companies that needs to communicate with the internet. Protect your network environment is mandatory, and prevents critical damage on your data or theft. Deploying a firewall solution, it acts as a defensive border by helping to protect your internal network and devices from hackers, as well blocking malware from penetrating your system.

Wireless setup

Wireless connectivity is one of the most fundamental requirements for any growing lab, specially when the lab also have a patient service center to have the exam assessment and the sample collected. Many labs understand the importance of installing wireless access points but don't separate the patient network from the lab network or position their access points without evaluating the lab premises regarding the signal strength or the number of concurrent connections in a limited area.

Smart Access and Surveillance

Regardless of the size, scope, or culture of your laboratory, physical access to the specific rooms in your lab should be monitored and controlled. For instance, the server room or the stock room are one of the most important physical places in a laboratory. Controlling access can help protect against both intentional and unintentional events that may result in significant productivity and revenue losses. Video surveillance also must be considered, specially for laboratories that requires stringent security to protect equipment from theft and other threats.

Printers

Every laboratory environment requires a printing solution for interacting with their internal LMS (laboratory management systems) and general workflows. The lab staff at their daily routine print multiple workload maps and tube labels to identify samples and perform the technical procedures. It is critical for a lab to have a print solution with several label and A4 printers available. The inadequate printing setup can affect your operation as your lab current size starts to grow with many staffs and workstations.

Assets Management

As the laboratory infrastructure grows up, it is mandatory a minimal management and inventory of all machines, computers and equipment in operation. By using a simple spreadsheet you can register all your equipment and keep log of maintenance due date, location and technical information such as model, serial number, etc.

LMS system

A laboratory that performs exams must have a laboratory management system (LMS) for registering, tracking and returning the patient's exam request. Nowadays there are several options for laboratories in any stage with several customer configurations.

Subsidiaries

As laboratory becomes more granular and mature in its business, it is usually common for the business strategy growth to open more subsidiaries in another district, city or even state. In the case of a lab without technical processing which is only responsible for the patient registering and sample collection, a minimal infra-structure is required, such as computers, printers, intranet network and wi-fi setup for the clients. Other concerns arises for the IT such as the communication between the subsidiary and the main laboratory to access systems and voice communication.

IT Monitoring

With the infra-structure growing with a small team of infra-structure analysts, it is crucial to have a tool for monitoring all the servers, machines and network devices in real-time. The goal is to prevent or reduce the occurrence of incidents that can affect directly the laboratory infrastructure. There are several tools available focused on IT monitoring.

So, after this short introduction let’s go straight to the configurations.

Lab setup

The goal is to present our lab setup mixing components both from the enterprise as well as the consumer-level hardware.

Computing power and Memory

As a genomics lab , we require high-end computing nodes for processing the bioinformatics pipelines from our clinical sequencing routines. There are options available on-line using cloud services from providers such as Microsoft Azure, RedHat OpenShift, Amazon AWS, Google Cloud or going local by building a local datacenter. In our scenario due to business restrictions, we started with a hybrid architecture having some services running on cloud and some services locally. At cloud the clear advantage is the possibility of rapid expansion vertically or horizontally of the infra-structure available on these cloud sites and the low costs in hardware maintenance and extension.

At our lab, we have rack-mount servers from 1U to 2U heavily equipped with CPUs , RAM and I/O modules and 2 gigabyte ethernet ports to support failover network scenarios.

PowerEdge DELL Hardware — good low cost solution for entry labs

The recommended features when you buy a server is to have a remote management software, that allows you to remotely manage your server until an operation system boots up, and the a RAID controller for redundancy in case of you experiment hardware failures with your storage.

For all our servers we use high-end virtualisation Citrix XenServer. It provides free versions of its basic virtualisation software, with some enterprise features missing, of course. There is also an open-source version called XCP-ng, the platform based on Xen Source and Citrix® Hypervisor (formerly XenServer).

XCP-ng management center

Virtualization creates several virtual machines (also known as virtual computers, virtual instances, virtual versions or VMs) from one physical machine using software called a hypervisor. Because these virtual machines perform just like physical machines while only relying on the resources of one computer system, virtualization allows IT organizations to run multiple operating systems on a single server (also known as a host). During these operations, the hypervisor allocates computing resources to each virtual computer as needed. This makes IT operations much more efficient and cost-effective. Flexible resource allocation like this made virtualization the foundation of cloud computing. Source: Citrix Article — What is virtualisation ?

Storage

Storage solutions are required for genomics labs for storing the raw and processed data from sequencing platforms as well for any data produced internally and exchanged by the laboratory LMS and staff. There are several options available for storage such as external HDs, cloud or NAS (network attached) storages.

Despite the various alternatives, for a starter lab they continue to rely on traditional configurations such as: direct-attached storage (DAS), network-attached storage (NAS), and the storage area network (SAN). Each approach offers both advantages and disadvantages, but it’s not always clear when to use one over the other or the role they might play in more modern technologies such as cloud storage.

In our scenario, we have NAS storage devices, which enables multiple users and applications to access data from a centralised system via the network. With NAS, users have a single access point that is scalable, relatively easy and cheaper set up. NAS also includes built-in fault tolerance, management capabilities, and security protections, and it can support features such as replication and data deduplications.

A NAS device is an independent node on the local area network (LAN) with its own IP address. It is essentially a server that contains multiple HDDs or SSDs, along with processor and memory resources. The device typically runs a lightweight operating system (OS) that manages data storage and file sharing, although in some cases it might run a full OS such as Windows Server or Linux.

An example of NAS Storage from the QNAP Provider

We also have several external HDDs for any backup in our laboratory machines that don't have network interface, it is important to have some in case of any emergency.

Networking

For a genomics lab, it is quite usual to have a heavy data I/O at the local network. After all, the sequencer must output all the data into local storage at you LAN. Nowadays, with the rapid advance of the sequencing platforms, a usual sequencing run produces an average of 400GB-1TB of data. Considering these requirements, you should buy a 10 Gigabit ethernet switch , or if there are many network machines running at your lab, you will need more than one switch. In this scenario, we cascade them all together using SFP ports, in order to allow redundancy and prevent loop. Remember that all other hardware in you lab should be equipped with 10 Gigabit NICs if you really want to take advantage of full network speed.

HP is a famous provider for high-end switches with Gigabit Ethernet and SFP ports

For a starter lab you can start with one ISP (Internet Service Provider), but as the volume grows and more outside connections will be required for sending genomics data to the cloud, you will require to have at least two ISPs for any network ISP redundancy. You can configure the primary/backup ISP link using specific router softwares or compliances such as Mikrotik, Pfsense, Sophos, etc.

A simple illustration of a Mikrotik Router as a gateway for ISP link load balacing

One useful tip is that some storage devices and servers, you have the option of link aggregation (or port bonding, or port trunking). With this option, you essentially combine more than one ports of your Gigabit device for the equivalent speed of their sums or can be configured as a failover backup port.

Uninterruptible Power Supply (UPS)

UPS failure is one the most common causes of downtime in data centers worldwide. Any equipment at your laboratory that is crucial to its operation, such as the storage node, sequencing machines, servers , and lab routine machines should be connected to a UPS. Operating these nodes on UPS ensures that a sudden power blackout (power is lost completely) or brownout (temporary low dip in voltage levels) does not cause these nodes to crash, which risks file system or hardware damage.

Based on the structure where you laboratory is located you will have the opportunity to build a battery room with large units powering your entire laboratory server room and critical machines. Nevertheless, most of the labs during the facility construction don't consider this into the account, and comes with a simple small room for the data center without empty space to store your UPS units. If that it's you case, probably you will have several units installed close to the equipment that it will be powering.

The recommendations are to select larger UPS depending of the machines that will be consume. Don't forget to notice the maximum energy consumption from some lab machines, that can vary depending on the routine size. More sophisticated UPS include monitoring solutions using serial ports or network interfaces. For example, some units with a remote terminal could display realtime information (battery voltage, charge/discharge current and remaining battery life) about the UPS operation.

Example of a Rack UPS unit at the data center.

Physical inspections are also important, but no UPS maintenance process is really complete without load bank testing to find out how well the system performs when it’s needed most. A load bank test simulates a power disruption by connecting the uninterruptible power supply to an electrical load that matches the load it would need to support in a real outage. The test will quickly reveal any problems or failures, including the UPS components itself and associated battery strings.

Backup

Backup solutions is critical for any genomics laboratory. The laboratory is responsible for storing the raw and processed genomics data from all the sample through all the test lifecycle. Furthermore, the new genomics sequencers only come with a limited space in hard disk capable of storing only the current sequencing routine, that mandates the lab have a larger network or USB device to store the long-data routines. Don't forget to mention the archives produced by the analysts such as workflows, notes, patient reports and consent terms. All this data needs storage and a backup procedure.

Having spent a considerable amount of money at that point for your lab, you need to ensure that no matter how catastrophic a failure is, you always have a way to get your data back. Your two backup options are on-site and remotely.

On-site, can be easily implemented by just plugging a USB disk into your NAS device and setup a backup job to it. As you realise, in such a setup you will have a backup of all your data, but it will physically coexist with the remaining of your lab. If something goes terribly wrong, your backup drive may also be affected. There are plenty of options of backup monitoring tools with automatic and scheduled backup plans (paid and open-source). In our lab, we decided to implement an internal backup solution for backing up the media and databases of our intranet systems due to specific requirements of our business.

Our internal backup monitoring solution — to guarantee that all the scheduled backup plans worked as expected

Remote backups, on the other hand, take longer to be completed (and longer to be restored when necessary), however, your data is securely stored in a remote location and should always be accessible no matter what happens to your home lab. There are many online backup services with affordable plans, even some that provide unlimited data plans.

CLoudberry Backup Cloud Backup Tool — One of the tools used at our laboratory for remote schedule backup.

For servers and NAS storages backups consider buying ones with RAID technologies.

A RAID is a data storage technology that takes multiple physical disk drive components into a single logical unit. RAIDs are designed for two purposes: improved performance and data redundancy. There are several different RAID configurations that have both upsides and downsides so let’s dive into the different RAIDs and which one is the best for performance and redundancy. — Mathew Mister.

For instance, the RAID 1 array which consist of two drives. In this configuration one disk drive is a mirror of the other, meaning they both have the same data on them. The upside of this array is an increase in performance, boasting 2x the read rate of a single disks. However, this comes with the downside of limiting your capacity to 50% of the total disk space.

Finally, for your virtual machines backup, if you have Citrix virtualisation servers, there are interesting options from internal scripts, that can create a snapshot from you VM and produces a XVA image from it or improved solutions such as Xen-Orchestra, an open-source web tool that provides several features to get a complete overview of Xen infrastructure.

Internal Phones

For any laboratory the communication between the staff teams is critical. The basic solution is to start with analog channel with cordless phone devices with easy and fast setup just needing the landline for the main receptor. This solution is a good solution for wireless communication needs.

Analog wireless terminals with easy setup and extension for low cost infra-structure

But for the outside world? If your laboratory doesn't have a dedicated call center or third-party contractor, your phone lines will be overflown with internal and outside calls with clients and providers. As your organisation grows, you will require a system with the flexibility to handle increasing demand, specially when you start to have multi-site company subsidiaries across different locations.

A possible office communication solution is to have a hybrid IP based PBX system, which combines the traditional and IP technologies into one central device, which manage all the communications , reducing costs and improving efficiency. This way, your lab can migrate to IP functionality gradually, allowing you to manage your communications budgets more easily.

One of the biggest benefits to having a hybrid PBX is for remote offices and secondary locations. Calls from one office to the other go through the hybrid PBX, so there are no long-distance charges.

In our scenario we have one PBX Hybrid system at our main office connected with our analog telephone network (PSTN). So all the outside calls are routed to our PBX system and direct them to our internal phones in a specified order.

The Panasonic Hybrid IP PBX system — one of the best cost x benefits at the market

As our laboratory started to grow we have been migration our internal phone devices to IP-based ones, since we don't have to build a specific cable infra-structure for the old analog network and take advantage of all LAN network that we have to place a new phone when needed.

Firewall

A firewall plays a critical role in the network security, keeping the organizations protected from data leakage and cyberattacks. Any laboratory who deals with sensitive data, including patient, genomics and billing information, the lab IT must have a firewall implemented with limited inbound traffic from the internet.

For example, all of your servers, storages and machines that provide services over the internet (web servers, email servers, virtual private network (VPN) servers, etc.) should be placed into a dedicated zone that will allow limited inbound traffic from the internet (this zone is often called a demilitarized zone or DMZ). Servers that should not be accessed directly from the internet, such as database servers, must be placed in internal server zones instead. Likewise, workstations, point of sale devices, and voice over Internet protocol (VOIP) systems can usually be placed in internal network zones.

One option very popular among the starters offices is the distribution pfSense.

pfSense is an open source firewall/router computer software distribution based on FreeBSD. It is installed on a physical computer or a virtual machine to make a dedicated firewall/router for a network. It can be configured and upgraded through a web-based interface, and requires no knowledge of the underlying FreeBSD system to manage. — Wikipedia

A screenshot of pfSense firewall dashboard

It is very robust for any laboratories who don't have an IT security specialist. pfSense also includes network capabilities: DNS, VPN (OpenVPN, IPsec, PPTP, etc), DHCP, NAT and much more. Don't forget to configure the outbound and inbound ports accordingly to your needs, so don't use the default configuration. The web interface is rich with several configurations and customisations available for anyone that wants deep in the firewall.

If open-source isn't an option, consider look after other paid solutions such as Palo Alto , SonicWall or Fortigate. They have several configurations available which includes the router and firewall.

A simple screenshot of Palo Alto firewall Dashboard

Assets Management

Inventory management is becoming a mandatory component as your laboratory grows, with more machines, servers and computers. You can start with a simple spreadsheet containing the list of all your equipment, including information such as serial number, model, purchase date and even an internal unique number (SKU) that represents that equipment as unique at your inventory.

There are also several options of inventory management systems available at market that supports barcode tracking, scheduling maintenance audits, etc.

At our lab we decided to develop an internal solution called Genomika-labs, which is responsible of tracking and registering of all equipment located at any of our units. You can find further information about our solution here.

On-premise x cloud

Deciding between on premise and cloud environment is a complex process, whereby a number of factors come into play. For starters, the business requirements must be properly analyzed to tip the scale toward one approach or another. Once decided, there is a second decision to choose a reliable technology partner, which can be confusing as there are multiple vendors offering services of varying levels. There are several articles discussing these options.

At our laboratory, based on our IT security premises, we have deployed a hybrid architecture, which there are some critical services hosted at our intranet with limited access to our staff , and other websites/portals for our partners and patients on cloud.

With on-premises infrastructure all services are run, managed and maintained from within the business on our own hardware and servers. Some businesses may choose to do this for compliance or security reasons, although often find it’s more expensive and offers less flexibility. Cloud services provided by global businesses such AWS, Azure gave us the power of enterprise level infrastructure at a fraction of the costs. Benefits such as per minute consumption-based billing benefits businesses that only want to pay for data and services when want to use them.

An old infra-structure architecture of our lab with hybrid on-premise and cloud services.

Devops Infra-Structure

If your laboratory decides to have IT team developing in-house softwares for solving some laboratory business problems, probably you will come to face the challenges about the infrastructure that all services are run and managed. In the previous section, I detailed the differences between on-premises vs cloud infrastructure, but independently of the selection, you will want to achieve business outcomes faster, and improve the IT team productivity to get things on-line faster, you must have a Devops team.

DevOps brings typically siloed teams together — Development and IT Operations. DevOps promises to help teams work collectively and collaboratively to achieve business outcomes faster.

The Devops team will be the guardian of all the infrastructure that runs your in-house softwares and have hardware and programming soft skills. One of the tasks of the devops team is define and maintain the tools that supports the applications development, testing and deployment. Many organizations are using Docker and Kubernetes technologies for their production environments.

Docker is an open-source project that automates the development, deployment and running of applications inside isolated containers. Containers allow developers to bundle up an application with all of the parts it needs, such as libraries and other dependencies, and ship it as one package.

There are orchestrations tools that help the team manage these containerized applications , by orchestrating the complete application life cycle based on given specifications. Currently, there is a large variety of container orchestration tools.

At our laboratory we use two orchestrators : Rancher , an open-source alternative for deploying and managing containers in productions, it supports Kubernetes, Mesos and Docker Swarm; and Dokku, an open-source platform as service (PAAS) powered by docker, that also helps the team build and manage the lifecycle of applications.

Screenshot of Rancher Server's Dashboard

We illustrate one of our deployment pipelines for Dokku below, where we host all of our code in Gitlab repositories and when deployment signal starts from the development team, it queues continuos integrations (CI) runners that runs procedures that performs tests, build and deploy the updated code at our orchestrator, in this case the PAAS Dokku. Once the application is deployed, the webservice is restarted automatically with minimal delay. As you can notice all those steps are automated so you can deploy easy any application at your preferred environment.

Illustration of one of our pipeline's deployment using Gitlab CIs + Dokku PAAS.

Laboratory Management System (LMS)

A laboratory management system is critical for genomics laboratory that have to handle the management, tracking and workflow of the genomics sequencing tests. By using it, your lab can automate workflows, integrate instruments, and manage samples and associated information.

But when decide to buy or build one LMS ? There is an interesting article from Illumina discussing the benefits and drawbacks between the two options. Based on our experience, at the time we didn't finde any qualified LIS/LMS vendor specialized in genomics workflows. There were several brazilian vendors focused on biochemical and medical images (PACS) exams. We decided to build our own LMS focused on genomics workflow called Gensoft and integrated with other thirdy-party LIMS that focused on the patient's assessment and report delivery. Once we grew, we started to personalize our patient and doctors relationship and built our own LIMS for the all patient's lifecycle, replacing the third-party LIMS.

Some screenshots of our in-house genomics workflow LIMS : Gensoft

Was it a good decision ? Based on our needs, yes. As we started to grow and define our patient flow since from the first contact to our lab until the report feedback, we needed to make multiple customizations at our LMS interfaces, which for vendor was inviable or when viable quite expensive for us.

You can have a glimpse of our LIMS here at this article I wrote about our previous experience with a on-line portal for COVID-19 tests scheduling.

For starters labs looking after to LIMS options, there are options in portuguese at Brazil such as Pixeon Pleres , ND Softlab and Neovita Sallus.

Subsidiaries and communication

Laboratories with mature business operations will generally come up to opening new subsidiaries in different districts or states. Open a new subsidiary lab comes with new challenges for the IT team. There are decisions to make about the local infrastructure and voice/data communication between all the subsidiaries and the main laboratory.

We won't focus in this topic about the local equipment required to operate the subsidiary because it depends on several factors such as size, local team, sample collection and local technical processing. We will bring the communication discussion, because it is a common issue between all the laboratories in development.

The communication between subsidiaries and the main laboratory requires that the networks have to transfer data over public internet providers. It is critical a virtual private network (VPN).

VPN is a technology enabling a logical extension of a network or a sub-network of the organisation through the addition of workstations or sub-networks outside its physical boundaries. More specifically, employees working from home will be virtually acting within the internal network of the organisation, or between two remote sites or even a world apart, and sharing the same network.

It guarantees data integrity, confidentiality and identification of who is accessing the data. We had to set up a site to site VPN between our VPN server and a VPN client using protocols such as OpenVPN or IPSEC.

In each subsidiary we have a Mikrotik router with VPN Site client connected to our VPN server. Mikrotiks routers became quite popular among the IT hardware teams at Brazil, because of their multiple network features available (firewall, DHCP, DNS and VPN protocols) for a low cost price.

A example of MikroTik routerboard with SFP and Gigabit Ports

With the VPN established among all the networks, we now have the possibility of a staff located at one of our subsidiaries in São Paulo accessing our intranet web services hosted at the main laboratory at Recife as a local network. Furthermore, with a IP phone system set up in each subsidiary we also have a VoIP VPN enabling secure phone calls without call rates.

And how we managed the external calls ? We also needed to reduce costs and have a unique national telephone number so can people could call from anywhere at Brazil with no charges the person placing the call. We researched several toll-free numbers providers (0800 , 4004, 3003 prefixes) and came to Atende Simples provider, which provides toll-free numbers to business and have several attendance solutions for recording, reports and forwarding services that directs the call to one line, several lines or a call center.

Screenshot of Atende Simples Solution Provider Dashboard

For illustration purposes , we present below our VPN communication architecture that we set up at every laboratory and branches.

An overview of our VOIP/Data communication through all subsidiaries using OpenVPN with Mikrotik routers .

IT Monitoring

As your infrastructure grows , it means more active machines, computers, servers , network providers, devices on-line in your laboratory, you will eventually get used to critical errors and warnings due to malfunctioning, bugs and errors. Investments in monitoring tools to detect problems and keep an eye on the devices in your network, so that you can take appropriate actions in the event that it becomes necessary.

For network monitoring there are several tools available, from open source applications to those that require comercial licenses. Two popular of such monitoring tools are Zabbix and Nagios.

At our laboratory we have implemented the Zabbix monitoring tool, the easy installation and setup with web interface were some of the reasons we went with it. Nowadays we have a TV with a live dashboard powered by Grafana-Zabbix, which allow us to visualize monitoring data from Zabbix

Screenshot of Zabbix DashBoard
One of our Zabbix monitoring dasboards live at our IT monitor.

For bug monitoring in realtime applications, we have implemented the web service Sentry. Sentry Python error reporter tracks errors and exceptions that happen during the execution of our applications. All our development team has access to this monitor and it is critical to check recurring issues and crashes.

Screenshot of Sentry Dashboard

Another tool that we also developed and empowers our laboratory to improve more engagement with our internal clients ( lab , development, customer teams , etc) is a simple status page with all our services communicating incidents. Geninfo is web tool focused on communicating various types of system statuses and incidents to all our teams.

There are several alternatives on-line such as Cachet, Status.io or Uptime Robot , but we didn't find one focused on infra-structure and network components such as Internet link, Active directory, printers, etc.

Our internal status page for reporting incidents at our laboratory infrastructure

WIFI / Digital Access / Surveillance

For the laboratory wireless connectivity, you will need wireless access points. When you start building your lab, you decide to put your wireless devices anywhere with different specifications and vendors. Soon as more people join the wireless network, the problems come with loss signal and poor connection with limited bandwidth. The source of the problem must be investigated, but we have some clues: the wireless initial setup was not planned resulting into wireless issues affecting your laboratory routine.

Invest time and plan a wireless integrated solution for all your lab. Prefer solutions with the same vendor and check the specifications of the routers/access points.

At our laboratory we decided to build our wireless infrastructure with Intelbrás access points. They are long range routers managed by a smart control software, with interesting features such as easy firmware updates and configuration replication at all other Intelbrás wireless devices.

Wireless Access Point from Intelbras
Wisefi: Smart Application Management for Wireless Devices powered by Intelbras

For access control into specific laboratory locations such as the IT servers room or the inventory room, you will need smart solutions that includes electric door strikes and card/biometric readers.

Electric strikes are electromechanical locks that are placed on the inside of the door frame. When electrical power is supplied to the electric strike it holds the latch or lock bolt in place, and keeps the door securely locked until the release system is activated. An access control device is used to trigger the electric strike and release the lock bolt or latch. Release systems for electric strikes include electric key cards or biometric readers. At our laboratory we use access control devices with RFID and biometric for entering into specific locations. There are several vendors available with a range of features and options. Look for ones with support at Brazil with network support and integration with CRM softwares.

Example of biometrical and card reader used to release the electromechanical locks

Although they may seem innocuous, labs, testing facilities require stringent security to protect expensive equipment from theft, and other threats. Video surveillance and security cameras are a valuable security tool for laboratories and can help keep intellectual property, experiments, equipment and researchers safe. There are several solutions for video surveillance nowadays at Brazil. One of the popular options among the laboratories are the surveillance kit from Intelbrás which includes Infrared cameras and a DVR with 2–4TB of video recording storage.

Intelbrás solution including cameras and DVR.

Active Directory

Active Directory is a helpful solution for medium and large sized organizations, from which IT administrators can manage from a single point network resources and their associated security objects.

One of the great benefits is the single sign-on to allow access to network resources located on any server within the domain. So the lab staff is identified and authenticated by Active Directory once. After this process is complete, the user signs on once to access the network resources that are authorized for, according to his or her assigned roles and privileges within Active Directory.

Active Directory simplifies resource location by allowing files and print resources to be published on the network. Publishing an object allows users to securely access network resources by searching the Active Directory database for the desired resource. So your network printers can be all managed from the Active Directory including drivers and print queue jobs.

There are, of course, some issues regarding when to install Active Directory at your lab. It can can be expensive as you will need Windows Server Edition licences and you may need to upgrade the hardware on the server so it can run Windows Server editions and the wrong set up can lead you time and money to set it up again. It also means high maintenance and infrastructure costs.

Screenshot of Active Directory Management Software running in Windows Server

Printers

When it comes to printing, there are several options available. One decision you will have to work through is whether local printers or network printers will better serve the needs of your laboratory.

As your laboratory grows, it is common to lave local printers installed via USB cables to specific computers. But buying and maintaining multiple printers is costly in and of itself and the problem of the staff productivity will be impacted when a local printer is busy or goes down.

Network printers, on the other hand, offer flexibility and efficiency to all teams in the lab, they can operate with multiple platforms and a variety of different printer types (laser, inkjet, thermal, etc). This means users throughout the office can access network printers regardless of the platform in use on their device.

As a downside, with multiple users capable of accessing the same printers, projects can be delayed when the printer is already in use. Additionally, if the network goes down then printers become inaccessible. In overall, we believe even with these drawbacks, the network printers are better for today's laboratories.

Our recommendations for any laboratory is to start with native network printers if it's possible or rent ones with printing solution companies . If your printer infrastructure have some only local printers, you can purchase a USB print server which can serve a USB printer as a network printer.

Example of a USB Print Server — it converts USB connection to a ethernet connection
Example of a thermal label network printer

Our recommendations for any laboratory is to start with native network printers if it’s possible or rent ones with printing solution companies . If your printer infrastructure have some only local printers, you can purchase a USB print server which can serve a USB printer as a network printer.

Finally, if you have Active Directory (AD) implemented at your infrastructure, you can have the benefit of don't have to install the printer manually on every computer at your lab. By adding to your AD, if the printer is replaced, you can remove it without people trying to print to something that isn’t there. You can also apply the computer to only certain computers that make sense, or set the default so users don’t accidentally print to the wrong printer.

HelpDesk

Helpdesk management solution is critical as your laboratory team and park grows and your IT team is the same. How can they handle all the support calls for complaints, lost passwords, service downtime, even employees asking their in-house IT guys to assist with issues concerning tools like computer hardware, software systems, among other things.

Considering the sheer number of help requests, internal or external, the IT laboratory have to deal with on a daily basis, systematically keeping track of everything — to ensure all requests are addressed in a timely manner and effort duplication among customer-facing personnel is eliminated — can become a seriously formidable task.

The helpdesk solution, or issue tracking system, enables help desk operators to keep track of user requests using a unique identifier, easily find solutions to common queries, prioritize tasks, and so on.

There are paid and open-source alternatives. One we use at our lab is open-source named GLPI. It is a service desk system running as webservice allowing your users create tickets with the issues and follow ups until the final solution.

Screenshot of GLPI issues board

DataCenter

A typical server room contains the infrastructure necessary to support the network and computing systems. This includes design features when building your room like environmental controls, fire suppression systems, cable management solutions, power systems redundancies and airflow planning. Server rooms also could feature a raised floor that helps facilitate better cooling and cable management.

When you start building your server room, the budget is generally limited and you may neglect the critical features mentioned above, resulting in high maintenance costs or have to redo all the structure again because of bad installation or inadequate cable organization. Based on our experience, we quite believe that those items should be considered and can avoid time and money in the future.

Server racks house physical hardware such as the server units , switches, routers , and other network equipment. Prefer racks that feature their own power circuits, cabling solutions, it will be easier for you to handle and move around the hardware installed.

An example of server room : structured cabling and open-spaces for a better cooling. It reduced almost 20% of maintenance time.

The right temperatures are essentials for every server room. This is because of the sophisticated devices that are housed inside. To maintain the right temperatures, you are required to meet all the requirement for air conditioning. A good air-conditioned environment needs to have an air conditioner and the right sensors to trigger an action whenever temperatures are lower or higher than the recommended levels. You also have to worry with a backup conditioner in case of the failure of the main supplier.

At our laboratory we have a backup portable air- conditioner unit that activates during any emergency in case of the main one fails. It is important that you have an exhaust hose, so the portable air conditioners pull in warm air, cool it, and expel the cooled air out of the front of the unit and the remaining warm air and moisture vent out of the room you are currently cooling.

To monitor the temperature we have an in-house monitoring solution with a NODEMCU board connected to wi-fi network and sends the temperature data to cloud using webservice API that stores the temperature and datetime of the collection into a data store in order to show the data. We have detailed the monitor in a previous post of this blog.

Some screenshots of the sensor ESP8266 working in one of the servers rooms. The TV dashboard showing green when the temperature is in the proper range and red screen showing the alert when the temperature is out of range.

There are also commercial solutions for temperature monitoring available at the market such as Telemetric, a brazilian company located at Recife which have the hardware + software solution with sensors connected to a 3G internet chip to send data over the internet into a real live dashboard with several configurable alerts.

A simple example of Telemetric dashboard presenting in details the temperature variation in a room (Photo from Telemetric Website)

Concluding remarks

Running a laboratory infrastructure will lead you to experiment with software installations, hardware configurations, distributed systems and a lot of IT planning. It is a hard work but can be extremely fun and educational. If you start building your genomics laboratory do not be discouraged by the complexity or the cost. Always start small and expand your lab based on your needs and growing pace. All different lab setups have something to teach you.

We have discussed some practical guidelines on software and hardware options for a genomics healthcare laboratory, which might be useful for IT managers and infrastructure lab team.

We would like to mention some people that helped us during this journey at Genomika Diagnósticos from 2014–2020.

  • Our current infrastructure IT team : Lucas Eduardo, Victor Nunes and Leonardo Silva. These three guys makes all this infrastructure park working together!
  • Our alumni team who helped us during this journey of evolution and learning about IT management: Fábio Alex, Filiphe Vilar, João Victor Uchoa, Rayana Gonçalves and Aécio Ventura.
  • Our CEO João Bosco Oliveira who believed in this work and all lab team, our real customers!

--

--

Marcel Caraciolo
genomics-healthcare-systems

Entrepreneur, Product Manager and Bioinformatics Specialists at Genomika Diagnósticos. Piano hobby, Runner for passion and Lego Architecture lover.