How NVMe storage powers government analytics

In 2019 alone, government agencies made around $175 billion in improper payments. While integrated data and modernized data storage solutions can help mitigate this loss, many agencies continue to rely on legacy storage systems, where information is not only siloed and prevented from serving citizen needs in real-time, but data is placed at an increased risk of unintended loss and security breaches.

As technology rapidly advances and e-discovery, compliance and analytics burdens multiply, government agencies must modernize their storage solutions for maximum reliability.

Non-volatile memory express solid state drives provide the needed boost in performance and low latency to tackle these issues. As a result, NVMe has gone from being the new kid on the block to the dominant SSD interface in a few short years. But adding NVMe SSDs to legacy storage systems is like putting new chrome wheels on an old car. It looks better but performs the same. This is because legacy storage systems were designed to support a hard disk drive using SATA or SAS protocols, not an NVMe SSD.

When data grows, application performance must be increased just to maintain the status quo. Adding more compute and storage resources will solve the problem, but it is costly and only minimally better. Because SATA or SAS interfaces for flash don’t access the data fast enough, agencies should look to high-speed NVMe SSDs.

But there is a harsh reality. Government infrastructure (servers, storage, networks) has a new bottleneck – NVMe storage networking.

Adding NVMe SSDs to existing fiber channel storage arrays will not gain any throughput improvement because the storage array ports and network fabric is dramatically slower than the aggregate throughput of NVMe SSDs.  (See table above). 

Upgrading to the latest fiber channel networking technology (32Gbps/port) requires new storage array controllers, new fiber channel host bus adapters (HBAs) and new fiber channel fabric switches. A 48-port fiber channel switch alone is approximately $166,000.

32Gbps fiber channel HBAs will cost $2,463 per server. At 48 servers, that's an additional investment of $118,224. So, $284,224 will buy a 32Gbps fiber channel network capable of handling more NVMe SSDs, but that will require new storage array with 32Gbps fiber channel controllers. 64Gbps fiber channel (still in early infancy) will require another fork-lift upgrade. The roadmap for fiber channel with > 100, 200, 400Gbps networks is hard to find from any supplier.

Alternatively, more organizations turn to NVMe-over-fabrics as a modern approach to storage networking. NVMe-oF leverages NVMe SSDs using the Ethernet network transport. NVMe/TCP is an easy deployment where 100Gbps Ethernet networks are easily half the cost of 32Gbps fiber channels. A 48-port 100GbE switch is approximately $87,500 and most modern servers are already equipped with dual-port 100GbE network interface cards.

NVMe-oF can also be implemented for RDMA access across Ethernet, a protocol known as RDMA-over-converged Ethernet. While RoCE requires specialized NICs (R-NICs) and switches, it delivers extreme low-latency and operates at 100- 200- and 400Gbps Ethernet speeds.

Deploying NVMe SSDs using NVMe/TCP or NVMe/RDMA can be done in several ways:

1. Direct attached storage (DAS) where NVMe SSDs are installed inside servers with application-based storage management.
2. NVMe SSDs inside servers with software-defined storage (SDS) management.
3. NVMe SSDs in NVMe-oF storage arrays.

NVMe-oF with DAS can boost application performance and leverage Ethernet-based networks' low cost. However, it comes with a variety of issues. It requires investment in more SSDs and servers for RAID, backup and other copy operations. Like DAS, SDS uses NVMe-oF and Ethernet networks but limits performance during high-IO and demanding workloads. The storage management overhead of SDS eats away at the CPU scheduling of any hypervisor. Network latency and metadata management contribute to significant penalties in write operations. Compression and deduplication not only slow operations, but they consume more CPU, all of which collide and compete with other workloads.

Another alternative is NVMe-oF storage arrays. NVMe-oF arrays offer NVMe/TCP, NVMe/RDMA and lower network overhead than DAS or SDS. In addition, these systems provide RAID, snapshots and replication and deliver block, file and object storage protocols. Some vendors even have explicitly designed arrays for use with the latest NVMe-oF drivers on Windows, Linux and VMware vSphere 7. 

Regardless of the deployment option, storage networking with NVMe-oF provides government agencies with an exciting roadmap for the future. By taking advantage of NVMe SSDs and the ubiquitous Ethernet protocol, data analytics bottlenecks can be a thing of the past.