Enterprise Flash Drives (EFDs) have made a big splash in the storage world over the past 2 years. Many customers are interested in how they might apply this new technology to speed up applications. And given a cost of roughly 20 times that of the same size 15K Fibre Channel (FC) drive, they want to be sure they can get a good return on their investment. Finding the right targets to make that happen can be fairly simple.
The Problem
A retail customer purchased a set of 8 EFDs with a new array. They had a few systems that were critical to their overall operations that they were hoping to improve. The business would get the most value from an improvement in the PeopleSoft HR batch runs, ensuring that payroll computations were completed on time. They had upgraded to new HP-UX servers, and found that they were now usually waiting for disk I/O.
The systems had been running on a CLARiiON CX3 array with dedicated groups of 15K FC drives. They were to be moved to a new Symmetrix DMX-4 which had the EFDs. The natural discussion was what data to place on this valuable EFD space.
The initial idea from the customer was that the Oracle logs would be best served by this faster space, since Oracle transactions cannot complete until the log writes are on disk. And the logs do tend to be very busy with lots of small writes. However, the nature of the logs is that they are all small and sequential writes. Disk array cache eats small sequential writes for breakfast. Not only does the cache get to reduce the number of physical disk I/Os by coalescing them, but it also gets to perform optimized RAID writes. So while RAID 1 protection may be desired for data availability, RAID 5 will work great for the overall performance need. And EFD would be wasted on this workload, where the only reads are sequential reads for archiving (or during recovery).
Since this is an OLTP system, most of the random I/O will be directed at the index files. It happens that this database was already separated by index and data. The total of the index space was about 1 TB, which fit well since they had a 7+1 RAID 5 group of 200 GB EFD, which provided more than the needed 1 TB. New devices were created, and one weekend the file systems were moved over.
The Results
As expected, the results were magical. The weekend batch runs were not just faster, they flew. Overall wall clock time was cut roughly in half. The maximum host device busy went from 100% to around 25%, clearly showing that the disks were no longer the bottleneck. The combination of being able to have more I/Os on these devices and having the response times drop was clear.
Then Monday morning came, and the phone lines on the IT Help Desk lit up. Everyone using the PeopleSoft system was wondering what was going on. They system had never been this fast. The comments were effectively, "whatever you did, do that again!" That is always a nice way for the IT team to start a week.
Getting both the batch performance improvement and the tremendous user feedback was like lightning striking twice. Moving the rest of the database onto the DMX-4 also went well. Performance was slightly better than it had been on the older CLARiiON, though the difference from this array move was nothing like the change to EFD.
Additional Information
For those not familiar with EFD, this is the high end of solid state storage. These devices use Single Layer Cell (SLC) technology, which provides faster access, lower error rates, and a much higher numbers of erases (re-writes) before the cells die than MLC. EMC provides these drives with a similar maintenance cost to other hardware items as a percentage of purchase price - so having them wear out in much less than 5-6 years could get really expensive for EMC. To help this, about 20% of the raw space is reserved for spare tracks (as opposed to a couple of cylinders on the average hard drive), so that the drive continues to operate normally even after some of the cells have failed. And EMC holds a lot less space in reserve than other vendors shipping these same drives, perhaps because of the additional experience we got testing them for 2 years before we released them.
EFD Magic
The first magic of the EFD drives is that they can do a lot more I/O – think at least 30 times more iops per drive than a 15K FC drive before the response time starts to head up because of queuing. The additional magic is that all of these I/Os are handled in less than 1ms. This compares to around 6-7ms for random reads on a 15K FC drive. And the response time and throughput for EFD are about the same whether the workload is random or sequential, and fairly independent of block size. This is definitely a big change.
Data Placement
For those looking for general Oracle LUN suggestions, the index files should be placed on the fastest random read drives, since that will deliver the most value. This is what makes index data such a great target for EFD. A large SGA can cut the amount of physical I/O dramatically, making the performance demands on the drives much lower – the fastest I/O is the one you never do. If a large portion of the data can be served from the SGA, it is likely that EFD drives will have limited impact and may not be worth the investment. Data files should also be placed on fast drives, though they are less important for most OLTP systems than the index files. Log files can be placed on almost any cached RAID type, since the cache will handle the writes. Logs should be placed on their own host devices (LUNs), so that their I/O does not wait for any other devices. The archive logs are fine to place on large ATA space, especially since they are only used in a sequential manner and ATA drives have good sequential performance. Systems will generally perform better if there are at least 10 active devices per HBA port driving I/O, so that queuing does not add latency. These recommendations are in place for this customer, and have served them well.
Array Cache
The Oracle SGA can do a fine job of caching recently used data. Customers need the array cache to work on holding onto data that the SGA was not able to retain. Or reading ahead to get data into cache that the database is likely to use in the near future. And, of course, the array can cache writes in a fault tolerant design that is not available in the server memory. If there are multiple workloads sharing a single array, and there usually are these days, then the ability to partition the cache by workload may be necessary to ensure that high bandwidth applications (like backups) don't swamp out the data from more response time sensitive applications. The Symmetrix cache is very good in these areas. When selecting an array to use for Oracle databases, customers should consider the cache management abilities of the array.
Database Logs
One final note on Oracle logs. They should always be placed on separate physical drives (separate RAID groups) from the data in the database. This is a consistent recommendation from Oracle, and applies to Microsoft SQL Server and most other databases as well.
Conclusion
Enterprise Flash Drives can make a huge difference in system and application performance. By placing the most heavily used random read data on these devices, response times will drop and the number of I/Os will increase dramatically. This can be a huge political win for an IT organization by driving increased satisfaction for internal and external customers. It can also drive competitive advantage for the business.
