Magnet AXIOM vs. Magnet IEF: Going Supersonic

By Jad Saliba, Founder & CTO of Magnet Forensics

It’s been about five months since I did some performance profiling between Magnet IEF and Magnet AXIOM and published a blog documenting the results. It was awesome to see the improvements that AXIOM has enabled, especially when it comes to examination/review time (keyword searches, filters, etc). We continue to hear from you about how you and your teams are dealing with more cases and more data every year, and how important it is to process and review that data quickly.

While we were happy with the performance numbers, we knew there was more that could be done and areas for improvement. The other piece of this was to target hardware similar to what most of our customers have and optimize for that – we generally hear that you are using dual CPUs, lots of RAM (64GB+), and fast SSDs or drive arrays.

We asked our R&D team to make this their focus and see if they could uncover more speed improvements. Or, in the ageless words of Maverick:

Pretty much everyone at Magnet Forensics knows about my affinity for Top Gun so I couldn’t resist including this. ? (Please tweet at me if you’re a Top Gun fan as well!)

I’m very happy (and super impressed with our team) to report that they delivered on that mission! Details of my new performance testing are below but the tl;dr version for those of you who don’t need the details is that we are consistently seeing 40% speed improvements on processing compared to IEF (and earlier versions of AXIOM! More on that in a future blog...) and AXIOM is on average 40 times faster (and sometimes 75+ times faster) than IEF when it comes to reviewing case data.

Performance Testing Part II

For this testing, instead of using consumer hardware, we used a machine that is more comparable to the forensic workstation you probably use every day. Here are the specs:

Testing Machine

• Server based desktop
• Dual Intel Xeon E5-2640 v4 CPUs @ 2.40GHz (20 cores, 40 logical processors)
• 128 GB DDR4 RAM
• 8TB RAID10 array for the image, 2TB SSD for the case
• Windows 10 Pro (x64)

The Dataset

For consistency, I used the same dataset from the previous blog, but here are the high-level details:

• 500GB hard drive
• 89.9GB allocated
• NTFS filesystem
• Windows 7 Home Premium OS installed on drive – in use approximately 4 years
• E01 image created – size of image is 82.2GB (medium compression setting used)
• Approximately 723,000 records/artifacts

The Tests!

First, let’s look at processing time. I used the same settings in IEF and AXIOM:

• All artifacts selected
• “Full Search” on all partitions
• No additional processing options selected

Processing Time

IEF (v6.18.1.12503)

• Search duration: 04:24:43
• Indexing duration: n/a

AXIOM (v2.5.1.11408)

• Search duration: 02:31:49
• Indexing duration: 00:00:31

Comments

Much better times on this machine than the laptop but the biggest difference is seen in the AXIOM times. AXIOM completed the same search almost 2 hours faster, including indexing time. That’s a 42% decrease in processing time! It’s important to note that we’ve seen that ~40% number in most of the test images we work with internally, so it’s not a one-off or a “cherry picked” image.

Review / Analysis

Okay, we’ve got our new cases. We’ll run through the same common operations in both review tools that I conducted in the first blog post. First, we need to open the cases:

Case Load Time

• IEF: 00:00:04
• AXIOM: 00:00:09

Seconds apart, IEF is still faster here but it’s using a flat database structure so opening cases is generally super quick. AXIOM uses a more complex/relational database schema which allows us to save a ton of time later on when we get into keyword searching, filters, etc (i.e. the operations that are repeated many times and are a lot more time-consuming compared to loading the case once). We’ll continue to look for improvements here though!

Now, let’s load those LNK file artifacts and sort on one of the columns.

Load “LNK Files” and sort on column “Last Accessed Date/Time” (65,648 records)

• AXIOM: 00:00:07
• IEF: 00:01:02

IEF got a little faster on this machine but AXIOM still gets you into the data much quicker than IEF and then sorts the data faster too – a decrease in time of 88.7% to be exact! (Again, the other thing to note is that if this was a bigger dataset (over 100,000 records) or you had the “max records” setting in IEF to a number less than the total number of records, IEF is only sorting the visible/loaded records – not the entire dataset for that artifact. This was a pain point in IEF and something we made sure to resolve in AXIOM.)

In the last blog post, I talked about scrolling speed – IEF was faster than AXIOM in this respect but we’ve made some great improvements in 2.5 with AXIOM. Smoother scrolling that allows you to scan records quickly without waiting for records to load/blank rows. Check it out and let us know what you think!

Next, let’s run through those three single-keyword searches we did last time and then a keyword list.

Single Keyword “guest” (~50k Results)

• AXIOM: 00:00:09
• IEF: 00:06:12

Such a huge difference — even on beefed-up hardware. AXIOM went from 50 secs to 9 on this hardware and IEF from 10:17 (m:s) to 6:12. AXIOM is still getting the search results in 97% less time than IEF here (over 41 times faster)!

Single Keyword “chris” (~347k Results)

• AXIOM: 00:00:13
• IEF: 00:16:14

This keyword has a lot of results. As you can see, the large number of hits only affected AXIOM slightly versus the last keyword, but IEF more than doubled in time due to the additional hits. The numbers almost get a bit crazy here if you look at the percentages: almost a 98% decrease in time, or 75 times faster.

Single Keyword “craigslist” (1,197 Results)

• AXIOM: 00:00:08
• IEF: 00:07:20

Less results this time, AXIOM still in the single digits, but IEF did a bit worse compared to the “guest” keyword which had more results for some reason. Could be related to the length of the keyword possibly.

Next let’s try the keyword list we used last time. We’ll import a text file containing these keywords:

• angel
• car
• rental
• yahoo
• facebook
• phoenix
• insurance

Keyword List – 7 Keywords (~24k Results)

• AXIOM: 00:01:27 (near instantaneous to display after selecting the keyword list in the filter bar menu – see below screenshot)
• IEF: 00:05:49

AXIOM’s time increases a bit here compared to the single keyword searches (but about 28% faster than AXIOM on a laptop), but it’s doing something additional – data is saved in the case database so that future searches of those keywords will be near instantaneous. See the below screenshot for how you would access those keywords – you can also see the hit count next to each keyword:

IEF stayed in the lower range due to the hit counts not being high, but still takes much longer than AXIOM and future searches of the same keywords will require the same amount of time each time they are run.

Okay, let’s do some filtering now. Let’s say you want to see all activity in the year 2010 for this case.

Filter All Dates/Times in the Year 2010 (~119k Results)

• AXIOM: Near instantaneous
• IEF: 00:18:42 (using “Global Date/Time Filter”)

I said it last time and I’ll say it again: wow! This especially illustrates how AXIOM leverages the new back-end changes to really accelerate filtering, something you probably do a lot of in your cases. Think of how that time adds up, or which filters you are not using due to time constraints?

Filtering in AXIOM also gives you additional features like selecting days of the week, filtering time independent of date and vice versa, and a cool anchoring feature that allows you to see all activity X number of minutes before/after a specific point in time.

Let’s try filtering on a single artifact, and on a single column.

Filter All Windows Event Logs for Event ID 4624 (“an account was successfully logged on”) – (~14k Results)

• AXIOM: 00:00:02
• IEF: 00:01:08

Again, AXIOM finishes much faster than IEF – not quite as dramatic as the last filter but still pretty good (34 times faster, or an increase of 3400%)!

Finally, let’s try loading up the Timeline view to see all artifacts with one or more timestamps in a visual representation.

Load Timeline View (~509k records)

• AXIOM: 00:00:40
• IEF: 00:06:53

That’s a lot of records, but AXIOM gets through it in short order. IEF does its best with a large number of records but falls far behind AXIOM. Again, imagine being able to see a visual timeline view of a filtered set of records, then tweaking the filters, maybe adding a keyword search, and looking at the results within a few seconds. AXIOM lets you do this! If you’re looking at AXIOM in the same way you do IEF, I hope this helps you realize that the two are quite different. We’re not even delving into some of the new and unique features that AXIOM offers here. ?

Total Times / Total Impact

What does this all mean for your day-to-day examinations? Or impact over a year?

The numbers are awesome:

• IEF: 04:24:43 (processing time)
• AXIOM: 02:31:49 (processing time)

Again, that’s a 42% reduction in processing time, or 74% faster!

On the review side, if we assume an average review of a case consists of similar searches/filters/view loads as above, multiplied by seven (I’m making these numbers up, and grossly oversimplifying, let me know what you think is more accurate!) per case, and using the above search/filter/load times as an example, here are the cumulative review times (per case):

• IEF: 07:15:24
• AXIOM: 00:19:29

Pretty amazing when you look at it that way. Not saying you’d spend 20 minutes in a case in AXIOM vs 7 hours in IEF necessarily…but the numbers are what they are and they represent the potential time savings.

If we extrapolate to a year’s worth of cases, assuming you can get through ~2.5 cases a week at the above pace (accounting for reporting time, admin work, court time, search warrants, etc), that’s about 130 cases a year!

Accumulated review time in a year:

• IEF: 39 days, 07:22:00 (that’s 117 work days @ 8 hr work days, 133 days if you take out an hour for lunch)
• AXIOM: 1 day, 18:12:50 (3 work days, or 3.4 with an hour lunch per day)

Wow. Imagine how many more cases you could get through if you could take ~39 days down to 1 day of time required to review your cases. And that’s not even taking into account the ~40% reduction in processing times seen in AXIOM 2.5. Again, these are very rough estimates but they are representative and directional. Do your own tests, run your own calculations – I’m confident you’ll end up somewhere in this range.

Conclusion

I wanted to do this “part 2” blog post to show how far we’ve come just in the past 5 months and to also show the commitment we have to performance and responding to your feedback. In the last blog post, I showed the how far we’d come since the launch of AXIOM. I hope this illustrates that we’re not slowing down (no pun intended ?). And please, try a similar test out and see for yourself! I haven’t picked an image that favors AXIOM but there’s no better way to validate that than to get a trial of AXIOM and do a comparison yourself. You can get a trial here!

If you’ve been using IEF for a long time, I understand that you’re comfortable with it and like how it works. I don’t always like change when it comes to software either. ? But that’s why we kept a lot of familiar things in AXIOM so the switch wouldn’t be difficult at all. And as you can see above, even if you just use AXIOM the way you do IEF, you’ll save tons of time just based on the performance improvements. But I’m confident that if you start to try some of our new AXIOM-only features (Connections, Dynamic App Finder 2.0, file system and registry access, Volatility integration, Magnet.AI, and much more) you’ll be very happy you made the switch.

As always, we welcome (and appreciate!) your feedback, good or bad – our goal is to support you in the work you do, and provide the best support possible. For us, it’s not about accolades, it’s about impact. So help us help you ? and thanks for your support throughout the years and being on this journey with us. We’re looking forward to continuing on it with you for many more!

– Jad & the Magnet Forensics team