[Ian Mapleson]

Mark writes:
> All of the above makes sense to me, just working out some last minute
> logistics issues if you don’t mind.

No prob! If you’d like more detailed info, feel free to email me (send to
mapesdhs@yahoo.com), I can explain a few other issues specific to a build
using that case, data that doesn’t belong here, eg. small fans for
cooling the chipset, where/how to put them, pictures, etc.

> The issue I’m now seeing, as I’m sure you’re aware, is it is quite
> difficult to find new GTX 580’s except for the hydro 4 GB models which is

I was really only referring to buying used. 😀

I’ve won two on eBay this past week. The first should arrive tomorrow,
the 2nd either tomorrow or Monday. See:

https://cgi.ebay.co.uk/ws/eBayISAPI.dll?ViewItem&item=171044128930
https://cgi.ebay.co.uk/ws/eBayISAPI.dll?ViewItem&item=200925198261

However, they are just the 1.5GB models. 3GB models are harder to find &
obviously usually go for higher amounts.

Are you in the UK? Or in the US/Canada? Or elsewhere?

The 580 is still available new in the UK, and there are some good deals
available from reputable sellers, eg. right now Scan has the EVGA 3GB 580
for 223 UKP as a refurb:

https://www.scan.co.uk/products/3gb-evga-geforce-gtx-580-classified-refurb

Hmm, might call the guy who bought the system I built, let him know…

> looking at getting one of the GTX Titans since in the benchmark test that

I suppose it does have the large RAM advantage.

> … Any reason I should avoid this besides the fact it is
> more expensive than I could probably secure two GTX 580’s?

Well, let’s see… (checking!) I’ll use UK prices here, but I expect the
same ratios would likely apply elsewhere.

A basic Titan is 800 UKP (837MHz), good ones about 888 UKP (876MHz). If
one is going to spend that much, may aswell get a good version, so let’s
say 900 UKP typical.

With some hunting around, you could maybe get 2 or 3 new GTX 580s for
that much, which would hammer a single Titan. For example, here:

https://cgi.ebay.co.uk/ws/eBayISAPI.dll?ViewItem&item=221230167382

However, something one must be careful of with these cards is how many
real slots they use up. Although the backplate of most 580s is 2-slot
(ASUS is an exception, one of their 580s does have a 3-slot plate), in
reality many of the coolers are so big that overall the card uses 3
slots. The above 3GB Gainward Phantom is typical, it uses 3 slots because
of the cooler. Thus, you’d only be able to fit two on the board. Of
course if one had the budget then one could use watercooling, but that’s
a whole other scenario. Anyway, at least for my research, I’ve been
deliberately choosing 1.5GB cards which definitely only use 2 slots.

Or instead of a Titan, you could get two GTX 780s, which would be about
a 3rd more than the cost of a single Titan, but obviously more than a
3rd better CUDA performance. OTOH the 780s only have half the RAM.

Having said that, if you did get a Titan, it would at least give you the
maximum upgrade potential for the future (4x Titan, blimey! Need a better
PSU though), and the luxury of large RAM. It’s just that in terms of
upfront initial performance, without a doubt 2 or 3 580s would be faster
and cost about the same or less.

I guess it’s a tossup between having better performance right now but
less RAM with multiple 580s, or a better upgrade path in the future & the
luxury of large RAM now.

Note I was going to mention the latest GTX 770, but I see those only have
2GB RAM. Nah, too much of a sacrifice really, though they are cheaper,
about 330 UKP each here, so for not really much more than the cost of a
Titan you could get three 770s, but the RAM would only be 2GB; hmm, I’d
rather expend the effort to find 3GB 580s.

> Also if I do secure two GTX 580’s do they need to run in SLI mode, or is
> it enough to just have them in the box?

No, do not use the SLI connector. Not needed for CUDA tasks.

> Case: Cooler Master HaF 932
> Board: ASUS P9X79 WS LGA 2011

Definitely email me about this, I can pass on some build tips. For
example, I wasn’t impressed with the stock rear 140mm fan in the 932, so
I replaced it with a much better Noctua NF-A15.

> PSU: CORSAIR Professional AX1200 Modular

Yup, that should do.

> Processor: Intel Core i7-3930K Six Core 3.2 Ghz

Remember to get the C2 stepping. Always ask the seller to confirm (ie. if
their specs are not absolutely clear). Ask for the SPEC code from the CPU
box, which you can check via ark.intel.com. The code should be ‘SR0KY’.

> Cooler: Phanteks PH-TC14PE – hoping that will fit in the case, looks like
> it should

Yes, it does; it’s what I have, though note the side panel 240mm fan will
have to be replaced with 2x 120mm instead on the lower half of the panel.
Naturally, I used AS5 for the HS attachment instead of the paste that came
with the fan. I use Articlean 1&2 for CPU/HS surface preparation, and
lint-free cloths for cleaning/wiping/etc.

And remember to get a 3rd fan for the Phanteks. The Black edition is
widely available now and it’s often the cheapest aswell (biggest demand).
Here’s a pic of my own board with the HS/fans in place before I put it
into the 932 case (bottom of the picture):

https://www.sgidepot.co.uk/misc/pcbenchmarking2.jpg

Email me, I can send you some build pics.

Likewise, for my own build, I put the PSU at the top, so the upper 240mm
fan had to be removed; I’ve installed an extra 120mm towards the front of
the case to act as an extra intake, giving fresh air for the CPU cooler.

Doing it this way means one can fit four 2-slot GPUs on the board. To
compensate for the missing top fan, I fitted two 120mm exhaust fans at
the bottom of the case (the side panel fans are intake). I use only Gelid
Wing Blue UV 12 fans for 120mm jobs, PWM editions where that’s sensible.
They’re normally available in a 4-pack for a good price (I invested,
bought 8 packs last autumn). If you live in a hot climate, faster fans
are available, but they’ll be louder.

> Internal HDDs: Samsung 840 Pro – 128 GB for Cache & 256 GB for Applications

Excellent!!

For general data storage, I recommend Enterprise SATA if you can afford
them (I struck lucky, bagged several lightly used drives locally for a
silly price). In the build I did for the AE guy, I fitted 2x 2TB
Enterprise SATA in RAID1 for his long term archive, and a 3rd just on its
own as the main work area disk (RAID speed just isn’t needed when working
with AE, ie. the key processing bottlenecks are CPU, GPU and cache I/O).

If you can’t obtain or afford Enterprise SATA, then stick with the upper
end of normal SATA such as WD Black (don’t bother with low-end SATA). For
Seagate, go for the models which have NS at the end, they’re more reliable
than the generic consumer AS models.

Oh, if you want a really nice & tidy way to hold the SSDs, I definitely
recommend the Startech 4-bay SAS/SATA enclosure. Holds 4x 2.5″ devices,
uses one 5.25″ bay. Looks like this:

https://cgi.ebay.co.uk/ws/eBayISAPI.dll?ViewItem&item=370668792942

Includes LED power/activity indicators on the front for each sled. I gave
one to my brother for xmas, here’s a pic of it installed in his Antec 300:

https://www.sgidepot.co.uk/misc/keithpc2.jpg

If you’re in the USA, Thermaltake makes a case that holds 6x 2.5″ devices
in a single 5.25″ bay (not available in the UK); I bought two, asked a
friend to post them over.

> Memory: 2x GSKill Trident X series 32 GB DDR3 2400

Note you’ll have to remove the top fins from each DIMM as they won’t fit
under the HS. Again, I have pics.

I’ve just fitted one of these kits to my P67/2700K setup; somewhat to my
surprise I didn’t have to change the BIOS settings at all, just up &
running straight away, 2133 as it was before with 4 x 4GB.

> Monitor: Dell UltraSharp U2713HM

Now that’s funny; after a morning of 2560×1440 monitor hunting, trying
to find something that supports sync-on-green so it’ll work with my
SGI stuff aswell, I bought that very model of Dell about an hour ago. 😀

Cheers! 🙂

Ian.

SGI Guru

[Ian Mapleson]

Mark writes:
> Thanks for all of the deep insights Ian!

Most welcome!

> This looks like quite the machine. …

Short of going expensive multi-socket, it’s about as good as it gets
apart from the GPUs used.

Note that I’ve just secured two GTX 580 cards, so I’ll be able to
test a wide range of configs soon.

> … My thoughts right now are that at
> current I’m going to stick with just one GFX card …

If so, then get a GTX 580 3GB with the highest core clock you can afford.

> … since I’ve been reading
> that After Effects only pulls from one GFX card for rendering. Can you
> confirm or deny this?

No, that’s not true at all. Ray Trace 3D mode uses multiple GPUs, as can
RAM Preview in many cases.

However, depending on the effects used, _sometimes_ the performance
scaling from multiple GPUs is not as one might expect. It depends on the
nature of the scene. I have one particular test scene (fire effects, 3D
letters, motion blur, lens flare, lots of particles) which imply that
perhaps the use of particles can affect how well AE exploits multiple
GPUs, but I’ve yet to confirm this – I need to set it up again and do
proper tests.

Certainly though, in numerous cases, multiple GPUs do scale the
performance nicely, which (if you have the option with respect to mbd
slots/spacing) means more flexible choices with what you can buy in terms
of how to spend your budget. For example, two cheaper cards might cost
less but outperform a single better card, though cheaper cards do tend to
have less RAM. This is why I initially fitted the system I built with two
800MHz 1GB GTX 460s (they only cost 87 UKP each). Combined with a Quadro
4K as the primary display card, they’re faster than a GTX 580. However,
as mentioned above, if one is doing something that can’t exploit multiple
GPUs quite so well, then performance may be limited to the power of a
single card, but this isn’t usually the case.

Ideally, AE itself would be able to monitor what’s happening with multiple
GPUs and adjust how it uses them accordingly, but I don’t think that
happens atm.

For example, see:

https://forums.creativecow.net/thread/2/1019120

I call that test, “CS6-11.1”, in my notes.

The original poster states they get 5 mins and 42 secs with their 3930K +
GTX 580 3GB (an expensive GPU).

At the time, my test systems were (both of these are 4-core with 8 threads):

ASUS Maximus IV Extreme
5GHz i7 2700K
16GB DDR3/2133
Quadro 4000 2GB
2x 800MHz GTX 460 1GB

Asrock X58 Extreme6
3.5GHz XEON X5570 (similar to an i7 950)
24GB DDR3/1600
3x 850MHz GTX 460 1GB

Render results were as follows:

2700K system:

1 Card: 17m 53s (rendering via the Quadro 4000 only)
2 Cards: 8m 35s (Quadro 4000 + 800MHz GTX 460)
3 Cards: 5m 59s (Quadro 4000 + 2x 800MHz GTX 460)

X5570 system:

1 Card: 11m 58s
2 Cards: 7m 15s
3 Cards: 5m 17s

So the first system comes close to a GTX 580, while the 2nd system beats
a GTX 580 (not surprising really).

In the case of the 3930K system I built, the two 800MHz 460s only cost 87
each – a lot less than a 3GB 580. Also, some models of GTX 460 can oc to
crazy levels (I have one that happily zips along at 1025MHz core), and
indeed the two 2GB cards I’m sending to the end user will be forced to
run at 800MHz instead of their default 700 in order to match the
performance of the old 1GB cards (I’ve modified them with extra heatsinks
on both sides of the PCB, etc. to cope with the extra heat under load).
Anyway, that’s a side issue, I just wanted there to be no performance
drop with the replacement cards. With the two 800MHz 1GB cards returned,
I’ll then have 4x 800MHz 460s for testing with the replica 3930K system
I’ve built, but I’ll run the same tests on a very different board which
can also use four GPUs (ASUS P7P55 WS Supercomputer with an i7 870, a P55
board with NF200 to support x8/x8/x8/x8).

In all cases, I’ve maxed out the RAM on the boards to allow for the best
possible render tests that involve a lot of RAM, to show what’s possible
with a more limited budget (P55, X58, P67, Z68, etc.)

> How much does this rig rely on overclocking …

If the task is CPU-bound, then obviously it relies a lot on the oc’d CPU,
eg. rendering via Classic 3D mode, but this is the big win with respect
to price/performance, because a 6-core SB-E at this speed easily
outperforms a much more expensive 8-core XEON. It certainly leaves my
dual-X5570 Dell T7500 completely in the dust.

> … – my goal would be to keep
> everything in as stock a configuration as possible …

IMO that’s just wasting free performance. At the very least one can
exploit what oc is possible without raising the core voltage.

That’s not to say some effort isn’t required to sort out an oc’d
configuration, but the effort is well worthwhile. I built the AE setup
in two days.

Having said that, I wouldn’t bother oc’ing a GPU like the GTX 580. It
already generates quite a bit of heat and uses a fair amount of power,
so the gains via oc’ing are less useful considering the impact on the
rest of the system. It would be more sensible to just add a 2nd GPU.

Likewise, I don’t oc the RAM. Instead, I just bought a kit rated high
enough (DDR3/2400) so that I could run the RAM at the desired speed (2133)
without any issues.

Btw, the end user of the AE system I built provided me with his own test
scene consisting of a number of moving ray-traced 3D words with multiple
reflections, etc., a less complex scene than the CS6-11.1 test above. I
can’t provide a screenshot yet, so I hope that description will suffice.
Anyway, the render scaling results were as follows:

2700K system:

1 Card: 4m 8s (rendering via the Quadro 4000 only)
2 Cards: 2m 2s (Quadro 4000 + 800MHz GTX 460)
3 Cards: 1m 28s (Quadro 4000 + 2x 800MHz GTX 460)

X5570 system:

1 Card: 2m 44s (1x 850MHz GTX 460)
2 Cards: 1m 36s (2x 850MHz GTX 460)
3 Cards: 1m 17s (3x 850MHz GTX 460)

Note that I obtained the same X5570/3-card result when testing with 3 x
800MHz GTX 460s on a simple P55 board with a 4.7GHz dual-core i3 550 (the
CS6-11.1 result was 6m 35s).

I actually signed up for an account on this site because I was planning
on writing a lengthy summary post for the above mentioned benchmark
thread, but your query prompted me to post early.

Hope this helps! 🙂

Ian.

SGI Guru

[Ian Mapleson]

Mark writes:
> technical background for designing and building PC’s, I am not as
> familiar with what improves performance in those areas.

I’ve been doing a lot of work on AE build issues in the past few months.
As with all things, it’s a balance of compute, GPU, I/O, RAM, etc.

I built an AE system for someone back in January, exploiting used parts
where possible or sensible (the fans, mbd, CPU, RAM, SSDs, DVDRW and
Quadro were new, everything else used, though some new items were for
eBay and thus saved some pennies). There wasn’t the budget for a single
good CUDA card, so I fitted two 460s instead which is just as quick; the
plan is to replace them later (when the user’s budget permits) with two
or more 580 3GB cards. Here’s the config:

Case: Cooler Master HAF 932 (max EATX)
Rear Case Fan: Noctua NF-A15 140mm PWM
Side Case Fans: 2x Gelid Wing Blue UV 12 1600rpm
Modular PSU: Thermaltake Toughpower 1000W Standard
MOTHERBOARD: ASUS P9X79 WS
CPU: i7 3930K 6-core 3.2GHz 12MB S2011
CPU Heatsink: Phanteks PH-TC14PE Red with three fans
RAM: GSkill 64GB (8x8GB) DDR3/2400 CL10
System Drive: Samsung 830 256GB SSD
AE Cache: Samsung 830 256GB SSD
Main Storage: 1x Hitachi 2TB Enterprise SATA OEM
2x Hitachi 2TB Enterprise SATA OEM
DVDRW: Sony Optiarc 24X
Graphics Primary: Quadro 4000 2GB (256 cores)
Graphics CUDA: 2x Palit GTX 460 2GB (672 cores total)

I’m about to send the user two GTX 460 2GB cards to replace the 1GB
cards, a temporary boost in capability (ie. RAM capacity will then match
the Quadro 4K).

The main display card is a Quadro in order to have better primary GPU
image quality, reliability, etc.

Note that I was so impressed with the ASUS mbd, I built a replica system
for myself, which I’m currently finalising atm (all up & running, just
sorting out the same oc), though I built it in a different way, with the
PSU at the top of the case so that the system can be fitted with four
GPUs instead of three. Hindsight is always wonderful. Hopefully I change
the first system sometime to be the same, if the user can visit again.

Here’s some pics (that’s me in a couple of them; the client took the pics
as he collected in person to avoid any courier issues):

https://www.sgidepot.co.uk/misc/aepcpics.zip

and a CPU-Z:

https://valid.canardpc.com/show_oc.php?id=2656382

In your case Mark, I would say the following…

> Antec Three Hundred Illusion Black Steel ATX Mid Tower Computer Case

Too small. I have *lots* of Antec 300 cases, they’re great for quad-core
builds, but not for an X79 setup. There just isn’t the air flow to cope
with the heat kicked out by a 3930K, multiple GPUs, disks, etc.

> ASUS P9X79 PRO LGA 2011 Intel X79 SATA 6Gb/s USB 3.0 ATX Intel
> Motherboard with USB BIOS

I’d recommend the WS instead.

> EVGA 04G-P4-3687-KR GeForce GTX 680 FTW+ w/Backplate 4GB 256-bit GDDR5
> PCI Express 3.0 x16 HDCP Ready SLI Support Video Card

The CUDA performance of the 680 is not so good.

You’re better off with one or more GTX 580 3GB cards.

> CORSAIR Professional Series Gold AX850 (CMPSU-850AX) 850W ATX12V v2.31 /
> EPS12V v2.92 80 PLUS GOLD Certified Full Modular …

Not enough. You need at least 1kW, preferably 1200W, and if you intend to
fit multiple exta GPUs later then 1500W (especially if they’re 580s).

> Intel Core i7-3930K Sandy Bridge-E 3.2GHz (3.8GHz Turbo) LGA 2011 130W
> Six-Core Desktop Processor BX80619i73930K

Yes, good choice, but make sure it’s a C2 stepping.

> Crucial M4 CT512M4SSD1 2.5″ 512GB SATA III MLC 7mm Internal Solid State
> Drive (SSD)

Eek, not what I would choose, not for a system like this. Samsung 840 Pro,
OCZ Vector or Vertex4 would be better.

I’ve fitted mine with a standard Samsung 840 250GB since the 830s are not
available anymore (and I couldn’t afford a Pro), have to say I’m very
impressed with it’s long term behaviour. Here’s an HDTach I did this week:

https://www.sgidepot.co.uk/misc/samsung_840_250GB_HDTach_22-May-2013.gif

If I had the budget though then I’d get an 840 Pro as its write
performance is quite a kick up the scale. Note the only reason why I’d
choose the Pro is atm the Vector and Vertex4 are unnecessarily expensive.
I struck lucky though, bagged two new Vector 256GB units off eBay for
good prices (150 UKP each).

> G.SKILL Ripjaws Z Series 32GB (8 x 4GB) 240-Pin DDR3 SDRAM DDR3 1600 (PC3
> 12800) Desktop Memory or G.SKILL Ripjaws Z Series 64GB (8 x 8GB) 240-Pin
> DDR3 SDRAM DDR3 1600 (PC3 12800) Desktop Memory

Memory bandwidth helps for AE. Get two of the TridentX 2400 kits
instead. The higher rating means they’ll be stable at 2133. Ironically,
the 2400 kit was cheaper than 2133 kits.

> COOLER MASTER Hyper 212 EVO RR-212E-20PK-R2 Continuous Direct Contact
> 120mm Sleeve CPU Cooler

No, not renmotely good enough. Get at least the Phanteks PH-TC14PE.

Also, you should absolutely have an SSD dedicated to being an AE cache,
it definitely helps. If you want to save some pennies, grab a used 120GB
Vertex3 off eBay, but obviously newer models will be somewhat better
(256GB Vector, Vertex4, 840 Pro or the other top-tier models would be best).

Note that I will be benchmarking my system with 4 x GTX 460 once the oc
is sorted out, also several other 3/4-way boards (P55, P67, X58), and
I’ve jsut obtained my first 580 (hoping to get at least 2 more in time).

For those commenting about CUDA performance of the 680 vs. 580, etc., in
my research since last year I have concluded that what matters most of
all is aggregate bandwidth per CUDA core. Thus, the 680, despite it’s
large number of cores, is unable to feed them as effectively as the 580.
There are certain cases where the 680 isn’t so bad, but – like the 780 –
the 580 has a major advantage with its high bw-per-core ratio. This is
also why multiple lesser GPUs like 460s work well, though of course the
power efficiency of the latter is not as good.

More data later. Let me know if you’d like some pics of my own 3930K
build. Finished it on Wednesday.

Johnny is right btw, definitely max the RAM. Also Mark, XEON boards
generally can’t be oc’d so much, if at all, and my 3930K easily stomps
on my dual-XEON Dell T7500 (two X5570s).

As for IvyBridge, unless one des the cap mod to replace the internal
chip paste, it runs far too hot, severely limiting oc potential unless
one uses water cooling.

Walter, RAID doesn’t seem to be necessary for the main ‘work’ disk when
dealing with AE, but it’s definitely a good idea to have something like
RAID1 for long term archiving.

Mark, don’t split an SSD in two; just have one for the system disk
(really, 256GB ought to be sufficient), and another for an AE cache. If
you can’t afford a 2nd 256GB, just get a 120GB or 128GB model to begin
with. Definitely do NOT use the server to hold the cache (that would
defeat the purpose of the cache completely, with respect to speed).

Re monitor, there are numerous well priced IPS 2560×1440 displays these
days, but if you’re on a budget, then the Dell UltraSharp U2412M 1920×1200
IPS is quite good for its prices.

I bought the HP LP2475W 1920×1200 HIPS, but that was a while back and
prices have dropped since then.

Cheers! 🙂

Ian.

SGI Guru