Deep learning is a field with intense computational requirements and the choice of your GPU will fundamentally determine your deep learning experience. With no GPU this might look like months of waiting for an experiment to finish, or running an experiment for a day or more only to see that the chosen parameters were off and the model diverged. With a good, solid GPU, one can quickly iterate over designs and parameters of deep networks, and run experiments in days instead of months, hours instead of days, minutes instead of hours. So making the right choice when it comes to buying a GPU is critical. So how do you select the GPU which is right for you? This blog post will delve into that question and will lend you advice which will help you to make a choice that is right for you.
Having a fast GPU is a very important aspect when one begins to learn deep learning as this allows for rapid gain in practical experience which is key to building the expertise with which you will be able to apply deep learning to new problems. Without this rapid feedback, it just takes too much time to learn from one’s mistakes and it can be discouraging and frustrating to go on with deep learning. With GPUs, I quickly learned how to apply deep learning on a range of Kaggle competitions and I managed to earn second place in the Partly Sunny with a Chance of Hashtags Kaggle competition using a deep learning approach, where it was the task to predict weather ratings for a given tweet. In the competition, I used a rather large two layered deep neural network with rectified linear units and dropout for regularization and this deep net fitted barely into my 6GB GPU memory. The GTX Titan GPUs that powered me in the competition were a main factor of me reaching 2nd place in the competition.
Should I get multiple GPUs?
Excited by what deep learning can do with GPUs I plunged myself into multi-GPU territory by assembling a small GPU cluster with InfiniBand 40Gbit/s interconnect. I was thrilled to see if even better results can be obtained with multiple GPUs.
I quickly found that it is not only very difficult to parallelize neural networks on multiple GPUs efficiently, but also that the speedup was only mediocre for dense neural networks. Small neural networks could be parallelized rather efficiently using data parallelism, but larger neural networks like I used in the Partly Sunny with a Chance of Hashtags Kaggle competition received almost no speedup.
I analyzed parallelization in deep learning in depth, developed a technique to increase the speedups in GPU clusters from 23x to 50x for a system of 96 GPUs and published my research at ICLR 2016. In my analysis, I also found that convolution and recurrent networks are rather easy to parallelize, especially if you use only one computer or 4 GPUs. So while modern tools are not highly optimized for parallelism you can still attain good speedups.
The user experience of using parallelization techniques in the most popular frameworks is also pretty good now compared to three years ago. Their algorithms are rather naive and will not scale to GPU clusters, but they deliver good performance for up to 4 GPUs. For convolution, you can expect a speedup of 1.9x/2.8x/3.5x for 2/3/4 GPUs; for recurrent networks, the sequence length is the most important parameters and for common NLP problems one can expect similar or slightly worse speedups then convolutional networks. Fully connect networks usually have poor performance for data parallelism and more advanced algorithms are necessary to accelerate these parts of the network.
So today using multiple GPUs can make training much more convenient due to the increased speed and if you have the money for it multiple GPUs make a lot of sense.
Using Multiple GPUs Without Parallelism
Another advantage of using multiple GPUs, even if you do not parallelize algorithms, is that you can run multiple algorithms or experiments separately on each GPU. You gain no speedups, but you get more information about your performance by using different algorithms or parameters at once. This is highly useful if your main goal is to gain deep learning experience as quickly as possible and also it is very useful for researchers, who want try multiple versions of a new algorithm at the same time.
This is psychologically important if you want to learn deep learning. The shorter the intervals for performing a task and receiving feedback for that task, the better the brain able to integrate relevant memory pieces for that task into a coherent picture. If you train two convolutional nets on separate GPUs on small datasets you will more quickly get a feel for what is important to perform well; you will more readily be able to detect patterns in the cross-validation error and interpret them correctly. You will be able to detect patterns which give you hints on what parameter or layer needs to be added, removed, or adjusted.
I personally think using multiple GPUs in this way is more useful as one can quickly search for a good configuration. Once one has found a good range of parameters or architectures one can then use parallelism across multiple GPUs to train the final network.
So overall, one can say that one GPU should be sufficient for almost any task but that multiple GPUs are becoming more and more important to accelerate your deep learning models. Multiple cheap GPUs are also excellent if you want to learn deep learning quickly. I personally have rather many small GPUs than one big one, even for my research experiments.
NVIDIA vs AMD vs Intel vs Google vs Amazon
NVIDIA: The Leader
NVIDIA’s standard libraries made it very easy to establish the first deep learning libraries in CUDA, while there were no such powerful standard libraries for AMD’s OpenCL. This early advantage combined with strong community support from NVIDIA increased the size of the CUDA community rapidly. This means if you use NVIDIA GPUs you will easily find support if something goes wrong, you will find support and advice if you do program CUDA yourself, and you will find that most deep learning libraries have the best support for NVIDIA GPUs. This is a very strong point for NVIDIA GPUs.
On the other hand, NVIDIA has now a policy that the use of CUDA in data centers is only allowed for Tesla GPUs and not GTX or RTX cards. It is not clear what is meant by “data centers” but this means that organizations and universities are often forced to buy the expensive and cost-inefficient Tesla GPUs due to fear of legal issues. However, Tesla cards have no real advantage over GTX and RTX cards and cost up to 10 times as much.
That NVIDIA can just do this without any major hurdles shows the power of their monopoly — they can do as they please and we have to accept the terms. If you pick the major advantages that NVIDIA GPUs have in terms of community and support, you will also need to accept that you can be pushed around by them at will.
AMD: Powerful But Lacking Support
HIP via ROCm unifies NVIDIA and AMD GPUs under a common programming language which is compiled into the respective GPU language before it is compiled to GPU assembly. If we would have all our GPU code in HIP this would be a major milestone, but this is rather difficult because it is difficult to port the TensorFlow and PyTorch code bases. TensorFlow has some support for AMD GPUs and all major networks can be run on AMD GPUs, but if you want to develop new networks some details might be missing which could prevent you from implementing what you need. The ROCm community is also not too large and thus it is not straightforward to fix issues quickly. There also does not seem to be much money allocated for deep learning development and support from AMD’s side which slows the momentum.
Overall I think I still cannot give a clear recommendation for AMD GPUs for ordinary users that just want their GPUs to work smoothly. More experienced users should have fewer problems and by supporting AMD GPUs and ROCm/HIP developers they contribute to the combat against the monopoly position of NVIDIA as this will greatly benefit everyone in the long-term. If you are a GPU developer and want to make important contributions to GPU computing, then an AMD GPU might be the best way to make a good impact over the long-term. For everyone else, NVIDIA GPUs might be the safer choice.
Intel: Trying Hard
My personal experience with Intel’s Xeon Phis has been very disappointing and I do not seem them as a real competitor to NVIDIA or AMD cards and thus I will keep it short: If you decide to go with a Xeon Phi take note that you might encounter poor support, computing issues that make code sections slower than CPUs, very difficult to write optimized code, no full support of C++11 features, some important GPU design patterns are not supported by the compiler, poor compatibility with other libraries that rely on BLAS routines (NumPy and SciPy) and probably many other frustrations that I have probably not run into.
I was really looking forward to the Intel Nervana neural network processor (NNP) because its specs were extremely powerful in the hands of a GPU developer and it would have allowed for novel algorithms which might redefine how neural networks are used, but it has been delayed endlessly and there are rumors that large portions of the developed jumped the boat. The NNP is planned for Q3/Q4 2019. If you want to wait that long, keep in mind that a good hardware is not everything as we can see from AMD and Intel’s own Xeon Phi. It might well be into 2020 until the NNP is usable in a mature way.
Google: Cheaper On-Demand Processing?
The Google TPU developed into a very mature cloud-based product that is extremely cost-efficient. The easiest way to make sense of the TPU is by seeing it as multiple GPUs packaged together. If we look at performance measures of the Tensor-Core-enabled V100 versus TPUv2 we find that both systems have nearly the same in performance for ResNet50. However, the Google TPU is more cost-efficient.
So the TPU is a cost-efficient cloud-based solution? Yes and no. On paper and for regular use it is more cost-efficient. However, if you use best practices and guidelines as used by a fastai team and fastai library you can achieve faster convergences at a lower price — at least for convolutional networks for object recognition.
With the same software, the TPU could be even more cost-efficient, but here also lies the problem: (1) TPUs are not available for the use of the fastai library, that is PyTorch; (2) TPU algorithms rely mostly on the internal Google team, (3) no uniform high-level library exist which enforces good standards for TensorFlow.
All three points hit the TPU as it requires separate software to keep up with new additions to the deep learning algorithm family. I am sure the grunt-work has already been done by the Google team, but it is unclear how good the support is for some models. The official repository for example only has a single model for NLP with the rest being computer vision models. All models use convolution and none of them recurrent neural networks. With comes together with a now rather old report from February, that the TPUv2 did not converge when LSTMs were used. I could not find a source if the problem has been fixed as of yet. On the other hand, one big milestone in NLP was BERT which is a big bidirectional transformer architecture which can be fine-tuned to reach state-of-the-art performance on a wide range of NLP tasks. TPUs were critical for training the training bidirectional transformers on a lot of data. In total 256 TPU-hours were needed to train a base model of BERT. How does this compare to GPUs? I wrote a detailed analysis on this and find that new RTX GPUs are critical for transformer performance and that one can expect a run-time of about 400 GPU-hours. This shows that TPUs perform quite well on this task and TPUs have a big advantage over GPUs for training transformers.
To conclude, currently, TPUs seem to be best used for training convolutional network or large transformers and should be supplemented with other compute resources rather than a main deep learning resource.
Amazon: Reliable but Expensive.
A lot of new GPUs have been added to AWS since the last update of this blog post. However, the prices are still a bit high. AWS GPU instances can be a very useful solution if additional compute is needed suddenly, for example when all GPUs are in use as is common before research paper deadlines.
However, if it ought to be cost-efficient then one should make sure that one only runs a few networks and that one knows with a good certainty that parameters chosen for the training run are near-optimal. Otherwise, the cost will cut quite deep into your pocket and a dedicated GPU might be more useful. Even if a fast AWS GPU is tempting a solid GTX 1070 and up will be able to provide good compute performance for a year or two without costing too much.
So AWS GPU instances are very useful but they need to be used wisely and with caution to be cost-efficient. For more discussion on cloud computing see the section below.
What Makes One GPU Faster Than Another?
Your first question might be what is the most important feature for fast GPU performance for deep learning: Is it CUDA cores? Clock speed? RAM size?
While a good simplified advice would have been “pay attention to the memory bandwidth” I would no longer recommend doing that. This because GPU hardware and software developed over the years in a way that bandwidth on a GPU is no longer a good proxy for its performance. The introduction for Tensor Cores in consumer-grade GPUs complicates the issue further. Now a combination of bandwidth, FLOPS, and Tensor Cores are the best indicator for the performance of a GPU.
One thing that to deepen your understanding to make an informed choice is to learn a bit about what parts of the hardware makes GPUs fast for the two most important tensor operations: Matrix multiplication and convolution.
A simple and effective way to think about matrix multiplication is that it is bandwidth bound. That is memory bandwidth is the most important feature of a GPU if you want to use LSTMs and other recurrent networks that do lots of matrix multiplications.
Similarly, convolution is bound by computation speed. Thus TFLOPs on a GPU is the best indicator for the performance of ResNets and other convolutional architectures.
Tensor Cores change the equation slightly. They are very straightforward specialized compute units which can speed up computation — but not memory bandwidth — and thus the largest benefit can be seen for convolutional nets which are about 30% to 100% times faster with Tensor Cores.
While Tensor Cores only make the computation faster they also enable the computation using 16-bit numbers. This is also a big advantage for matrix multiplication because with numbers only being 16-bit instead of 32-bit large one can transfer twice the number of numbers in a matrix with the same memory bandwidth. I wrote in detail about how this change from 32-bit to 16-bit affects matrix multiplication performance, but in general, one can hope for speedups of 100-300% when switching from 32-bit to 16-bit and an additional of about 20% to 60% for LSTMs using Tensor Cores.
These are some big increases in performance and 16-bit training should become standard with RTX cards — never use 32-bit! If you encounter problems with 16-bit training then you should use loss scaling: (1) multiply your loss by a big number, (2) calculate the gradient, (3) divide by the big number, (4) update your weights. Usually, 16-bit training should be just fine, but if you are having trouble replicating results with 16-bit loss scaling will usually solve the issue.
So overall, the best rule of thumb would be: Look at bandwidth if you use RNNs; look at FLOPS if you use convolution; get Tensor Cores if you can afford them (do not buy Tesla cards unless you have to).
Cost Efficiency Analysis
The cost-efficiency of a GPU is probably the most important criterion for selecting a GPU. I did a new cost performance analysis that incorporated memory bandwidth, TFLOPs, and Tensor Cores. I looked at prices on eBay and Amazon and weighted them 50:50, then I looked at performance indicators for LSTMs, CNNs, with and without Tensor Cores. I took these performance numbers and averaged them to receive average performance ratings with which I then calculated performance/cost numbers. This is the result:
From this data (1, 2, 3, 4, 5, 6, 7), we see that the RTX 2070 is more cost-efficient than the RTX 2080 or the RTX 2080 Ti. Why is this so? The ability to do 16-bit computation with Tensor Cores is much more valuable than just having a bigger ship with more Tensor Cores cores. With the RTX 2070, you get these features for the lowest price.
However, this analysis also has certain biases which should be taken into account:
(1) Prices fluctuate. Currently, GTX 1080 Ti, RTX 2080 and RTX 2080 Ti cards seem to be overpriced and they could be more favorable in the future.
(2) This analysis favors smaller cards. The analysis does not take into account how much memory you need for networks nor how many GPUs you can fit into your computer. One computer with 4 fast GPUs is much more cost-efficient than 2 computers with the most cost/efficient cards.
Required Memory Size and 16-bit Training
The memory on a GPU can be critical for some applications like computer vision, machine translation, and certain other NLP applications and you might think that the RTX 2070 is cost-efficient, but its memory is too small with 8 GB. However, note that through 16-bit training you virtually have 16 GB of memory and any standard model should fit into your RTX 2070 easily if you use 16-bits. The same is true for the RTX 2080 and RTX 2080 Ti.
General GPU Recommendations
Currently, my main recommendation is to get an RTX 2070 GPU and use 16-bit training. I would never recommend buying an XP Titan, Titan V, any Quadro cards, or any Founders Edition GPUs. However, there are some specific GPUs which also have their place:
(1) For extra memory, I would recommend an RTX 2080 Ti.
(2) For extra performance, I would recommend either (a) an RTX 2080 Ti or (b) an RTX 2070 now and sell & upgrade to an RTX Titan in 2019 Q1/Q2.
(3) If you are short on money I would recommend a Titan X (Pascal) from eBay or a GTX 1060 (6GB). If that is too expensive go for a GTX 1050 Ti. If that is still too expensive have a look at Colab.
(4) If you just want to get started with deep learning a GTX 1050 Ti (4GB) is a good option.
(5) If you can wait definitely wait it out: Both GTX 1080 Ti and RTX 2080 Ti are great cards but they have crazy prices right now. Their prices likely stabilize in a month or two.
(6) You want to learn quickly how to do deep learning: Multiple GTX 1060 (6GB).
(7) If you already have a GTX 1080 Ti or GTX Titan (Pascal) you might want to wait until the RTX Titan is released. Your GPUs are still okay.
I personally wanted to get an RTX 2080 Ti, but since the RTX 2070 release it is a much more cost-efficient card and with a virtual 16-bit memory which is equivalent to 16 GB in 32-bit I will be able to run any model that is out there.
Deep Learning in the Cloud
Both GPU instances on AWS and TPUs in the Google Cloud are viable options for deep learning. While the TPU is a bit cheaper it is lacking the versatility and flexibility of AWS GPUs. TPUs might be the weapon of choice for training object recognition pipelines. For other work-loads AWS GPUs are a safer bet — the good thing about cloud instances is that you can switch between GPUs and TPUs at any time or even use both at the same time.
However, mind the opportunity cost here: If you learn the skills to have a smooth work-flow with AWS instances, you lost time that could be spent doing work on a personal GPU, and you will also not have acquired the skills to use TPUs. If you use a personal GPU, you will not have the skills to expand into more GPUs/TPUs via the cloud. If you use TPUs you are stuck with TensorFlow and it will not be straightforward to switch to AWS. Learning a smooth cloud work-flow is expensive and you should weight this cost if you make the choice for TPUs or AWS GPUs.
Another question is also about when to use cloud services. If you try to learn deep learning or you need to prototype then a personal GPU might be the best option since cloud instances can be pricey. However, once you have found a good deep network configuration and you just want to train a model using data parallelism with cloud instances is a solid approach. This means that a small GPU will be sufficient for prototyping and one can rely on the power of cloud computing to scale up to larger experiments.
If you are short on money the cloud computing instances might also be a good solution, but the problem is that you can only buy a lot of compute per hour when you only need some little for prototyping. In this case, one might want to prototype on a CPU and then roll out on GPU/TPU instances for a quick training run. This is not the best work-flow since prototyping on a CPU can be a big pain, but it is a cost-efficient solution.
With the information in this blog post, you should be able to reason which GPU is suitable for you. In general, I see two main strategies that make sense: Firstly, go with an RTX 20 series GPU to get a quick upgrade or, secondly, go with a cheap GTX 10 series GPU and upgrade once the RTX Titan becomes available. If you are less serious about performance or you just do not need the performance, for example for Kaggle, startups, prototyping, or learning deep learning you can also benefit greatly from cheap GTX 10 series GPUs. If you go for a GTX 10 series GPU be careful that the GPU memory size fulfills your requirements.
Best GPU overall: RTX 2070
GPUs to avoid: Any Tesla card; any Quadro card; any Founders Edition card; Titan V, Titan XP
Cost-efficient but expensive: RTX 2070
Cost-efficient and cheap: GTX Titan (Pascal) from eBay, GTX 1060 (6GB), GTX 1050 Ti (4GB)
I have little money: GTX Titan (Pascal) from eBay, or GTX 1060 (6GB), or GTX 1050 Ti (4GB)
I have almost no money: GTX 1050 Ti (4GB); CPU (prototyping) + AWS/TPU (training); or Colab.
I do Kaggle: RTX 2070. If you do not have enough money go for a GTX 1060 (6GB) or GTX Titan (Pascal) from eBay for prototyping and AWS for final training. Use fastai library
I am a competitive computer vision or machine translation researcher: GTX 2080 Ti; upgrade to RTX Titan in 2019
I am an NLP researcher: RTX 2070 use 16-bit.
I want to build a GPU cluster: This is really complicated, you can get some ideas here
I started deep learning and I am serious about it: Start with an RTX 2070. Buy more RTX 2070 after 6-9 months and you still want to invest more time into deep learning. Depending on what area you choose next (startup, Kaggle, research, applied deep learning) sell your GPU and buy something more appropriate after about two years.
I want to try deep learning, but I am not serious about it: GTX 1050 Ti (4 or 2GB). This often fits into your standard desktop. If it does, do not buy a new computer!
Update 2018-11-05: Added RTX 2070 and updated recommendations. Updated charts with hard performance data. Updated TPU section.
Update 2018-08-21: Added RTX 2080 and RTX 2080 Ti; reworked performance analysis
Update 2017-04-09: Added cost efficiency analysis; updated recommendation with NVIDIA Titan Xp
Update 2017-03-19: Cleaned up blog post; added GTX 1080 Ti
Update 2016-07-23: Added Titan X Pascal and GTX 1060; updated recommendations
Update 2016-06-25: Reworked multi-GPU section; removed simple neural network memory section as no longer relevant; expanded convolutional memory section; truncated AWS section due to not being efficient anymore; added my opinion about the Xeon Phi; added updates for the GTX 1000 series
Update 2015-08-20: Added section for AWS GPU instances; added GTX 980 Ti to the comparison relation
Update 2015-04-22: GTX 580 no longer recommended; added performance relationships between cards
Update 2015-03-16: Updated GPU recommendations: GTX 970 and GTX 580
Update 2015-02-23: Updated GPU recommendations and memory calculations
Update 2014-09-28: Added emphasis for memory requirement of CNNs
I want to thankfor helping me to debug and test custom code for the GTX 970; I want to thank Sander Dieleman for making me aware of the shortcomings of my GPU memory advice for convolutional nets; I want to thank Hannes Bretschneider for pointing out software dependency problems for the GTX 580; and I want to thank Oliver Griesel for pointing out notebook solutions for AWS instances.