I usually use 3 monitors and have no on-board video. I just attach all three monitors to one GPU and it is just fine. The one GPU usually has about 400 MB to 600 MB less GPU memory because the monitors taking up a bit of space, but for most deep learning models that is just fine.

It is powerful but you can get issues with compatibility of libraries. I have no direct experience with AMD cards and there is not much data out there how good the experience is so I cannot give you a full recommendation. If you go ahead with the AMD card it would be great to hear how your experience has been.

I like that thinking! I think that could work. A couple of things though: For BERT I would recommend a GPU with more memory (RTX 2080 Ti). Also, you will live with some caveats: The CPU does not have many cores which can become a problem with you need to load large amounts of data with each mini-batch; this can cost 10-20% runtime performance. For NLP this should not be the biggest problem though. Theoretically, PCIe 3.0 GPUs are compatible with PCIe 2.0 motherboards, but once someone commented here that it did not work for him. So theory does not always work out practically. I would recommend keeping an eye on the return policy if you buy a GPU so that you can return it if you find it is not compatible with your motherboard.

thanks

For NLP research I recommend a RTX 2080 Ti. If you want to run large transformers you need that extra memory. 8 GB of RAM can be limiting.

So, my question is now what GPU would be the best choice for Natural Language Processing like text-to-speech, speech-to-text and other speech recognition models? A Titan RTX is financially impossible, so we thought of the RTX 2080 super (to be launched soon) or the RTX 2080ti, whereas from AMD the Vega VII seems to be quite attractive due to its large VRAM.

Do you have any experience with the hardware concerning compatibility with TF, PyTorch drivers for Linux etc.? Which one would be the best choice? If Nvidia is preferred, is larger VRAM or higher base clock more important for NLP? There is so much contradictory information on the internet that it is hard to discern who really has experience and who just wants to do some marketing (or simply voices prejudice against the companies)

eGPUs should be better for deep learning since, unlike in gaming, you do not need to transfer as much data back and forth between the CPU and the GPU. Most of the computation will stay on the GPU. There are some bottlenecks in certain applications like computer vision, but they are less severe than for gaming. I would expect a decrease of 25-30% in performance which is not too bad!

ASICs are specialized hardware that serves a single purpose. For example, a TPU can compute matrix multiplications. An ASIC for bitcoin mining computes hashes and that is it — it cannot be used for deep learning. If bitcoin tanks it will be useless in any case.