Across vendors, generally, Nvidia still dominates currently. People are adding more support into ML libraries for other vendors via (second-class imo) alternate backends but expect to be patient if you're waiting for the day when there is healthy competition.
IMO, I'd say: if you can save up for it, get a 4090; if you can save up for half a 4090, get a 3090 - seen many going for 600-800 now. If you can save up for half a 3090, I'm not sure - depends on if you prefer speed or VRAM. If it were me, I'd pick more VRAM first.
I think the main qualitative change (beyond bigger numbers in the spec) for an enduser of machine learning libraries from 8.6 -> 8.9 (ie 3090 -> 4090) is this line:
> 4 mixed-precision Fourth-Generation Tensor Cores supporting fp8, fp16, __nv_bfloat16, tf32, sub-byte and fp64 for compute capability 8.9 (see Warp matrix functions for details)
ie new precisions will be builtin to eg pytorch with hw-level/tensor core support
edit: btw you probably ought to stick to a consumer gpu (ie not professional) if you want it to be generally versatile while also easy to use at home.
This. If you insist on being as cheap as possible, shoot for a 12gb card but be aware that you'll be missing out on the throughput of higher-end models. The 3060 is popular for this I think, but you'll probably want a better card with more CUDA cores to max out performance.
Cards like the A770 are awesome, but barely even support raster drivers on DirectX. Your best bang-for-buck options are going to be Nvidia-only for now, with a few competing AMD cards that have fast-tracked Pytorch support.
I purchased a 3060 specifically for the 12gb of memory last November and I've been able to run llama, alpaca, stable diffusion out of the box for everything without ever having any memory issues. Training is usually overnight, and a stable diffusion will render in ~5 seconds, llama will do 20 tokens/second.
I would say start with the 3060 for 250 bucks, and if you're still loving it after a couple months, drop 10x more on a quadro.
My only word of advice is get docker setup and install the nvidia docker toolkit to passthrough your gpu to docker images -- the package management for all these python ai tools is a hell-scape, especially if you want to try a bunch of different things.
> If I grab a random model from Hugging Face, will it accelerate?
Probably, it depends more on how you configure the inferencing software. Most software that supports acceleration starts with CUDA or CUBLAS, so you should be good.
> If I run Blender, kdenlive, or DaVinci Resolve, will it accelerate?
Yep. If you're running Linux, some distros might be a little iffy about shipping the proprietary/accelerated versions of this software, but most are fine. The Flatpak versions should all have Nvidia acceleration working out-of-box, if you do encounter any issues.
> Is there a line where things break?
Yes, but you can avoid it by choosing smaller quantizations and giving yourself a few gigs of VRAM headroom. In my experience, it's always better to select a model smaller than you need so you're not risking an OOM crash (I've got a 3070ti).
Lotta other great advice in this thread, though! Good luck picking something out.
For Blender, you can actually check crowd-sourced public benchmarks for basically any CPU and GPU you want to compare https://opendata.blender.org/
That site is a goldmine for perf benchmarks, I actually use that site if I want to do a rough comparison of GPU performance across models for 3D / animation / gaming uses. Even though that is Blender specific, I'm pretty confident the results apply to any usage in the same class of applications.
For Blender, you must carefully read requirements of your software.
Unfortunately, only NN software are more or less standardized, so in many cases, you could choose best fit for your pocket, but all other could be tightly coupled not even to one brand, but to one model. For example, I've seen some software which work in Nvidia-960; I'm not sure about 1060; it don't work on 2060 (for some reason, developers avoid this series).
Also remember, Nvidia prohibited to virtualize their hardware for all gaming cards (only professional lines allowed), even pushed virtual machines vendors to extract support of Nvidia gaming cards (for example, Xen have official statement on this).
But AMD and Intel does not follow Nvidia in this controversy, and all their officially supported cards could work under virtual environment.
This is not unbreakable issue, for example could use old drivers or from independent open source, but in some cases this could be very annoying.
Across vendors, generally, Nvidia still dominates currently. People are adding more support into ML libraries for other vendors via (second-class imo) alternate backends but expect to be patient if you're waiting for the day when there is healthy competition.
IMO, I'd say: if you can save up for it, get a 4090; if you can save up for half a 4090, get a 3090 - seen many going for 600-800 now. If you can save up for half a 3090, I'm not sure - depends on if you prefer speed or VRAM. If it were me, I'd pick more VRAM first.
re: compute capability, you can see here:
- which GPUs have what cc: https://developer.nvidia.com/cuda-gpus
- what cc comes with what features: https://docs.nvidia.com/cuda/cuda-c-programming-guide/index....
I think the main qualitative change (beyond bigger numbers in the spec) for an enduser of machine learning libraries from 8.6 -> 8.9 (ie 3090 -> 4090) is this line:
> 4 mixed-precision Fourth-Generation Tensor Cores supporting fp8, fp16, __nv_bfloat16, tf32, sub-byte and fp64 for compute capability 8.9 (see Warp matrix functions for details)
ie new precisions will be builtin to eg pytorch with hw-level/tensor core support
edit: btw you probably ought to stick to a consumer gpu (ie not professional) if you want it to be generally versatile while also easy to use at home.