The exact number of digits you need to know isn't relevant. The problem is that you can spend someone else's money just by knowing their digits, the recipient is expected to reject that money if it's not your digits, but is given no way to truly verify identity or ownership.
The problem is that you can spend someone else's money just by knowing their digits
That's part of it. The other parts are that unlike a PIN or password, people routinely tell others what those digits are, and that the system works as a pull (the merchant decides when to collect the money and informs the customer's bank via the payment processing system) instead of a push (the customer decides when and where to send the money and informs their own bank).
Most of the problems with security, fraud, chargebacks and related areas in the card payment industry ultimately start from this fundamentally flawed model.
"The exact number of digits you need to know isn't relevant"
The number of digits doesnt matter but CVV vs. no CVV does make a difference. CVV is the way to verify that you are allowed to use the given credit card number (its actually the second V in the initialization).
CVV is completely separate from the way the credit card number is generated. If someone else has your card number and CVV it implies:
Either your numbers were stolen directly from your card or your information was stolen from some third party server.
If your information was stolen from, say a merchant's server, it implies that they did not properly encrypt your credit number and that they stored CVV which should not even be in their database to begin with.
