It's not too much a matter of opinion. In England during the 14th century, a gentleman whose lands brought in an income of at least 40 pounds per annum was expected by law to become a knight.
So was there a difference between a 40 lb / year knight and a Duke? Of course. But that minimum barrier to entry still meant there was no such thing as a 'poor' member of the knightly class. Even the poorest knight was still obscenely wealthy compared to the masses. It's how society was set up. The three estates and all that. Those who fight were intended to be rich.
The aristocracy, of which knights were all members generally ranged in income from 20lb/year to over 12,000 lb per year the at high end of the most elite landed, titled nobility. That accounts for 1% of the population. Another 1% were clergy, and the other 98% were everybody else.
So when you say lower or middle class, you're talking about the lower portion of a fraction of the top 1%, not the implication that so many make that there were knights who barely had a cottage to call home, and no servants.
And even then, the lower portion of the 1% were likely gentle born squires, or men-at-arms, they couldn't afford to be Knights Bachelor, much less Knights Banneret. And yes, you're absolutely right when you say some people avoided being knighted because of the military responsibility that came along with it.
To put it in modern terms, it would like comparing it to the 1%'ers here in the US. While the richest of the 1% of the population controls like 90% of the wealth or whatever it is these days. Even the poorest 1%'er still makes us look like 'bums.' And even the poorest 1%'er can afford some of the finest luxuries in life. So too in the Late Middle Ages was the wealth distribution so skewed that even the poorest knight was by no means not RICH. So yes, you can still portray a knight on the low end of the spectrum, but that low end is still in the top 1% of all the wealth in whatever nation you're talking about, and there's still a minimum level of luxury required to accurately portray someone of that status.