You're not the only one. My brother in law is five days older than me and just got his first pair of jeans last year. It's not completely unusual.
In this case, from a style standpoint, if all you have is slacks, you're not completely out of options. You'll just have to consider the top a little bit more.
There is a rule that I break often enough to teach you the way.
Slacks and t-shirts. This rule can be broken if you're wearing khakis. Especially in the spring/summer months. A pair of khakis, a (VERY CLEAN) white t-shirt or designed shirt (think threadless.com), along with some flip-flops or sandals or some other casual shoe, and maybe a hat of some sort. You will (surprisingly) turn heads.