So I was reading an older post over at the Points of Light blog entitled D&D Doesn't Use Vancian Magic. David's point is that although the magic system in D&D is often called "Vancian", it isn't actually an accurate simulation of how magic use is described in Vance's books, the Dying Earth novels in particular. David provides this quote from Vance's work:
Maziriam made a selection from his books and with great effort forced five spells upon his brain: Phandaal's Gyrator, Felojun's Second Hypnotic Spell, The Excellent Prismatic Spray, The Charm of Untiring Nourishment, and the Spell of the Omnipotent Sphere. This accomplished, Maziriam drank wine and retired to his couch.
So, yes, the mage does go through his spell books and load spells into his brain. But there are not spell slots, or specified numbers of spells of particular levels--although the mages in Vance's books do seem to have a limit of how much spell power their brains can handle.
Therefore to make D&D magic more Vancian you would need to retain the spell books and spell prep time, but remove the spell slot grid where at any given level you may only memorize X number of spells of each level. But this being a game you still want to have a way to limit the numbers/levels of spells which a mage with a given Intelligence and casting level can have mentally prepared at any given time.
And then I thought of giving each magic-using class a sheet with a grid and tiles of different shapes for the spells. Larger spells would have larger tiles, thus taking up more space on the grid. As long as there's room left on the grid to fit in a particular spell you can memorize it and place the tile. That brought to mind the old game Tetris, where you have to fit tiles of different shapes (made of connected squares) into a rectangular space. So what if the spell-tiles were not just larger, but also of more complex shapes (but probably still an assembly of squares) as the spell level increased?
That started sounding cool, until I realized that making up a set of spell tiles would be work, especially if you wanted more complex shapes. And then I remembered those puzzles where you place geometric shapes (triangles, squares, and parallelograms) to make shapes or fill in a square: tangrams. I think the easiest gaming aid would use magnetic pieces and a metal board. The board would be big enough to handle your game's maximum spell-slot capacity. I'd also want a quick reference guide printed on the back for each spell casting class showing the capacity area and how many of each geometric shape the character gets at each casting level.