The problem is... the arrays HAVE to be visible AND accessible to the mouse, it's part of the abs. concept: you have to be able to draw your own graphics using the mouse.
the arrays being visible alone should't cause excessive cpu consumption, but if you built a mechanism that refreshes the content of those (visible) arrays, then it could happen..
i haven't seen your patch, so i can not tell you more, but, generally, using many GUI objects at a high refreshing rate, may cause tremendous cpu consumption, since pd lacks the option for hardware acceleration for graphics, as i recently learned from this list.