I’m not surprised with so few pixels. You can improve things by ‘blending’ some pixels near the rough bits. The same way laser and inkjet printers smooth the edges of fonts.
But you could spend more time blending than the results give. You’re back in the realm of the old 8x8 fonts on early computers.
Try looking at your circle at 10m away and I bet it looks much better…
Anyway back to your original problem: You might find it better to ‘union’ two sets of variables over each other. The first for your LED matrix as it is laid out. The second over the top for how the LED strip is actually wired.
Then you can write to the matrix as you wish it to appear. Once you have finished writing, you can directly read out the union directly to the LED strip. That should simplify your coding and make it simpler to understand or scale to a larger array. This is because you have the two different mappings using the same memory.