If there are no open ports between the attic and your home’s interior, it is unlikely that the spores have penetrated the ceiling surface. Even the attic access panel / hatch is only open periodically and assuming that some spores would find their way into your home (while it is open), they need favorable environment to grow.
The situation would be different if you start noticing dark stains on the ceiling below the attic. This could be also a result of improper attic ventilation and insulation deficiencies as well.
Spreading into your walls… again, there must be favorable environment for this to happen.
Open space between the exterior brick and interior wall finishing material (drywall, plaster, paneling, etc.)
Improperly insulated or not insulated wall cavities
Still, getting from your attic into the walls… I don’t think so. However, if there is no insulation inside the walls and they are open somehow into the attic space they might just act as a warm air “feeder” and accelerate mold growth.
might be a serious health concern if you’re exposed to it and if you have mold related allergic reactions. While performing home inspections a few minutes in a mold contaminated attic or crawlspace would always make me dizzy, give me a headache, and sometimes even cause nausea. Others may have no such reactions at all or end up in an emergency rooM after beeing exposed.