I ran into a problem like this recently where new version of numpy was incompatible with the matplotlib version and blocked its installation. In miniconda, you can work around this by specifying the version of numpy like this (tweaking one of our example configuration commands) :
conda create -y \
-n py39_afni_tiny \
python=3.9 \
"matplotlib>=2.2.3" numpy=1 scipy \
flask flask-cors