Early-stage fruit yield prediction plays a key role in supporting timely agronomic decisions, enhancing market planning, and empowering farmers with data-driven insights. Over the years, most approaches to yield estima tion have focused on fruit counting techniques, typically performed just before harvest. While these methods have proven useful, they often come into play late in the cultivation cycle, limiting their impact on early planning and resource optimization. In this work, we introduce a com prehensive baseline framework for predicting mango yield at an earlier stage- during flowering- using image-based learning. Our contributions are twofold: (i) Our approach combines a SegFormer-based segmentation model with at tention fusion and a regression pipeline to estimate yield from images, while also exploring the role of contextual features such as weather and scale. (ii) This work introduces a novel benchmark and an enriched dataset, paving the way for scalable, automated tools that can assist farmers and stakeholders in making proactive decisions throughout the mango growing season. Our work demonstrates that, for multi-modal yield prediction, weather factors complement visual features, while scale and weather features further reinforce each other, leading to improved results. Our single image model, basedontheSegFormer-B1encoder, achieved a mean absolute error (MAE) of 7.21, R² of 0.8, and mean squared error (MSE) of 96.83. These results highlight the potential of vision-based models for yield estimation from early-stage flowering cues. To the best of our knowledge, this is the first work to address the prediction of mango yield using images from the flowering stage and weather data.