Autonomous systems are those that evolve over time, and through learning, can make intelligent decisions when faced with unidentified and unknown situations. Artificial Neural Networks (ANN) has been applied to an increasing number of real-world problems with considerable complexity. Due to their learning abilities, ANN-based systems have been increasingly attracting attention in applications where autonomy is critical and where identification of possible fault scenarios is not exhaustive before hand. We have proposed a methodology in which the learning rules that a trained network has adapted can be extracted and refined using rule extraction and rule refinement techniques, respectively, and then these refined rules are subsequently formally specified and verified against requirements specification using formal methods. The effectiveness of the proposed approach has been demonstrated using a case study of an attitude control subsystem of a satellite