The #3 common mistake…Improper Retention of Verification Results
DO-254 has several objectives related to verification, which can be accomplished through reviews, analysis or test. The results of these activities must be retained. A common mistake in the DO-254 process centers on retention (or lack thereof) of verification records associated with reviews and analysis as well as retention of test results. This, in addition to review of the final verification results (tests and analysis) is an area that applicants commonly miss or do not give proper emphasis.
Verification records should contain a clear correlation to the pass/fail criteria, which could be a development standard or a test result. These verification records should contain the author/reviewer, date, and any items used in the including their versions. Any failures or issues found should be correlated to the standard that has been violated. Test results should be clearly linked to their associated tests and requirements, and should be reviewed and summarized in the verification results documents. Test Results should be reviewed to be sure that the actual and expect results are giving the correct results and that the tests are passing. Review results, which must also be retained, are important because they show compliance to standards during development (and for DAL A and B projects the review participant’s names provide evidence of independence).
If you need help understanding how to manage the verification environment and artifacts in your DO-254 program, this is covered in the Patmos Engineering Services “DO-254 Airborne Electronic Hardware Certification” training.
Tammy Reeve has been involved in the certification of hardware (DO-254), software (DO-178C), and systems (ARP 4754A and related) for nearly 20 years.
The # 4 mistake is…Not Understanding the Purpose of Traceability.
DO-254 is a requirements-based process. As part of this process, requirements must be captured and validated, and then traced (via a Traceability Matrix) into both the corresponding design implementation and verification (test cases, test procedures and results). Traceability is an integral (i.e., Supporting) process verification activity. What traceability provides, if done properly, is assurance that all requirements have been implemented, that all portions of the design tie to requirements, and that the design as implemented behaves as the requirements say it should. Traceability done throughout the development activity will identify derived requirements that need validation. But all of this entails integrating traceability into the process as you go. By integrating traceability, it offers insights into design completion and verification coverage, and can even help find design bugs!
Instead what I often see is project teams who think of the Traceability Matrix is an artifact to be generated after the fact, with no thought about it until it needs to be reviewed during SOI audits. If you wait until the last minute to create a traceability matrix, as opposed to incorporating traceability into each phase of design (used as a analysis tool as part of requirements, design, code and test reviews), you miss the point and the benefits that this activity provides. So in your program, try to remember that Requirements Traceability isn’t an output of the program as much as it is a tool to be used throughout the program to help you understand your progress and provide an added measure of verification and visibility for change impact for each phase of the process.
If you need help understanding how to best implement traceability in your DO-254 program, this is covered in the Patmos Engineering Services “DO-254 Airborne Electronic Hardware Certification” training.
Tammy Reeve has been involved in the certification of hardware (DO-254), software (DO-178C), and systems (ARP 4754A and related) for nearly 20 years.
No one really likes filling out checklists for reviews. So do you have to have them? Truthfully, the answer is no. But for practical reasons, I’d like to say Yes; because they can be a very helpful tool in aiding reviews and showing evidence of compliance to DO-254 objectives.
Many of the verification objectives in DO-254 will be satisfied by reviews and analysis. When you perform reviews to satisfy objectives in DO-254 you need to provide evidence that they were performed with the correct independence and on a controlled, retrievable version of the data under review. You also need a way to show that the specific objective was evaluated and any actions were recorded and resolved in a revised, released version of the data. Checklists provide an excellent means to do this.
Now if you’re doing to take my advice and use checklists, here are a few helpful hints:
Annotate and connect checklist items to the development and verification planning standards and activities.
Develop checklists of a reasonable size so as to be manageable. The point is to help remind reviewers to check items that are required by the standards and activities.
Remember to include the names and roles of the reviewers, the version and title/number of the items under review (including any other items needed to evaluate the data item under review such as trace matrix, standards, etc.)
Be sure that actions recorded from the review are tied back to the questions on the checklist. This ensures you can show evidence that all the review evaluations have been considered and actions closed.
Show how each action was resolved and in which version of the data item it was resolved.
Most importantly know where you keep all these checklists for when the auditor asks for them. Remember these are verification outputs and should be treated with the same formality as Test results.
Remember checklists are verification outputs and should be treated with the same formality as Test results.
If you need a set of checklists, our partner company Airworthiness Certification Services offers templates and checklists. These industry-leading checklists and templates have been used in programs in all regions of the world for many years, and are very reasonably priced.
Tammy Reeve has been involved in the certification of hardware (DO-254), software (DO-178C), and systems (ARP 4754A and related) for nearly 20 years.
A common inquiry from applicants is whether they have to do target testing, why, and how much to do. DO-254, along with FAA Order 8110.105, both emphasize the expectation that the testing of custom micro-coded components should be achieved (as much as possible) on the target hardware to be installed in the aircraft system. It is understood that observability and robustness aspects at the board level related to testing FPGAs, PLDs and ASICS (Airborne Electronic Hardware, or AEH, devices) is not practical for devices with any amount of complexity. Therefore, it is well established that analysis through simulation may be used for verification of the functional aspects of the design. Applicants typically perform functional simulation as well as behavioral simulation for the requirements-based testing and robustness testing aspects for AEH devices. However, the applicant must still ensure these devices perform their intended function in the actual hardware environment (i.e., aircraft system) that will be delivered to the aircraft.
Testing the target (with the integrated AEH device) should cover as much of the AEH device function as possible. The applicant should justify why or where testing on the target is not possible and document this as part of the Hardware Verification Plan and the Hardware Verification Test Procedures. It is important to plan ahead at the conceptual design phase to ensure pin level access to safety-critical functions performed by the AEH device. Keep in mind too that Target Testing does more than just test the AEH device function. Integration testing on the actual target hardware to be provided to the aircraft validates the AEH device interfaces to/from the board circuity and also validates the models used during functional and behavioral simulation of the AEH device.
This is one of the many topics discussed and captured in papers produced by the DO-254 User’s Group. If you’d like to join this group, contact me.
Tammy Reeve has been involved in the certification of hardware (DO-254), software (DO-178C), and systems (ARP 4754A and related) for nearly 20 years.
Process assurance is a really important role in a DO-254 program. If done well, it can save companies a lot of time and expense by making SOI audits go smoothly and avoiding unnecessary rework. As an example, I was recently holding a SOI-3a audit, which includes checking the ability to rebuild and regenerate the test environment and results. While the company had the necessary documentation as part of their Hardware Verification Plan (HVP) there was still a lot of confusion about which files to pull and where the evidence resided – which boiled down to versioning and control of the files. So I suggested two things.
Creating a VCI (verification control index) document to control all the test bench files, models, scripts –basically the collection of items used when testing the FPGA. After all, this repository of data is very similar to the design (hardware) repository, which is controlled by an HCI.
Holding a Process Assurance “test readiness review” prior to the formal test for credit audit to run through the process internally prior to the SOI audit. The Job Aid, Section 3-3, provides a really good guide for what to review.
Running tests without proper conformity of the test environment and design could result in a full rerun of tests, which can take weeks! So doing these two fairly simple things can save time and frustration in the long run.
“… the Process Manager referred to these suggestions as one of the most important tips he ever learned from a DER”
In my example above, these small changes made a world of difference the next time around, and the Process Manager referred to these suggestions as one of the most important tips he ever learned from a DER. Try it yourself and see the difference it can make in your processes.
Patmos Engineering services offers both guidance for your program as well as auditing services. If you need help preparing for SOI-3 – or any audit for that matter – contact me to set up a discussion.
Tammy Reeve has been involved in the certification of hardware (DO-254), software (DO-178C), and systems (ARP 4754A and related) for nearly 20 years.
This blog is a little different from my past ones. I was invited to participate in the “Streamlining Assurance Processes Workshop” (which internally we called the “Meta Objectives working group”). The idea behind this was to provide an “alternative means of compliance” to DO-254, DO-178C, and ARP 4754A, such that at all levels of development, any team on any project at any level of design could demonstrate compliance to a consistent set of development assurance objectives. (Of course, the activities associated with meeting those objectives would vary depending on the program and level of design).
After three meetings, several web meetings, online commenting and much heated discussion, the committee came up with three Overarching Properties that should be met at any/all levels of design. Each Overarching Property contains the following:
statement capturing the property
definition
pre-requisites
constraints
assumptions.
That’s about all I can say about it in this forum as its being officially rolled out at the upcoming FAA Streamlining Assurance Workshop (being held September 13 -15, 2016).
Tammy Reeve has been involved in the certification of hardware (DO-254), software (DO-178C), and systems (ARP 4754A and related) for nearly 20 years.
NOTE: See below for notes from this event, which were added after this blog was first published.
If you’d like help understanding more about how to adopt streamlining processes in your program, Patmos Engineering Services offers custom consulting to assist you with your program. We can provide a package of training, guidance and/or auditing that can assist you at every step of your program.
FAA Streamlining Assurance Processes Workshop
What follows is a recap of what was presented to introduce this topic at this workshop. For further information, please contact the author himself or the FAA.
Streamlining Framework
Peter Skaves, Chief Scientist AEH, Security FAA Peter covered a variety of topics at a high level, foreshadowing the contents of the conference. He mentioned that “streamlining” has been a goal and theme for 20-25 years, but has a newfound focus. The FAA, working with other authorities, is trying to 1) Reduce duplicate approvals across the authorities, 2) Reduce number of audits and stages of involvement, 3) Allow meeting of “Meta Objectives” (uniting objectives for 178C, 254 and ARP 4754A), 4) Use a risk-based approach when creating new policy for SW/HW/Systems.
In terms of the “risk-based” approach, this means doing things that prevent accidents – identifying areas where there have been systematic design escapes and focusing there (potentially relaxing other areas). So they are trying to use this rationale when modifying documents and guidance material. For example, up until five years ago, the focus was on software and hardware, with little policy at the system level. There were escapes in software and hardware, but these sorts of problems would have showed themselves earlier in the systems development process. Thus the need and invocation of ARP 4754A.
He discussed different aircraft types, with potentially different criteria for each type. One of the more interesting things he discussed was the notion of new “00” advisory circulars. These will be published to provide examples and what was previously considered “prescriptive” guidance. (These were referred to later frequently, as an example of where some info currently in Order 8110.105 may move).
He mentioned streamlining SOI audits for software (which was covered in a later session in more detail).
He also talked about two types of focus for AEH: Programmable (custom microcoded components) and COTS. He talked about the questions and challenges surrounding COTS and how/where to address these concerns.
He then spent time talking about the effort to harmonize with EASA on AC 20-152(A), which has been in the works for a while, while getting rid of Issue Papers and Orders and putting content in ACs where they belong.
Section 7 of DO-254 tells us “The configuration management process is intended to provide the ability to consistently replicate the configuration item, regenerate the information if necessary and modify the configuration item in a controlled fashion if modification is necessary.” What do you do if you are using a tool to control some or all of your program data? This is becoming more and more common as the industry has delivered a number of tools that assist with various aspects of DO-254 (and DO-178C) compliance. This automation may be extremely helpful in many cases, but it introduces a new paradigm of tool based data control that requires understanding and exploration.
The inside scoop on what to do and what not to do in your DO-254 program, direct from the expert.
For example, you may have a tool that helps you with compliance by including checklists for various processes. These checklists might be modifiable, they must be reviewed by a team, actions may come from the review, and these actions need to be tracked. This is all done within the tool and all that data needs to be controlled and available, and subject to the pertinent requirements for Life Cycle Data (DO-254 Section 10.0 and DO-178C Section 11.0). So do we meet the content/retention requirements if the artifacts remain embedded in the tool? Or what if you are using Clearcase and you just have a path to a Version Based Object (VOB) for some compliance artifact? Does that meet data configuration and control requirements for that artifact? And what if your auditor doesn’t have access to the tool?
The underlying concern of the DO-254 configuration management requirements are to ensure that the data is always available and always modifiable. So you must ensure this happens even if the tool is not used again in the future. Because of the obligation to continued airworthiness, the applicant must maintain the data for the lifetime that the product flies. This may be far longer than a tool lifetime. Therefore long-term (20+ year) archive/data recovery is the responsibility of tool user, who must be able to extract, archive and potentially resurrect this data for their project. So ensure that whatever tool you are using allows you to do this, that you capture this as part of your plans and execute appropriately as part of your program.
One of the things I see and hear is that we can “finish that certification stuff later.” Many people believe that you can add the compliance data items to the project after you “get it working” or you “know you are finished with design and implementation.” This bottom-up approach to Certification compliance under DO-254 and DO-178 is in direct opposition to why these design assurance documents were drafted. Let me be clear: while there is room for a rapid prototyping flow within the constraints of these design assurance documents, this sort of flow is only used as a way to validate and confirm that you have the right set of requirements.
A bottom-up compliance effort does not equal top-down unless you have relatively simple functionally. Design assurance processes described by DO-254 and DO-178C are both in place to acknowledge the complexity of avionics systems and the inherent knowledge that if you wait to perform verification and validation on the final image you will be faced with the inability to show you adequately identified potential errors in the design. This concept is discussed as the Complexity Barrier [Beizer90] principle: Software complexity (and therefore that of bugs) grows to the limits of our ability to manage that complexity. This is why DO-254 design assurance standard was written primarily for “complex” hardware and there is an acknowledgment that for simple systems the majority of DO-254 does not apply. DO-178C section 12.3.2 allows for “exhaustive input testing” of your software if it is truly simple as an alternate means of compliance. Top Down design assurance defined in DO-254 and DO-178C hinges on the concepts of integral processes and transition criteria throughout the development with levels of independence and process assurance that will help to identify design errors earlier and throughout development. In other words, the whole purpose of “design assurance,” be it in software or hardware, is to demonstrate that during the process, you have taken all pertinent steps to ensure you are catching and/or eliminating design errors at every step of the design (i.e., development) process.[Beizer90] Boris Beizer, Software Testing Techniques. Second edition. 1990
If your management or someone on your team is struggling to understand the purpose and/or basic requirements of compliance – whether its at the hardware, software or systems level – Patmos Engineering Services training can help. We offer a class called “Certification Overview” which covers the fundamentals of compliance at all these levels.
Tammy Reeve has been involved in the certification of hardware (DO-254), software (DO-178C), and systems (ARP 4754A and related) for nearly 20 years.
The regulatory world is in constant flux. This is due to ongoing change in technology as well as advancement and better understanding of safety related aspects for aviation. You might think that the authorities have free reign over this policy. But in reality, industry has a voice with regard to the oversight and regulatory process through FAA public comment, lobbying with congress, and regulatory committee participation. Many of the changes we’re seeing in new policy is due to industry.
The newly published FAA order 8110.105A, which provides guidance to the FAA, DERs and ODAs for regulatory information surrounding DO-254, is just one of the changes in regulatory compliance coming from the FAA. FAA Order 8110.105A is written to supplement RTCA/DO-254, and to provide additional guidance in approval of both simple and complex custom micro-coded components. Here is a summary of the main changes:
It removes Chapter 3 “Determining FAA Involvement in Hardware Projects”, and all references to the FAA Airborne Hardware Job Aid. These changes were made due to industry pressure to allow for a more flexible approach for conducting FAA oversite reviews such as Stage of Involvement audits (SOIs).
Chapter 2 “SEH/CEH Review Process” was restructured and removes references to the SOI #1-#4 audits and job aid.
Newly added Appendix C “Level of Involvement Worksheets” contains the worksheets for determining the Level of FAA involvement (LOFI), which is aligned with the FAA risk-based directives in Order 8040.4A.
These changes allow for a less prescriptive method for involvement for FAA, DER and ODA personnel. However, coordination and agreement on the appropriate level of involvement should be documented in the PHAC and submitted early in the program to ensure sufficient oversight and to avoid issues later. It is a good idea to review the LOFI worksheet in Appendix C of this Order, together with your FAA or ODA representative, and to include this evaluation in the appendix of your PHAC to show justification for the level of involvement reviews and schedule defined in your plan.
If you need help understanding the latest requirements of DO-254 (or DO-178C for that matter), check out the offerings of Patmos Engineering Services. We can consult with you in a number of different ways to assist you in jump starting your program.
Tammy Reeve has been involved in the certification of hardware (DO-254), software (DO-178C), and systems (ARP 4754A and related) for nearly 20 years.
