Over the years, I’ve had many people ask me how we make decisions in the development team with regards to choosing features, fixing bugs, and testing new versions. This week, I thought I would give you some insight into how we track and manage the bug reports and support issues that we learn about through our technical support department. Roy Faltesek manages our LabVIEW Testing and Product Support Engineers group. Part of their job is to be the interface with and be the escalation path for our Application Engineers who talk to you on the telephone or through email to answer your questions.
What happens when a customer reports a bug?
Roy Faltesek: Almost all bugs get reported to us via a call or email to our Application Engineering department. An Application Engineer (AE) works with the customer to confirm the problem and then files a Corrective Action Report (CAR), which is essentially a work order for Engineering to investigate and fix the bug. Our CAR database allows us to track and manage bug reports from reporting to their final resolution.
We encourage AEs to provide customers with the CAR number associated with their reported bug, which not only reassures them that the problem has been reported properly to engineering, but is also useful if anyone wants to inquire about the issue later, perhaps to check for additional workarounds or to see if the problem has been fixed in a subsequent release.
Do we fix all bugs that we get reports about?
RF: We don’t fix all bugs that are reported, but we fix a lot. For example, over a thousand bugs were fixed in LabVIEW 8.5. We prioritize and classify all bugs that are reported and, in general, we fix the highest priority bugs first. Factors that go into setting the priority are the number of users who would encounter the problem, the pain level associated with encountering the problem, and the ability of users to avoid the problem. Crashes and hangs are obvious high priority problems. Controls not aligned properly are obvious low priority problems.
Engineering reviews all reported CARs, assigns them to the right developer and tracks their status. Each developer decides how to designate their assigned CARs. Some may be closed as Not Reproducible, some as Not a Bug (meaning the behavior will not be changed) and some as Rejected (mostly used for what is essentially an enhancement request reported as a bug). Of the ones that are fixed, the delay between the time it is reported and the time the fix is available to our users can sometimes seem interminable. A bug that is reported shortly before release might be fixed relatively quickly, but even if the problem is minor, that fix might not be available in a released product until the next major or minor release, which could be a year later.
What can customers do in the interim?
RF: When a customer reports a bug, their immediate need is not for a product fix, but for a workaround. The user has a task they are trying to accomplish or a function they need to perform. If the bug they have encountered is keeping them from performing their job, our priority is to work with the user to develop an acceptable workaround. It’s extremely rare that we are not able to do this relatively quickly, but occasionally it can take a few days or even weeks and may involve working with engineering to develop a workaround. Customers should expect that the AE and possibly Engineering will establish a workaround every time a customer submits a CAR.
How can customers track their bugs?
RF: If the problem is of interest to numerous users or would be especially painful when encountered, we’ll document the problem in the Known Issues List for that release, a live web document which is updated as we become aware of new issues. We have been maintaining these documents since LabVIEW 8.5 and we’ll continue to do so for new versions of LabVIEW and other NI software in the future.
The LabVIEW 8.6.x Known Issue List
The LabVIEW 8.5.x Known Issue List
When the CAR is fixed in a new release, the CAR number will then be documented in the Bug Fix List for that release if we think the CAR would be of interest to users.
The LabVIEW 8.6.1 Bug Fix List
The LabVIEW 8.5.1 Bug Fix List
We are continuing to look into ways to make more information available to our users.
Roy plays a very important role in our development team – you can bet we’ll hear from him again. In talking with users, I know there are more things you would like to see from us regarding bug tracking and reporting. We are always discussing ways we can make this information available directly to you on the Web. For now, here are a few questions to spur some comments from you (I have a pretty good idea of what you want next, but let’s open it for comment):
1. Are you satisfied with our CAR tracking process and fix rate? Our current approach is to make sure you have a workaround and then you then work the fix into a future release. Do you feel like we do a good job at resolving the right issues? How would our being more transparent help you?
2. Do we document the right known issues? Is that a document that you refer to while developing? How would you like to see that documentation improve?
3. Once a bug you reported is fixed, do you want us to contact you? With our annual release cycle it can take several releases before your CAR gets implemented. Are you satisfied with just a workaround, or do you see value in us “closing the loop.”
March 26, 2009 at 12:27 am |
Hi,
great site! very interesting!
Bye, Andrea from italy!
March 26, 2009 at 1:44 am |
John, thanks for the insight into the process.
From my side of the fence, NI is typically unable to decouple LV bug fixes from new feature implementations. This *seems* to lead to a couple of negative effects: long bug fix cycles (that is, only with new releases), and the need to charge for bug fixes via the SSP (unlike, say, free replacement on similarly buggy hardware).
Are we likely to see more frequent, bug-fix-only releases of LV in the future?
Thanks, Joe Z.
March 26, 2009 at 1:21 pm |
Joe,
In the last few years, we have made a conscious effort to do exactly what you suggest – separate new features into “major” releases that generally come out at NIWeek from “maintenance” releases that are strictly bug fixes only. This is part of our move to annual releases.
In the past, we would release a huge set of features in a major release, and then try to monitor the discovery of issues as they came in and do a maintenance release if and when necessary. Now, we do it as a rule about 6 months after a feature release. Version 8.6.1 became available just a few weeks ago. Even for the most solid releases, we make fixes available 6 months later.
We’ve discussed the notion of ongoing bug fixes on older releases – for example, doing an 8.6.2, 8.6.3, etc. – but that is not sustainable. Instead, by releasing in smaller chunks annually, our goal is to make the upgrade process much smoother so you can stay up to date and get all the bug fixes as easily as possible.
March 26, 2009 at 8:11 am |
Hi John,
Thanks for all your team’s work fixing, tracking, and reporting back on bugs.
For every software project affected by a LabVIEW bug, we like to keep track of the CARs of each bug. Then, we try to identify when the bugs get fixed in newer versions of LabVIEW. If our project is affected by a bug that gets fixed in a newer version of LabVIEW, this is ammunition that we can use to justify upgrading the project/product to the newer version of LabVIEW. The LabVIEW upgrade process for existing projects is a serious decision and we need to understand all of the changes that might occur, for the better and worse.
Other reasons that, as a bug submitter, it’s nice to close the loop are 1) Closure, the desire to know “what happened, and 2) Impact, the desire to know that you’ve made a difference. Both of these add a lot of value to the bug submitter, who has probably invested a lot of time in reporting the bug and possibly also lost time being bitten by the bug or working around it. An investment in a little time and effort on NI’s side to recognize the submitter’s efforts makes a big difference.
Thanks,
-Jim
May 8, 2009 at 7:05 am |
Hi John,
I’d like to second Jim’s comment.
The new LV2009ß is the first after sevral cycles I decided NOT to participate. It’s just taking too much effort and leaving me with even tighter deadlines on my own projects. A little more feedback and ‘motivation’ could have made a difference…
Another point: I got the feeling that asking on Info-labview or Lava is much more helpfull that calling NIs AEs. In allmost all cases it took several days or even weeks to get a work-around, while typically the answers from community members are much quicker.
Just my € 0.02!
Greetings from Germany!
–
Uwe