| | english | español | français |
  Home|RARM Portal|Past Activities|2012-2014   Printer-friendly version

Return to the list of threads...
Forum closed. No more comments will be accepted on this forum.
Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#4740]
POSTED ON BEHALF OF HANS BERGMANS

----

Dear participants of the Online Forum,

It is an honour to me to moderate this session, which is a continuation of the discussions that we previously had on the modalities of testing the Guidance.

In this session 'Participants are invited to provide comments and suggestions for improving the draft concept note and questionnaire with a view to assisting the Secretariat to structure and focus the testing of the Guidance as per decision BS-VI/12'.

As before, the Secretariat and I are very much looking forward to your contributions, and may I suggest that you be as concise and specific as possible in your suggestions and comments.

Thank you very much for your efforts!

Hans Bergmans
posted on 2013-04-07 22:35 UTC by Ms. Manoela Miranda, UNEP/SCBD
This is a reply to 4740 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#4817]
Good afternoon to all! Thanks for the work done by the Secretariat and by you, Hans! We are writing from Mexico (Caroline Burgeff and Francisca Acevedo, CONABIO) with our first comments on the concept note.
We have some questions relating:

I.- “Actual cases of risk assessment” used for the testing.
We believe  an ideal “Actual case of risk assessment “ would be an actual application dossier being submitted for the environmental release of a particular LMO that is currently being analyzed or has been analyzed in a previous period, maybe no longer than 2or 3 years. Doing the testing departing from this kind of information is closest to reality and will probably allow a better testing of the real practicality, usefulness and utility of the Guidance. 
However, we are aware that there may be situations where this might not be possible, for example, cases of parties that haven’t yet received application dossiers for LMO environmental releases, or that haven’t received dossiers dealing with a particular section of the guidance (i.e. concerning the release of LMOs with stacked genes or traits, LM crops with tolerance to abiotic stress, LM mosquitoes or LM trees). In these cases risk assessment processes or risk assessment summaries will be useful for this exercise as long as the testing could be realized considering only the “input” information  used in these assessments and not the “results” obtained.
It is not clear to us what is then meant in the concept note by “Actual case of risk assessment”.
Point (e) on the “actual cases of risk assessment” section of the concept note includes “The technical and scientific data of actual cases of risk assessment used in the testing may originate from various sources, including application dossiers, risk assessment processes or risk assessment summaries”, but the kind of information available in these kind of documents might be very different in nature; in particular It should be considered that risk assessment processes or risk assessment summaries already include an analytical procedure of the initial information, so using them for testing the guidance could be as “trying to do a risk assessment from an already done risk assessment”.

II.- “BCH risk assessment records”.
We do not fully understand in detail why is it necessary/obligatory that  “the summary of each actual case of risk assessment used in the testing must be available through the Biosafety Clearing House (BCH) as a BCH Risk Assessment Record” (points (f,i) of the concept note and  4 of the Questionnaire).  What would the purpose be?
In the case, for instance of willing to use  currently submitted application dossiers for the testing process, relating the output of the testing to a record in the BCH would not necessarily be possible, as the submissions are in process and most probably the corresponding BCH risk assessment record won’t be available yet.  Besides, some parties haven’t finished the actualization process of their BCH submissions, and this would limit  their possibilities to do the testing. As an alternative to this, a section of the questionnaire could include general data of the case(s) considered  for the testing (organism, LMO, liberation site, liberation surface, purpose of the liberation, liberation scale (field trial, or commercial release) etc…) if this is the type of information you are looking for.
We will greatly appreciate if you Hans, and or the Secretariat could explain which is the idea behind this request in particular.

III.- Who gets to report.
We are not sure that limiting the response of the questionnaire to one per Party, other Governments and/or relevant Organizations is the best way of getting feedback from this process. We feel it will certainly limit the input the Secretariat will receive. Couldn´t it be a bit more flexible and admit the questionnaire being submitted by the experts themselves?

Kind regards form Mexico!
Caro and Fran
posted on 2013-04-10 21:48 UTC by Ms. Francisca Acevedo, Mexico
This is a reply to 4817 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#4833]
Dear Francisca and Carolina,

Thank you for breaking the ice by being the first to post and for your very useful feedback. I have consulted with the Secretariat and will try to explain the thought process that went behind the points you raised. 

In decision BS-VI/12, the COP-MOP stated that “The Guidance will be tested nationally and regionally for further improvement in actual cases of risk assessment [...]”. Article 20 of the Protocol also states that Parties shall submit to the BCH summaries of its risk assessments or environmental reviews of LMOs generated by its regulatory process. In order to allow relevant organizations to take part in the testing (as per the COP-MOP decision), the testers could use risk assessment records that were generated through independent or non-regulatory processes and registered in the BCH.   

Having the summary of the actual case of risk assessment used in the testing available through the BCH is a way to guarantee that the condition “actual cases of risk assessment” set out by the COP-MOP in its decision BS-VI/12 is fulfilled. It also ensures that at least part of the information contained in the actual case used to reach conclusions on the usefulness and practicality of the Guidance is publically available in a transparent manner. We feel that simply having questions related to the LMO, its use and likely potential receiving environment would not guarantee that an actual case of risk assessment was used. In short, this would ensure transparency in the testing.

In the event that countries test the Guidance in cases of risk assessment that are being conducted in response to an ongoing process for a decision, the risk assessment summaries may be submitted to the BCH at the same time as the results of the testing. The testing of the Guidance will continue for an extended period of time to enable countries to finish their risk assessments and submit their risk assessment summaries to the BCH. The submission of risk assessment summaries or environmental reviews to the BCH by Parties taking a decision regarding an LMO is an obligation which is being fulfilled in a timely manner by the great majority of Parties      

With regard to your last question “Couldn’t it be a bit more flexible and admit the questionnaire being submitted by the experts themselves?”, to allow experts themselves to submit the questionnaires would be contrary to the criteria set by the COP-MOP decision which encourages “Parties, other Governments and relevant organizations, through their risk assessors and other experts who are actively involved in risk assessment, to test the Guidance in actual cases of risk assessment and share their experiences through the Biosafety Clearing-House and the open-ended online forum”.  Consequently, the questionnaire reporting the results of the testing may not be submitted by the experts themselves but rather submitted through either their respective BCH National Focal Points or through their head offices.

We hope that this explanation helps you and other forum participants better understand the rationale behind the proposed concept note and look forward to further comments and feedback.

Best regards,

Hans Bergmans
posted on 2013-04-11 20:40 UTC by Mr. Hans Bergmans, PRRI
This is a reply to 4833 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#4866]
Dear Hans:
Thanks for answering us!
We are still not completely comfortable with the approach proposed by the Secretariat.
Requiring to link the results of the testing and the risk assessment summaries in the BCH we feel does not add much value but can create serious limitations to the testing process.
For example, in some cases, the risk assessment process involves several agencies inside a countries regulatory system, and a summary might not really be able to transmit the results of any particular of those agencies (even if they all did a risk assessment).
Also, the time lag between a risk assessment being done in the country and it being submitted as a summary to the BCH (because a decision was taken) might be considerable, i.e. we shouldn´t assume that it would work before COPMOP 7.
Regarding “transparency”, what is the added value of linking the testing results to the summaries? Could we think of possible other ways forward?

Finally, regarding our last question on flexibility, you have quoted the decision itself :
“Parties, other Governments and relevant organizations, through their risk assessors and other experts who are actively involved in risk assessment, to test the Guidance in actual cases of risk assessment and share their experiences through the Biosafety Clearing-House and the open-ended online forum”,
Interpreting that “the questionnaire reporting the results of the testing may not be submitted by the experts themselves but rather submitted through either their respective BCH National Focal Points or through their head offices.”……….we understand this, we are not questioning that the questionnaires be submitted through the BCH National Focal Points but rather why only one? Couldn´t it be that many of them (questionnaires) be submitted by Parties,  other Governments or Relevant Organizations…..this would best reflect what the experts are finding through the process of testing.

Again, kind regards to you and all!
Caro and Fran
posted on 2013-04-12 22:51 UTC by Ms. Francisca Acevedo, Mexico
This is a reply to 4866 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#4889]
Dear all,
First of all, thanks to the secretariat for their work on the questionnaire and Hans for moderating this discussion.

My opinion would be that for a proper testing of the Guidance only real applications/dossiers of LMO’s - that have been assessed in the last years by one or several countries- can be used instead of using summary formats. Summary formats would only be useful to others to understand what a reported test result is about. These actual dossiers contain more, and the essential relevant, information for risk assessment than summary formats. Moreover, the dossiers are the ‘actual cases of risk assessment’ that we have to deal with in our daily life and therefore would be the most logical choice with respect to testing. 
Like mentioned before, it would also be very helpful if some of these dossiers will be tested by all testers in order to  compare the feedback from risk assessors with different levels of experience. I realize that there may be a problem with getting access to these dossiers and getting permission to use these dossiers for the purpose of testing.

Secondly, looking at the questionnaire, I am not sure if the outcome will help in improving the Guidance due to the general nature of the questions. One solution could be to focus the testing of the usefulness of Guidance  so that it follows the methodology of the risk assessment, as described in the five steps in Annex III of the Cartagena Protocol. For me the Guidance is meant to ‘guide’ the risk assessors in a proper way through the five steps of the risk assessment in order to come to balanced outcome of the risk assessment. In addition, testers could be asked to also reflect on the more general preambular material, before the five steps. All this should probably be explained in the instructions for the testing, that should still be drafted.
In this way the outcome of the questionnaire could be interpretated in terms of  how far the steps in the Guidance (the Roadmap and specific cases of LMO’s) are actually doing what they should do. In addition this would help the testers to give useful suggestions to improve the Guidance to meet its goal.

Kind regards,
Boet Glandorf
GMO Office, The Netherlands
posted on 2013-04-15 12:29 UTC by Ms. Boet Glandorf, Netherlands
This is a reply to 4889 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#4936]
Dear All
I like to join the thanks to the Secreteriat and Hans for their efforts in developing and moderating this session.
I want to share what is my understanding especially of an actual case and the summary submitted to the BCH.
The summary submitted to the BCH is a short description of the case, a reference, but to do the testing a whole application dossier of this case should be used. Otherwise it cannot be concluded if based on the Guidance with its different parts this dossier has an adequate structure, enough and informative data and can be handled easily (or not). I also see the problem of having access to a complete application dossier when not being involved in the case.  But with only a summary the recommeded testing for practicality, usefulness and utility cannot be conducted. I hope this is the right understanding. Otherwise I would like to ask for further clarification.
best regards
Beatrix
posted on 2013-04-16 14:29 UTC by Beatrix Tappeser, Federal Agency for Nature Conservation
This is a reply to 4740 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#4970]
Dear Participants, 

Many thanks for your comments and valuable feedback on the draft concept note and questionnaire for the testing of the Guidance. We would like to take this opportunity to provide further clarification on a few issues that were raised.

The first issue emerging as a challenge in this process is access to documentation that provides sufficient technical and scientific data for the testing. The views shared to date suggest that the testing of the Guidance should be based on real application dossiers. However, it is also recognized that Parties who have not received their own LMO application dossiers may not have access to such data.

The Secretariat recognizes the value of testing the Guidance on the basis of application dossiers with extensive data sets for the release of LMOs into the environment. However, the Secretariat does not have access to any such dossiers that could be made available for the testing of the Guidance. We therefore take this opportunity to request participants to make available to the Secretariat, one or more dossiers with the extensive data sets for the release of LMO(s) into the environment for the purposes of the testing of the Guidance by Parties, other Governments and relevant organisations. As indicated earlier, the Secretariat intends to launch the tools for testing the Guidance in June 2013, it would therefore be appreciated if the application dossier(s) could be made available to the Secretariat as soon as possible but no later than 15 May 2013.

Furthermore, we would like to revise point (e) in the concept note to read as follows:

“(e)  The technical and scientific data of actual cases of risk assessment used in the testing may originate from various sources, including application dossiers, summaries of notifications, as well as previous and ongoing risk assessment processes;”

A second issue we would like to clarify is about the number of submissions from each country. As you may recall, the COP-MOP has requested Parties, other Governments and relevant organisations to test the Guidance and share their experience. Each entity identified by the COP-MOP (Parties, other Governments and relevant organisations) may therefore have to coordinate, as appropriate, among their experts to produce a single submission that best reflects its consolidated views.

We hope this helps in moving the process forward and look forward to further feedback to this discussion.

Thank you and best regards,
Secretariat
posted on 2013-04-17 19:43 UTC by Ms. Manoela Miranda, UNEP/SCBD
This is a reply to 4817 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#4980]
Dear Hans

It is somewhat late in the proceedings that I make my first posting, many apologies.

Caro and Fran [#4817] have suggested that in the case of in the actual “case of risk assessment” an actual application dossier. I have been struggling with this concept for some time because it implies that the applicant knows what information is required to be provided for a risk assessment, while the risk assessor / regulator does not. The object of this exercise, the road map (in particular) and guidance was to provide steps so that the risk assessor / regulator could request the required information and then assess it. If the applicant is responsible for deciding what information and in what quantity to supply then it is likely that many jurisdictions will be buried under data. I also think that there is a risk that the content of a dossier will be dictated by the requirements of those jurisdiction whose legislation goes well beyond the boundaries of Annex III. If an actual dossier is to be used the roadmap could be used to sieve out all of the superfluous material.

Regards
Geoff Ridley, Environmental Protection Authority, New Zealand
(edited on 2013-04-18 06:19 UTC by Geoff Ridley)
posted on 2013-04-18 06:18 UTC by Dr. Geoff Ridley, Environmental Protection Authority
This is a reply to 4817 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#4981]
Dear Hans

I have been giving this matter some more thought and it strikes me that there will only be a few jurisdictions that have the data from an actual case of risk assessment. These jurisdictions will have already done a risk assessment under their own legislation so why would they conduct another assessment using the roadmap and guidance? Surely, the whole point in these cases is for such a jurisdiction to look at the road map and guidance and determine whether or not it indicates what they do is in agreement or disagreement, and in the case of disagreement whether it is a lack in the roadmap and guidance or a lack in their own risk assessment.

In the case of jurisdictions that have not conducted a risk assessment then a hypothetical example should be chosen and the roadmap and guidance used to determine what information would be required and whether this is in accord with Annex III.

I believe that the requirement to use an actual case is going to result in a lack of participation.

Regards
Geoff Ridley, Environmental Protection Authority, New Zealand
posted on 2013-04-18 06:39 UTC by Dr. Geoff Ridley, Environmental Protection Authority
This is a reply to 4889 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#4982]
Dear Hans

I have some real reservations as to the value of the questionnaire. In particular I have concerns on the satisfaction scale which is weighted towards a positive outcome. For instance question 7 is a 5 point scale in which four are satisfied and only one is caste in a negative way (i.e. Unsatisfied). In some cultures there is a real reluctance to give a critical answer so such participants will always have some degree of satisfaction in this survey even when they are not. It would be far better to have a one to five scale where one is extremely unsatisfied and five is extremely satisfied. The question is also leading as it asks “how satisfied you are. The question needs to be neutral e.g. On a scale of one to five where one is very impractical and five is very practical rank the guidance for practicality.

While useful it is still insufficient for analysis and there needs to be a text field to indicate why they made the decision.

Regards
Geoff Ridley, Environmental Protection Authority, New Zealand
posted on 2013-04-18 06:54 UTC by Dr. Geoff Ridley, Environmental Protection Authority
This is a reply to 4936 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#4983]
Dear All,

I totally agree with Beatrix Tappeser and Boet Glandorf. The only way to check the Guidance is using a whole application dossier. A summary does not contain all the information you need to assess the LMO, and then it could be impossible to analyse the usefulness of the Guidance.
On the other hand, if, according to the concept note, the testing may be conducted by experts who are actively involved in risk assessment, I understand that it is not a big trouble the access to an appropriate dossier. Also, if it must be conducted by the Parties, and, as the moderator said, the summary could be submitted to the BCH at the same time as the result of the testing, I neither see a problem for accomplish that requirement.
Regarding the comment from Francisca and Carolina about the case of parties that haven´t yet received dossiers dealing with a special section of the Guidance, I hope that among all the parties it will be possible to cover all the sections. Anyway it would be a huge work for a same party to check all the sections, and, in my understanding, out of proportion.

Kind regards,

Victoria Colombo
posted on 2013-04-18 08:05 UTC by Dra Victoria Colombo Rodríguez, Spain
This is a reply to 4982 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#4984]
I would also like to thank the Secretariat and Hans for their efforts in developing and moderating this session.

A standard way of asking these sorts of questions is how they are already written in the questionnaire, ie “How satisfied are you with regards to the practicality of this section of the Guidance?”, rather than as Geoff Ridley has suggested.  However, if you wanted a more neutral set of tick-boxes, as Geoff Ridley requests, the usual way of doing this, using a 5-point scale, would be:

Very unsatisfied
Unsatisfied
Neutral – neither satisfied nor unsatisfied
Satisfied
Very satisfied

Geoff Ridley's suggestion that questions would be better phrased like this: “On a scale of one to five where one is very impractical and five is very practical rank the guidance for practicality” is a different type of question and requires a quite different method of responding to a questionnaire.  This method requires a type of ruler or line to be  placed at the end of the question, with 1 at one end of the ruler and a 5 at the other end.  The responder then puts a mark on the ruler or line.  This method often gives responses that are not whole numbers, and the person analysing the responses needs to get a ruler out and measure how far along the line the responder has put their mark.  For example, a responder may give a response that is 2.8 or 3.2.  However, now, the data are no longer categorical (as the secretary wants), but continuous, and need to be analysed in a different way.  If you take the data and “round it up” or “round it down” to the nearest whole number, you are not accurately representing what the responder was trying to say.  I would therefore suggest that it is best to stick with the existing tick boxes but to change the responses next to the boxes as suggested above.
posted on 2013-04-18 08:58 UTC by Dr Judy Carman, Institute of Health and Environmental Research
This is a reply to 4984 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#4985]
I also have some specific suggestions as to how the questionnaire could be improved.

In the section: “Information about the testing process”, question 5, part II, there is a list of certain types of LMOs or Traits.  What if someone uses the Guidance for something that is not specifically on that list?  On this forum, we have already discussed such possibilities, as such as LMO fish, LMO viruses, LMOs created through use of dsRNA techniques, LMOs with altered nutrition, LMOs designed to produce pharmaceutical or industrial products; or for integrating human health into environmental risk assessments.  I therefore suggest another tick-box be included in that section with “Other, (please specify): <Text entry>” next to it.

Similarly, in Part II: Specific types of LMOs or Traits, the list should be expanded to include “Other, (please specify): <Text entry>” at the end.

I also have some additional questions that I would like to have included in the questionnaire to assist the BCH to determine how useful the Testing is.  These questions are below.  Please note that the numbering on these questions does not map to specific question numbers on the BCH Questionnaire, but are stand-alone identifiers in this posting.

1.  Please describe any intended receiving environments of the LMO.

2.  Is this an application for a small-scale environmental release or a large-scale release?

3.  If the Guidance is written in another language, please provide a copy of the translation that was used. (This is added so that the quality of the translation can be investigated to see whether it is a cause of any good or bad results for the testing)

4.  Did you use all or only part of the Guidance in your testing?

If only part of the Guidance was used, which part(s) were used?

5.  With respect to Annex III section 8(a) of the Cartagena Protocol on Biosafety:
Did the Guidance and its associated bibliography help you to identify any novel genotypic and phenotypic characteristics associated with the living modified organism that may have adverse effects on biological diversity in likely potential receiving environments?  Please note that this includes risks to human health.

6.  With respect to Annex III section 8(b) of the Cartagena Protocol on Biosafety:
Did the Guidance and its associated bibliography assist you in evaluating the likelihood of any of these adverse effects being realised?  Please take into account the level and kind of exposure in the likely receiving environment(s).

7.  With respect to Annex III section 8(c) of the Cartagena Protocol on Biosafety:
Did the Guidance and its associated bibliography assist you in evaluating the consequences, if any, if these adverse effects are realised?

8.  With respect to Annex III section 8(d) of the Cartagena Protocol on Biosafety:
Did the Guidance and its associated bibliography assist you to estimate the overall risk posed by the LMO?  Please note that the overall risk is a combination of the likelihood of any adverse events occurring and the consequences if those adverse events did occur.

9.  With respect to Annex III section 8(e) of the Cartagena Protocol on Biosafety:
Did the Guidance and its associated bibliography assist you in coming to a recommendation as to whether or not any risks are acceptable or manageable, including, where necessary, identifying any strategies to manage those risks?

10.  With respect to Annex III section 8(f) of the Cartagena Protocol on Biosafety:
Did the Guidance and its associated bibliography assist you to identify any uncertainty about the level of risk and/or clarify how any risk may be addressed by, for example:
(1) Requesting further information about any concerns.
(2) Implementing appropriate risk management strategies
(3) Monitoring the LMO in the receiving environment(s)?

11.  Keeping to the references that were relevant to the actual case(s) used for testing the Guidance, did you find the background documents to the Guidance sufficient for your needs?

12.  Do you wish to recommend any additional documents that could be used?
If yes, what documents do you recommend?

13.  Are you aware that there is a procedure for recommending additional documents through the Biosafety Clearinghouse?

14.  Do you have any additional comments for possible improvements to the Guidance?  These comments will be included in the report for consideration by the Conference of the Parties serving as the meeting of the Parties to the Protocol at its seventh meeting.
posted on 2013-04-18 09:46 UTC by Dr Judy Carman, Institute of Health and Environmental Research
This is a reply to 4970 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#4986]
Dear all,

I would like to thank the Secretariat for their explanation on the dossiers that could be used for the testing. I have two comments. First of all I want to indicate that I will try to get hold of full dossiers in the Netherlands for testing of the Guidance, as requested by the Secretariat. However,  this will take some time  (a few weeks) since this will need some internal discussion on what dossiers to choose and what dossiers are available in English. But, we will give it a try!

Secondly, I agree with the proposed questions as suggested  by Judy Carman [4985]. Especially  questions  5 to 9, e.g. questions in which the risk assessor is asked whether the Guidance helps him or her in following the five steps in the risk assessment as described in Annex III.
These questions could be part of the questionnaire, or otherwise we think that questions like these can be included in the instructions for the testers. It would be extremely useful if the testers get instructions on how to incorporate the “actual case of risk assessment” into their testing.

Boet Glandorf
GMO Office, The Netherlands
posted on 2013-04-18 14:07 UTC by Ms. Boet Glandorf, Netherlands
This is a reply to 4970 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#4987]
Dear all,
First of all, I want to thank the Secretariat for developing and moderating the Session.

My colleagues from NCBC and I agree with the point of view of Dr. Boet Glandorf, Dr. Victoria Colombo and others about importance of using the real dossiers on risk assessment of LMOs and value of dossiers from experts who are actively involved in risk assessment.
I wand to add on my own, that this year in our country the first release of LMOs into the environment for small-scaled field trials under controlled conditions are planned. After the providing the Expert Council for risk assessment applications from candidates with information accordingly to our Legislation expertise will be conducted. We plan to test the Guidance during this expertise and make dossiers available to BCH, but unfortunately for objective reasons we have no opportunity to finish it till May, 15.

Sincerely yours,
Galina Mozgova,
NCBC Belarus.
posted on 2013-04-18 14:16 UTC by Dr. Galina Mozgova, Belarus
This is a reply to 4987 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#4988]
Dear all,

Thanks again to Hans and the Secretariat for moderating this forum and their valuable inputs throughout.

Judy Carman [#4985] presented a very thoughtful and useful set of questions to be included in the questionaire and support their inclusion.

I also join Beatrix [#4936] and others in support of the use of a dossiers for the testing.

In response to the Secretariat’s request for Parties to make available dossiers with the extensive data sets for the release of LMO(s) into the environment for the purposes of the testing of the Guidance by Parties, other Governments and relevant organisations, I will forward this request to the competent authorities here in Norway and see if we may also assist the Secretariat here. I thank Boet [#4986] for his efforts in advance as well!

Kind regards,

David Quist
posted on 2013-04-18 14:55 UTC by David Quist
This is a reply to 4986 AW: RA Forum: Development of tools to test the Guidance - A newmessage has been posted to the forum [#4989]
Dear colleagues,

in support of Boet Glandorf's offer to provide full dossiers there should be in principle the possibility to get full access to dossiers hosted by the European Food Safety Authority (EFSA) for GMO to be placed on the EU market including cultivation. The dossiers are all written in English and some of them contain mostly non-confidential data. There might even be dossiers which have no confidential data at all.

Detlef Bartsch

BVL - Berlin, Germany


-----Ursprüngliche Nachricht-----
Von: bch@cbd.int [mailto:bch@cbd.int]
Gesendet: Donnerstag, 18. April 2013 16:10
An: Prof. Dr. Detlef Bartsch
Betreff: RA Forum: Development of tools to test the Guidance - A new message has been posted to the forum

Dear Prof. Dr. Detlef Bartsch,

The following message has been posted by Dr. Boet Glandorf, Netherlands on 2013-04-18 14:07.

RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs <http://bch.cbd.int:80/onlineconferences/forum_ra/discussion.shtml?forumid=17310&threadid=4740#4986>  [#4986]

Dear all,

I would like to thank the Secretariat for their explanation on the dossiers that could be used for the testing. I have two comments. First of all I want to indicate that I will try to get hold of full dossiers in the Netherlands for testing of the Guidance, as requested by the Secretariat. However,  this will take some time  (a few weeks) since this will need some internal discussion on what dossiers to choose and what dossiers are available in English. But, we will give it a try!

Secondly, I agree with the proposed questions as suggested  by Judy Carman [4985]. Especially  questions  5 to 9, e.g. questions in which the risk assessor is asked whether the Guidance helps him or her in following the five steps in the risk assessment as described in Annex III.
These questions could be part of the questionnaire, or otherwise we think that questions like these can be included in the instructions for the testers. It would be extremely useful if the testers get instructions on how to incorporate the "actual case of risk assessment" into their testing.

Boet Glandorf
GMO Office, The Netherlands

See this post in the online forum <http://bch.cbd.int:80/onlineconferences/forum_ra/discussion.shtml?forumid=17310&threadid=4740#4986> | Reply <http://bch.cbd.int/onlineconferences/forum_ra/discussion.shtml?forumid=17310&threadid=4740&replyto=4986#4986> | Unsubscribe <http://bch.cbd.int/cms/ui/forums/unsubscribe.aspx?forumid=17310&threadid=&email=detlef.bartsch%40bvl.bund.de&validation=448156acdb144090b9f46182165febbf&returnurl=%2fonlineconferences%2fforum_ra%2fdiscussion.shtml%3fforumid%3d17310%26threadid%3d4740>

To reply to this post by email, please send your message to P4986GFAG14192@ocs.cbd.int.

FURTHER ASSISTANCE
If you have any questions, suggestions or problems with the use of this service, please contact the Secretariat of the Convention on Biological Diversity at: bch@cbd.int



2011-2020 United Nations Decade on Biodiversity <http://www.cbd.int/>
  ­­
posted on 2013-04-18 15:00 UTC by Prof. Dr. Detlef Bartsch, Germany
This is a reply to 4985 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#4990]
First, I appreciate all the work the Secretariat and Hans have done to produce this Questionnaire.  This was a very difficult task.

I believe questions 8, 10, 12 and 14 are the "guts" of this questionnaire.  I agree with those who have expressed concern about controlling bias in the questionnaire.  The challenge of rating individual "satisfaction" is its subjectivity.  We are not necessarily interested in the level of satisfaction of a tester, but rather in charting a course to make the guidance satisfactory for (welcomed by) the COP/MOP.  At a minimimum questions 7, 9, 11, and 13 should be modified to a numeric scale, and a text entry should be required to explain the rating.  In this way, the responses will be more objective and analysis would not be numeric.  I could accept deleting these questions and relying only on text responses to 8, 10, 12 and 14. 

Furthermore, I would not support expanding the list of questions beyond asking for:  (1) describe the nature of the test? (2) was the test for a commercial release or field trial/confined release? and (3) what was the specific LMO involved?  These questions should be added to the first section.

Thanks,
Tom
posted on 2013-04-18 15:01 UTC by Mr. Thomas Nickson, Consultant
This is a reply to 4988 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#4991]
With regard to the use of "real dossiers" in testing, it should be kept in mind in the process that one is not testing the risk assessment, but rather the guidance.  It is not clear to me how dossiers help in testing, but I am willing to learn from others who see this process as a means to improve the guidance. 

Tom
posted on 2013-04-18 15:05 UTC by Mr. Thomas Nickson, Consultant
This is a reply to 4970 public available dossie [#4992]
Dear Secretariat members,
the bean dossie is available for download at the site
http://www.ctnbio.gov.br/index.php/content/view/17991.html

Unfortunatly, it is in Portuguese, since this is the official language for our country processes.


Anyway it is available for anyone.


Regard


Deise


--
Deise M. F. Capalbo
Pesquisadora, Biossegurança / Análise de Risco Ambiental

Embrapa Meio Ambiente
Empresa Brasileira de Pesquisa Agropecuária (Embrapa)
Jaguariúna/SP

deise.capalbo@embrapa.br
T elefone: +55 (19) 3311-2631 | Fax: +55 (19) 3311-2640 | Skype: dmfcapalbo
http://www.cnpma.embrapa.br | twitter.com/embrapa
Confira também: http://www.facebook.com/agrosustentavel
_____________________________________________________________________
Envie seus comentários, sugestões ou reclamações para http://www.cnpma.embrapa.br/sac
"Embrapa Meio Ambiente - Certificada ISO 9001:2008"










________________________________________________________________________
____
Aviso de confidencialidade

Esta mensagem da Empresa  Brasileira
de Pesquisa  Agropecuaria  (Embrapa),
empresa publica federal  regida pel
o disposto  na Lei Federal no. 5.851,  de
7 de dezembro de 1972,  e  envia
da exclusivamente  a seu destinatario e pode
conter informacoes  confidenc
iais, protegidas  por sigilo profissional.  Sua
utilizacao desautorizada 
e ilegal e  sujeita o infrator as penas da lei. Se
voce  a recebeu indevid
amente, queira, por gentileza, reenvia-la ao emitente,
esclarecendo o equi
voco.

Confidentiality note

This message from Empresa  Brasileira d
e Pesquisa  Agropecuaria  (Embrapa), a
government company  established und
er  Brazilian law  (5.851/72), is directed
exclusively to  its addressee 
and may contain  confidential data,  protected
under  professional secrecy
  rules. Its unauthorized  use is illegal and  may
subject the transgresso
r to the law's penalties. If you are not the addressee,
please send it bac
k, elucidating the failure.
posted on 2013-04-18 16:15 UTC by Dr. Deise Maria Fontana Capalbo, Brazil
This is a reply to 4984 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#4996]
Dear Hans

I like to thank Judy Carman for comments on the questionnaire but I cannot agree that it is standard. Judy and I must move in different worlds as nearly every written and electronic survey and questionnaire that I have seen or answered in the last 20 years has asked to pick a point between two polar views rather than use check boxes with emotive titles. There is a whole science and experience behind the design of good questionnaires – designed to give good data for analysis and with apologies to the secretariat I do not think that this is one of them.

Again with apologies to Judy Carman if I have misunderstood but she seems to have created a whole new system using a ruler that I did not suggest and have never seen used anywhere! But I must agree that my rewording does make it a somewhat different question because the current question is biased and emotive and such is likely to direct a response rather than being neutral. This is hardly scientifically rigorous.

There are 5 core design principles for a question - 1. Should be easy to understand; 2. Should not be biased or leading; 3. Should provide a clear choice; 4. Should only ask one question; and 5. Fit for purpose.

If we look at question 9

1. Should be easy to understand: It is not easy to understand and definitions have had to be provided to try and make the question understandable.

2. Should not be biased or leading: As I noted in my previous posting these questions are not unbiased as the language is emotive and negative which will influence a respondent.

3. Should provide a clear choice: The relationships are not clear on the boxes. Is Satisfied the pivot point in the list so that “very satisfied” is the opposite to “somewhat satisfied” or should it be “very unsatisfied”. The wording skews the respondent to a positive response.

4. Should only ask one question: Nine asks two questions one about usefulness and one about utility but only allows one response.

5. Fit for purpose: Asking the testers of the roadmap if they satisfied does not tell us anything. They need to tell us specifically what works and what doesn’t work. This question is not fit for purpose.

Regards
Geoff Ridley, Environmental Protection Authority, New Zealand
posted on 2013-04-18 20:29 UTC by Dr. Geoff Ridley, Environmental Protection Authority
This is a reply to 4985 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#4998]
Dear Hans

Judy Carman [#4990] has suggest a raft of additional questions which need to be considered very carefully. Some will be useful, if written in an appropriate way, to establish the parameters of the test conducted by individual respondents however some go beyond the scope of this exercise and should not clutter this process.

Regards
Geoff Ridley, Environmental Protection Authority, New Zealand
posted on 2013-04-18 20:53 UTC by Dr. Geoff Ridley, Environmental Protection Authority
This is a reply to 4740 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#4999]
As it is my first time contributing to this present discussion, my thanks also to the Secretariat and to Hans for moderating.

I would like to bring to the attention of all that perhaps the most accessible and extensive set of dossiers are available on the USDA/APHIS website at http://www.aphis.usda.gov/biotechnology/petitions_table_pending.shtml.  This provides you with different types of dossiers, including some that have not been encountered by other countries.  Having said this, I would also suggest that testers should consider dossiers from more than one country, because that would correct for portions of dossiers that are submitted in order to comply with country-specific standing regulations.

I join with those who have commented that the questionnaire is biased toward respondent expressing a satisfaction with the guidance.  I also think that we are a bit off in asking the tester's satisfaction with the guidance as far as practicality, etc.  Instead we could ask instead of questions 7, 9, 11 and 13: "Was this section of the Guidance useful in practical application of Annex III?"  This would be answered with Yes or No and then space for text entry could be provided to provide reasons for the answer.  In particular, I think question 13 is superfluous, because we are applying the Guidance to existing dossiers for LMOs, which would mean that they would apply to either past or present LMOs.  Therefore, if respondents answer yes or no to the question I propose, question 13 is answered one way or another.

Finally, I also suggest that the Guidance should be tested against confined field trials and commercial release.  I suggest asking the question, "Was this section considered relevant for confined field trials, or for commercial release?"  The two choices could then be presented as tick boxes (where ticking both would be allowed) and then for answers where only one box was ticked a text entry section would ask for the reason why the testers did not find the section relevant for a particular application.

Regards to all,

Hector Quemada
posted on 2013-04-18 20:57 UTC by Mr. Hector Quemada, Western Michigan University
This is a reply to 4984 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#5007]
In response to Judy's intervention [#4984] I disagree that the a 'switch' to looking at practicality rather than satisfaction would require a different method of response, since in both cases you are looking for a judgement on the part of the person responding to the questionnaire.

But the more fundamental question is whether it is 'satisfaction' or 'practicality' that we interested in. 

I am not usre how useful it is to know about how 'satisfied' an assessor/regulator/whoever is with the guidance.  Are we not interested in the application of the guidance and whetehr or not it is 'fit-for-purpose'?  Or am I missing something?

regards
janet
posted on 2013-04-19 02:13 UTC by Janet Gough, Environmental Protection Authority
This is a reply to 4999 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#5008]
I would like to support Hector Quemada's intervention [#4999] as being practical and likely to produce useful responses.
In particular I think that we need to remember what the guidance is designed for and to ensure that the 'testing' relates to that purpose. Thus the question "Was this section of the Guidance useful in practical application of Annex III?" is critical.  I also agree strongly with the suggestion that the testing consider both confined field trials and commercial release. 

regards
janet gough
posted on 2013-04-19 02:19 UTC by Janet Gough, Environmental Protection Authority
This is a reply to 4740 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#5010]
Dear all,

First of all, I would like to thank Hans and Secretariat for opening this valuable discussion.

It would be important to develop a practical guidance for users, especially who have little experience of risk assessment.  When considering the practicality, it should be reminded the original terms of reference ordered by the COP-MOP, such that  “Develop a “roadmap”, such as a flow chart, on the necessary steps to conduct a risk assessment in accordance with Annex III to the Protocol and, for each of these steps, provide examples of relevant guidance documents”(BS-IV/11 Annex).

The roadmap should contain the minimum requirements for RA described in Annex III of the Protocol.  Reflecting the public concerns of each Party, there may be higher level of considerations (sometimes beyond Annex III of the Protocol) in the country-specific regulations.  Such specific considerations had better be supplemented by introducing examples of optional considerations in RA, rather than be included in the common procedure.

With the above understanding, I would like to propose a following type of question in order to limit the contents of the guidance and to make the guidance more practical;
Q. Are there any contents which can be deleted or changed to just optional examples with respect to the consistency with the Annex III of the Protocol?

Kind regards,

Isao Tojo
Ministry of Agriculture, Forestry and Ficheries, Japan
posted on 2013-04-19 07:15 UTC by ISAO TOJO, Ministry of Agriculture, Forestry and Fisheries
This is a reply to 4996 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#5011]
I agree with you, Geoff Ridley, that you and I must move in different worlds.  Answering a questionnaire does not mean that you have answered a GOOD questionnaire, and you cannot determine how to write a good questionnaire by answering the occasional questionnaire (good or bad) over the last 20 years.  To write a good questionnaire requires training and experience in the art and science of doing so, followed by practice.  Not only have I received substantial training in how to write a good questionnaire, I have further been employed by various State and Federal government agencies to write questionnaires and analyse the results.  These questionnaires were much longer and more involved that the questionnaire under discussion here.  I have further trained others in how to write questionnaires.

I have seen some very good and some truly awful questionnaires in my time, and I first want to congratulate the Secretariat in writing this one.  Even with my training and experience, I could find very little that concerned me.  This is rare! My comments therefore mostly involved suggestions for more questions. The fact that there have been so few comments about the actual questionnaire by forum participants, and that the discussion in this forum involves quibbling over a few details, mostly between Geoff and I, indicates that it is a good questionnaire. 

Geoff, the process of “pick a point between two polar views” while using a numeric system, does indeed usually use a ruler system. I'm sorry if you have not seen this system is use – if the questionnaires you have filled-in have not used it. The process you are suggesting of using a numeric system without a ruler is awkward, indirect, difficult to read and understand, and difficult to respond to.  In other words, it is everything that is opposite to a good questionnaire. Questions should be simply and clearly put, and they are.  A question such as “how satisfied are you with regards to the practicality of this section of the Guidance” is clear and unambiguous.  It is not “emotional”. The options for replying to the question should also be clear and unambiguous.  And unbiased.  As a result, I suggested five tick boxes labelled as follows:

Very unsatisfied
Unsatisfied
Neutral – neither satisfied nor unsatisfied
Satisfied
Very satisfied

Note that the middle box is a neutral response, and there are two boxes on either side of that neutral response that mirror each other for the level of satisfaction on one side and the level of dissatisfaction on the other side.  Therefore, there is no bias that could be construed in how the responses are set-out or in the language that is used.  The response choices are also clear and fit for purpose.

The use of definitions does not mean that the question is not understandable.  English is spoken in many different countries and cultures, and defining key words in this way simply avoids any misunderstandings that may arise.  There are even quite a few ordinary, every-day English words that have quite different meanings between Australia and the US, for example. I think that the questionnaire-writers have simply been thorough by including definitions.

As regards your suggestion that question 9 could be split into two questions, one asking about usefulness and the other about utility, I agree that this is something that should be considered.  At this point, I think that it may be wise to ask the questionnaire writers about their reasons for grouping them together.   It is quite possible that they have had experience with previous questionnaires that has lead them to conclude that the two concepts need to be linked in the one question in order to get meaningful results.
posted on 2013-04-19 08:22 UTC by Dr Judy Carman, Institute of Health and Environmental Research
This is a reply to 4999 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#5012]
I disagree with Hector Quemada's suggestion (#4999) of only allowing two responses for the questions, being yes or no.  In my view, it is too prescriptive.  I do however agree with him about the provision of an open field next to the answer where the respondent can add more information about why he/she ticked a particular box.
posted on 2013-04-19 08:33 UTC by Dr Judy Carman, Institute of Health and Environmental Research
This is a reply to 5012 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#5014]
Dear partcipants,

I sent the message bellow in a previous forum (on January) and as much as I couldn´t see any of my sugestions being part of the proposed questionnaire I´m sending the questions again. Those questions were sumarized based on the questions raised by other partcipants on that forum.

Thank you.

Best regards,
Luciana P. Ambrozevicius
Ministry of Agriculture, Livestock and Food Supply / Brasil

1 - Questions to contextualize how the testing was performed and by whom:
a) Which actual case was used
b) If the testers are Parties, non Parties or organizations, with or without experience
c) If the testers are evaluators in an establish regulatory systems or academics not bound by formal rules
d) If the evaluators have done risks assessment to applications before

2 - Questions about praticality usefulness and utility of each step described in the Guidance:
a) Is the flow chart clear, uselful and relelevant in conducting a risk assessment and do you understand the logic described in the steps of the risk assessment in the Guidance?
b) Are there parts of the guidance that do not provide a clear way forward when applied to actual cases?
c) Are there parts of the guidance that are ambiguous?
d) Can you locate the necessary information in the application?
e) Is it clear in the Guidance the criteria for the quality of scientific information to be used in the risk assessment?

3 - Questions about the source of information in the Guidance:
a) Are the Background documents associated useful as presented?  
b) The Guidance adequately points to the most important and relevant sources of information? (if not identify the missing sources of information)
c) Are other guidance documents than those being tested used for risk assessment?

4 - Questions about the Guidance consistency with Annex III of the Protocol:
a) Are there parts of the guidance that go beyond the Protocol?
b) Are there parts of the guidance that are in conflict with the Protocol?

5 - Questions to evaluate if the method described in the Guidance is compatible with the way risk assessment is normally performed in his legislation system:
a) Which components of the Guidance have corresponding sections in national legislation?
b) Which relevant components of national legislation are not covered by the Guidance?

6 - Questions about the language/terms used in the Guidance:
a) Is the technical language understandable? Proposals and suggestions
b) Is the glossary useful?
c) Which terms in the glossary should have a better definition? 

7 - Do you intend to use the present Guidance for risk assessment at your country?

8 - What are the benefits of using the present Guidance?
posted on 2013-04-19 13:07 UTC by Ms. Luciana Ambrozevicius, Brazil
This is a reply to 4985 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#5016]
POSTED ON BEHALF OF PHILIP BEREANO

----

May I add a short observation.

After many years of work with social science surveys (eg, 'how satisfied are you . . ."), it is clear to me that one needs to take any result obtained as merely indicative, not definitive.  Different people (as well as culturesd) have different undertandings of such terms (which might not even be in a language which is their most comfortable).

More importantly, a large body of research indicates that people's feelings about something ARE NOT SCALAR.  Thus, the implied linearity of the response possibilities distorts the answers--and surely the "mathematical" aggregation (which is almost always done) is inherently incorrect.

And the sample of respondents is always less than 100%.  There is distortion is the self-selection of respondents.  For example, those who think the issue is not very important are more likely to ignore the survey.  This can be reflected somewhat in an attempt to express a "level of confidence" but these levels are often later ignored in the seduction of using the NUMBER which has resulted.

Thus, we would thus be going off in incorrect directions if we used the results of any such polling as a basis for policy formulation.

my best regards to all,

Phil Bereano
WEashBAC and University of Washington
***************************
Philip L. Bereano
Professor Emeritus
HCD&E Department
Sieg Hall
University of Washington
Seattle, Wash. 98195
***************************
posted on 2013-04-19 13:49 UTC by Ms. Manoela Miranda, UNEP/SCBD
This is a reply to 5010 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#5020]
I previously posted suggestions for additional questions that could be added to the questionnaire.  I am now posting suggestions for alternative ways of asking existing questions in the questionnaire that was developed by the Secretariat.  In order to prevent confusion as to which of my suggested questions are being discussed by participants, I have listed these questions in this post as Q1, Q2 etc, compared to simply 1, 2 etc in my previous post, and I have further placed this post at the end of the discussion. 

In this post, I've made an attempt to re-write certain questions in the questionnaire in order to make them simpler and more direct and to remove the “satisfied” aspect of the questions, as a direct result of comments made Janet Gough [#5007], Geoff Ridley [#4996] and Thomas Nickson [#4990].

But first, a warning.  It looks to me as if the questionnaire writers have tried very hard to make the responses as consistent as possible between questions.  Hence they have repeatedly asked about how satisfied the respondent is about an issue.  Once you start asking questions in a different way to remove the “satisfied” aspect of the question, so as to remove any emotional or “feeling”undertones, then the questions tend to become more direct and the nature of the question and the nature of the response are both changed, sometimes quite considerably.  For example, when writing responses to my suggested, re-written question 9, strictly speaking, the opposite of “useful” could be “not useful” or “useless”, but the  two options have different meanings.  It also becomes quite difficult to find suitable words to express the middle, neutral choice of the 5 choices. 

Please note that writing a questionnaire usually takes several drafts and these suggestions, if accepted, are only the first draft and they will still need some work by the people who wrote the original questionnaire, in order to get the words right.

My suggested changes are here.  These question numbers map directly to question numbers in the original questionnaire that was sent to us all:

Q7.  How easy or difficult was this section of the Guidance to use? 
Very easy
Easy
Neither easy nor difficult 
Difficult
Very difficult

Q8.  Would you like to suggest improvements to this section to make it easier to use?
Yes
No
If yes, please indicate line numbers and explain how <Text entry>

Q9.  How useful did you find this section of the Guidance?
Very useful
Useful
Neither useful nor not useful [see what I mean about how difficult it is to get the words right?]
Not very useful at all [ditto]
Completely useless [ditto]

Q10.  Would you like to suggest improvements to this section to increase its usefulness?
Yes
No
If yes, please indicate line numbers and explain how <Text entry>

Q11.  How consistent do you consider this section is to the Cartagena Protocol on Biosafety?
Very consistent
Consistent
Neither consistent nor inconsistent
Inconsistent
Very inconsistent

Q12.  Would you like to suggest improvements to this section to increase its consistency with the Protocol?
Yes
No
If yes, please indicate line numbers and explain how <Text entry>

Q13.  How well does this section acknowledge and take into account your knowledge and skills on LMOs?
Very well
Quite well
Neither well nor poorly
Poorly
Very poorly

Q14.  Would you like to suggest improvements to this section to better acknowledge and take into account your knowledge and skills on LMOs?
Yes
No
If yes, please indicate line numbers and explain how <Text entry>
posted on 2013-04-19 14:33 UTC by Dr Judy Carman, Institute of Health and Environmental Research
This is a reply to 5020 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#5021]
Dear all

Please let me also say that I agree with Beatrix Tappeser [#4936], Boet Glandorf [#4889], Victoria Colombo [#4983], Galina Mozgova [#4987] and David Quist [#4988] about the need to use real dossiers when testing the Guidance.
posted on 2013-04-19 14:48 UTC by Dr Judy Carman, Institute of Health and Environmental Research
This is a reply to 5021 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#5024]
Dear Colleagues,

Since Norway evaluates information related to submissions made to the EU, this would necessarily be the most relevant source for dossiers that may be available from the European area.

With this in mind, I wanted to point out that data related to the dossier submitted to EFSA on NK603 is publically available online at:

http://www.efsa.europa.eu/en/press/news/130114.htm

This may be a starting point for acquiring information on an actual case of RA using regulatory data submitted in support of an application.

I also reflected a bit more on the usefulness of the whole dossier in the testing phase. Is it the dossier, or the scientific information contained in the dossier that is of most value?

Of course the structure of the regulatory information submitted by an applicant is tailored to the requirements of the competent authority, and this is perhaps the limitation here. Adapting between regulatory environments is not so straightforward. However, contrary to my prior suggestion to use the whole dossier, I rather think that the scientific and other information included to support the RA conducted on the basis of a dossier would be a good starting point, rather than the including the regulatory backdrop of the dossiers themselves. For instance, I would not need the dossier to evaluate the scientific relevance or quality of data on a non-target impact study, just the actual studies itself.

In conducting the testing and using a dossier, one will likely find differnces in informational requirements, their quality, quanitity or nature between the what the dossier requires and what the Roadmap might. That is, any actual case of risk assessment may attend to issues not included in the Roadmap (as would be the case with a Norwegian impact assessment), or perhaps go beyond those specified in the local context of testing. This seems to suggest that testing the utility of the Roadmap as guidance would need to be independent of whatever regulatory backdrop was used in request the data being analyzed in the testing. It is here where the testing may be of value, to identify the informatonal needs, available or not, in order to carry out all phases of the Roadmap to complete an RA.

Kind regards,

David Quist
posted on 2013-04-19 16:17 UTC by David Quist
This is a reply to 5014 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#5036]
Dear all,

I strongly agree with Professor  Bereano’s  statement on the  formulation and especially , the interpretation of a  survey results:

“There is distortion in the self-selection of respondents.  For example, those who think the issue is not very important are more likely to ignore the survey”.

I guess the opposite is also true: those who are highly interested in the issue (for whatever reason), including having time and resources to dedicate to this particular issue of biosafety, will engage heavily and give their strong opinions.  Legitimally, their opinions will be heard, but they will distort the general  feelings, agendas or needs of the majority of parties who did not contribute their opinions. Invariably, in life, those who speak,  shape the direction of things (for better or for worse).

I agree this is a difficult 22-catch situation. If you want to be heard, you must speak up.  It may be interesting to analyze  why so many parties are abstaining from participating more actively in these discussions of the Secretariat, where there is a very participatory spirit of inclusion and non-criticism.

We have just concluded the forum on socio-economic considerations, a topic less technically dense and more accessible to non-experts than risk analysis. Not many parties participated, even if  they have strong positions.  From a developing country perspective with overworked regulators with few resources, I respectfully offer possibly reasons why member countries/ regulators/experts  may not participate in the testing of the guidelines (even if a perfect questionnaire is developed, which I highly doubt we can do):

1. Time limitations - regulators are either too busy and/or too overwhelmed with other pressing issues in their everyday jobs  (often related to the protection of  the environment)

2. Apathy or mistrust – regulators don’t trust the transparency or possible hidden agendas  of the whole process, so they don’t want to invest their time on this. Some may sense, that some manipulation of the parties may take place (example: non-national experts writing statements on behalf of party- members). This may be especially apparent when flawless English text is posted by non-native speakers.  

3. Confusion – people don’t really understands what exactly is expected of them (I place myself in this category). The issues being discussed are technically dense and not clear.

4. Intimidation – people may have an opinion, but it is very intimidating to say something if you fear you might be “wrong”, or you are unable to express yourself correctly in a foreign language, in front of an audience of “experts” who may criticize your opinions  and the way you expressed them wrongly in a foreign language

5. Censorship in communication. It has been brought to the attention of some, that censorship is indeed taking place at the level of some governments, where only certain people in the Biosafety Committees are allowed to voice their opinions, which maybe more personal in nature, or reflecting those of a foreign  organization that backs them. 

6. Lack of commitment. This may be the most prevalent? People simply don’t care enough about certain issues and if it is not in their job description to spend time on something extra, like testing the guidance, they won’t.  

There may be many more reason, or none of the ones in the above list may be correct, but before going any further and investing more time and effort, it may be useful to stop for a minute and think who will participate or benefit from this very hard and time consuming  exercise of testing the guide. Maybe the answer is that it is essential to do so,  and all countries in the world will benefit. I just thought I would ask the question…


Best regards and happy weekend,

Maria Mercedes Roca
posted on 2013-04-19 17:12 UTC by Dr. Maria Mercedes Roca, CIBIOGEM, Mexico
This is a reply to 5024 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#5037]
Dear all,

I strongly agree with Professor  Bereano’s  statement on the  formulation and especially , the interpretation of a  survey results:

“There is distortion in the self-selection of respondents.  For example, those who think the issue is not very important are more likely to ignore the survey”.

I guess the opposite is also true: those who are highly interested in the issue (for whatever reason), including having time and resources to dedicate to this particular issue of biosafety, will engage heavily and give their strong opinions.  Legitimally, their opinions will be heard, but they will distort the general  feelings, agendas or needs of the majority of parties who did not contribute their opinions. Invariably, in life, those who speak shape the direction of things (for better or for worse).

I agree this is a difficult 22-catch situation. If you want to be heard, you must speak up.  It may be interesting to analyze  why so many parties are abstaining from participating more actively in these discussions of the Secretariat, where there is a very participatory spirit of inclusion and non-criticism.

We have just concluded the forum on socio-economic considerations, a topic less technically dense and more accessible to non-experts than risk analysis. Not many parties participated, even if  they have strong positions.  From a developing country perspective with overworked regulators with few resources, I respectfully offer some  possibly reasons why member countries/ regulators/experts  may not participate in the testing of the guidelines (even if a perfect questionnaire is developed, which I highly doubt we can do) and why this may distort the general views on biosafety:

1. Time limitations - regulators are either too busy and/or too overwhelmed with other pressing issues in their everyday jobs  (often related to the protection of  the environment)

2. Apathy or mistrust – regulators don’t trust the transparency or possible hidden agendas  of the whole process, so they don’t want to invest their time on this. Some may sense, that some manipulation of the parties may take place (example: non-national experts writing statements on behalf of party- members). This may be especially apparent when flawless English text is posted by non-native speakers.  

3. Confusion – people don’t really understands what exactly is expected of them (I place myself in this category). The issues being discussed are technically dense and not clear.

4. Intimidation – people may have an opinion, but it is very intimidating to say something if you fear you might be “wrong”, or you are unable to express yourself correctly in a foreign language, in front of an audience of “experts” who may criticize your opinions

5. Censorship in communication. It has been brought to the attention of some, that censorship is indeed taking place at the level of some governments, where only certain people in the Biosafety Committees are allowed to voice their opinions, which maybe more personal in nature, or reflecting those of a foreign  organization that backs them. 

6. Lack of commitment. This may be the most prevalent? People simply don’t care enough about certain issues and if it is not in their job description to spend time on something extra, like testing the guidance, they won’t.  

There may be many more reason, or none of the ones in the above list may be correct, but before going any further, it may be useful to stop for a minute and think who will participate or benefit from this very hard and time consuming  exercise of testing the guide.

Maybe the answer is that it is essential to do so and all countries in the world will benefit. I just thought I would ask the question…

Best regards and a happy weekend to all,

Maria Mercedes Roca
posted on 2013-04-19 17:18 UTC by Dr. Maria Mercedes Roca, CIBIOGEM, Mexico
This is a reply to 5024 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#5039]
Dear all,

I appreciate the efforts of the Secretariat and Hans to providing us with a draft of concept note and the questionnaire for testing of guidance on RA. These papers are very helpful to guiding our discussion on  this topic.

I am fully agree with the point of view of David Quist  [#5024] to focus the testing procedure first to the scientific component and nucleus of the guidance rather  then to the  regulatory backdrop of dossiers. In my understanding the quality of scientific data, completeness and multi-factorial analysis may give us the proof of relevancy and usefulness of the guidance.

I would like to thank participants to the forum for referring us to the available dossiers and the RA experiences that might be very helpful. The EFSA sources and experience is very much interesting for countries in European region, bringing in mind the similarity of variability of agricultural crop variety appropriate to the climate conditions and traditional consumption.

It would also be helpful to bring to the discussion the scientific component and data of the overviews, surveys and the consensus document on risk assessment available and provided by the OECD and other international organizations.

I would like to disagree with the proposed questions listed in the intervention of Luciana Ambrozevicius from Brazil [#5014], I consider this is discriminatory approach in relation to the countries with less experience in RA. I believe the success of the testing of guidance should be focused to the scientific values with multi- disciplinary approach, and bring together the scientific experience,  knowledge and opinion of  divers countries.

I also in agreement with Beatrix Tappeser [#4936] considering that the full dossiers should be involve in testing phase, not just summary that usually available via BCH and other websites.

My country would be willing to take part in the testing of guidance at a workshop mode and based to a real a full dossier, ex of corn or soybean. The specificity of the receiving environment as part of the ra procedural examination should be one of the attentive point to consideration and reflect the geographical and natural specific of the country.

kind regards,

Angela
(edited on 2013-04-19 18:14 UTC by Angela Lozan)
posted on 2013-04-19 17:24 UTC by Angela Lozan
This is a reply to 5010 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#5044]
I support broadening the idea that Dr. Tojo shared with us in #5010.  Without unduly biasing people, testers should be aware that they can make recommendations to delete sections, sentences or words. 

We have several options:  (1) The question he shared with us should be included as a broader statement to allow deletions or changes, or (2) Qs 8, 10, 12 and 14 could be expanded to include a second sentence "Suggested improvements could be specific deletions, changes or additions to text."  Again, a text explanation must be included.

Thanks,
Tom
posted on 2013-04-19 18:45 UTC by Mr. Thomas Nickson, Consultant
This is a reply to 5024 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#5048]
Dear Hans Dear Members of the Forum

Greetings to everyone and my thanks to Hans for moderating.

In my view we should aim to use a dossier with a comprehensive set of studies. The USDA source recommended by Hector [#4999] seems to only have the "notice" and "petition". While the petition holds scientific information, it appears to be a summary of data from the developer and lacks the underlying studies. The Guidance is best tested when a user can engage it on the science.

Therefore David Quist’s suggestion [#5024] to get a full dossier from either Norway or the EU is welcome, along with Deise’s application from Brasil [#4992].

I am pleased that we have among us those with expert backgrounds in developing surveys and questionnaires. I wish to thank Angela [#5039] for again offering her country/region for conducting testing.

Kind wishes
Jack
posted on 2013-04-19 21:14 UTC by Mr. Jack Heinemann, University of Canterbury
This is a reply to 5016 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#5049]
I would like to thank Philip for his intervention [#5016] and say that I am in strong agreement with his comments about social science surveys.

I would like to add that a number of us who are contributing to this forum have had extensive experience in the design (testing) and administration of social science surveys and also have experience and expertise in the evaluation of such surveys.   Philip's points about them are very pertinent and need to be remembered when we are looking how the results might be applied.

regards
janet gough
posted on 2013-04-19 21:48 UTC by Janet Gough, Environmental Protection Authority
This is a reply to 5048 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#5051]
Hello all

I agree with David Quist  [#5024], Angela Lozan [#5039] and Jack Heinemann [#5048] about the need to to focus the testing procedure on the scientific aspects of a dossier rather than on the regulatory backdrop.  I particularly agree with Jack Heinemann [#5048] when he says that “While the petition holds scientific information, it appears to be a summary of data from the developer and lacks the underlying studies. The Guidance is best tested when a user can engage it on the science.”  I therefore support the view that the Guidance should be tested against a full scientific dossier. 

I am also grateful to David Quist's suggestion [#5024] to get a full dossier from either Norway or the EU, as well as Deise's application from Brasil [#4992], and to Angela [#5039] for offering her country to conduct testing. 

As this forum comes to a close, please let me thank everyone involved.
posted on 2013-04-19 23:59 UTC by Dr Judy Carman, Institute of Health and Environmental Research
This is a reply to 5048 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#5052]
In response to Jack Heinemann's statement that the information on the USDA website appear to be only summaries.  Please go to the table entitled "Determinations of Regulatory Status".  Click on any of the green buttons in the column titled "Petition and Preliminary Assessment", and click on the "Petition" link, and download the entire document.  When you look at the contents of the document, I think you will find the detailed data, not merely summaries.

Regards,

Hector
posted on 2013-04-20 01:53 UTC by Mr. Hector Quemada, Western Michigan University
This is a reply to 5052 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#5054]
Thank you Hector. I did download the petition. The petition document looks to me like the application that the developer writes for food safety too. But usually this document does not have all the data from the underlying studies, but highlights of those studies. My apologies if this is not the case with USDA environmental risk assessments, but if that is the full extent of the data available it seems far too little for an environmental risk assessment. A food safety assessment can have thousands of pages of underlying research on top of a document such as the petition document.
posted on 2013-04-20 05:04 UTC by Mr. Jack Heinemann, University of Canterbury
This is a reply to 5054 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#5055]
Thanks, Jack.  However, I am a bit confused by your comments, in particular those that relate to food safety data.  Since we are talking about the Guidance Document to Annex III of the Protocol, food safety data are not relevant, since neither the Guidance nor Annex III can be used as guidance for food safety.  As for the inadequacy of the data for environmental purposes, I think that the experience of regulators around the world is that in fact these are adequate, since they have been used to support such decisions not only in the US but in other regulatory jurisdictions as well.

Regards,

Hector
posted on 2013-04-20 05:37 UTC by Mr. Hector Quemada, Western Michigan University
This is a reply to 4740 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#5056]
Dear Hans, Dear all

I have to admit that I am completely comfortable with the approach proposed by the secretariat. I see some merits in considering some of the additional questions in Judy’s intervention [#4985]. I also join Beatrix [#4936] David [#4988] and others in supporting the use of real dossiers for the testing.

Warm regards,
O.A.Elkawy
posted on 2013-04-20 13:05 UTC by Mr. Ossama Abdelkawy, Syrian Arab Republic
This is a reply to 5056 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#5057]
Dear All,

Thanks for all for all the messages that inspire my thinking.

Satisfaction is the first feeling that we expect when we read the guidance. It is the core thing as it means practicality, usefulness, and scientific foundation. Practicality can suggest usefulness but not necessarily reflect scientific foundation.

Practicality is important not only for the guidance but also for the questionnaires. We should not mislead the readers when they read the questionnaire. For instance, as a non-native speaker, it is difficult for me to tell the difference between 'usefulness' and 'utility'. I agree to Judy [#5011] that we should keep the questionnaire clear and simple.

In addition, scientific foundation should be one important factor for a questionnaire as well as for a guidance document. We also need to focus the testing procedure on the scientific aspects of a dossier. During the test we need enough participation of scientific experts. With the support from Secretariat I think that China maybe interested in participating in the testing with a face to face workshop held in China together with international scientists if the output would be considered by the Secretariat toward scientifically sound guidance docs.

Best wishes

Wei
posted on 2013-04-20 15:24 UTC by Mr. Wei Wei, China
This is a reply to 5057 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#5076]
Many thanks to Hans and the Secretariat for all their sterling efforts.

I offer the following suggestions regarding the proposed questionnaire based on the following principles;
• keep it simple and clear (in support of interventions by Wei Wei, Judy Carman and others)
• make only minimal changes to the suggested questionnaire (to stay close to a common understanding)
• make questions that assist the Secretariat to interpret the responses and assist with making any future improvements to the Guidance

Question 4. Add <Other source>
In light of the different suggestions that may provide useful actual cases that are readily available but not on the BCH, I would suggest adding this potential option. For example, Australia has a complete list of full risk assessments for both field trials and commercial releases at http://www.ogtr.gov.au/internet/ogtr/publishing.nsf/Content/ir-1

Question 7. Is this section of the Guidance practical?
In line with suggestions from Janet Gough [#5007], Geoff Ridley [#4996], Tom Nickson [#4990] and Judy Carman [#5020] it would seem to be simpler and clearer to ask questions of practicality directly, rather than for degrees of satisfaction. It is possible to believe that the Guidance practical, but be dissatisfied because you would like it to be even more practical! Furthermore, responses to the question of practicality (especially if testers provide an explanation of their answer) are more likely to assist with improving the Guidance rather than responses to the question of satisfaction.

Question 9. Is this section of the Guidance useful and have utility?

Question 11. Is this section of the Guidance consistent between this section and the Cartagena Protocol on Biosafety?

Question 13. Does this section takes into account past and present experiences with LMOs?

However, there would then be a need for a slightly modified set of tick boxes that are simple, common and balanced for responses to questions 7, 9, 11 and 13, such as,
No
Somewhat
Yes
Not sure (this is a standard choice in many social surveys to accommodate uncertainty, a notion that is strongly emphasized in the Guidance)

Question 15 should be repeated as subsets to Questions 7, 9 11, 13, such as,
7a. Please explain your answer. (These answers would be crucial to make improvements to the Guidance.)

Best regards
Paul
posted on 2013-04-21 21:36 UTC by Paul Keese, Australia
This is a reply to 5076 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#5084]
We have previously had a discussion about the nature of the materials to test.  I support the view that a full data set should be used in testing the Guidance.  I thank Paul Kees [#5076] for offering "a complete list of full risk assessments for both field trials and commercial releases at http://www.ogtr.gov.au/internet/ogtr/publishing.nsf/Content/ir-1" from Australia.  However, I note that this website contains notices of decisions and summaries only, and no full data sets.
posted on 2013-04-22 01:14 UTC by Dr Judy Carman, Institute of Health and Environmental Research
This is a reply to 5084 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#5085]
Dear Hans, Dear Participants

Thank you Hans for agreeing to moderate the forum and thank you members of the forum for such an interesting discussion.

I would start by addressing some of previous comments that I agree/disagree with.

I would like to disagree with the proposed questions listed in the intervention of Luciana Ambrozevicius from Brazil [#5014]. I also believe this could end up being a discriminatory approach in relation to the countries with less experience in RA.

I agree with David Quist  [#5024], Angela Lozan [#5039] and Jack Heinemann [#5048] about the need to to focus the testing procedure on the scientific aspects of a dossier rather than on the regulatory backdrop as said by Judy.

The suggestions of available full dossiers from David [#5024] and Deise's application from Brasil [#4992] are very welcomed.

As said by Ossama is his previous post, I am very comfortable with the approach proposed by the secretariat. And I do think that the additional questions proposed by Judy’s [#4985] could be very useful.

Best,

Sarah Agapito
posted on 2013-04-22 01:29 UTC by Dr. Sarah Agapito-Tenfen, GenØk Centre for Biosafety
This is a reply to 4740 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#5090]
POSTED ON BEHALF OF PATRICIA GADALETA

-----

Greetings to all the participants and thank you for giving us the opportunity to contribute to this interesting debate and also thanks Dr Bergmans for chairing the forum.

We found out that although some regulators read English in Spanish-speaking countries as Argentina we have found some difficulties in distinguishing between the terms ¨usefulness¨ and ¨utility¨ in the answer 9. Sometime we use these words as synonyms in general speaking and we understand their meaning by the context of the sentence in English, so we suggest splitting the question in two, one for utility and one for usefulness. Utility in general refers to satisfaction; usefulness has to do with functionality. While something that is very useful will probably have high utility, something with high satisfaction for someone might not really be useful. So we honestly believe that the questionnaire or these specific words in the questionnaire printed in italics should be translated to the language of the tester participants to obtain reliable outcomes.

We have also found that asking ¨how satisfied are you…¨, doesn’t give a real idea of individual level of usefulness, practicability, consistency, etc about the guideline, it rather seems to refer to feelings of testers related with this items. A 5-point scale as Dr Judy Carman has suggested is nice but we found numeric scale is more independent of feelings and a text could help to explain the meaning of the rating. We also found some additional questions propose by Judy Carman [#4985] [#5020] and Luciana Ambrozevicius [#5014] very appropriate to be included in the questionnaire.

According to the part Testing process, point c in the Concept Note where it says ¨testing may be conducted by individuals or as a group initiative (e.g. workshops)¨, we understand that more than one individual regulator can do the test as long as he/she belongs to a Party, other governments or organization, so several questionnaires of different regulators can be submitted.  Besides, in regard to workshops, we believe that in the questionnaire ¨Information about the testing process¨, particularly in the point 1. ¨Type of submission¨ could be a fourth tick box with the word ¨workshop¨ to select the option and a text entry section to indicate the reference of the workshop.

And finally we agree with some of you that the only way to check the Guidance is using full scientific application dossier for each particular section of the guidance (i.e. release of LMOs with stacked genes or traits, LM crops with tolerance to abiotic stress, LM mosquitoes, LM trees).

Kind regards,

Patricia Gadaleta
Biotechnology Directorate
Ministry of Agriculture, Livestock and Fisheries Argentina
posted on 2013-04-22 14:55 UTC by Ms. Manoela Miranda, UNEP/SCBD
This is a reply to 4740 RE: Opening of the discussion on the development of tools to test the Guidance on Risk Assessment of LMOs [#5091]
POSTED ON BEHALF OF PHILIP BEREANO

-----

Colleagues--

In regard to the thread of this conversation that suggests that USDA/APHIS materials might be useful, please let me remind you of a point I have made many times during the negotiation of the Protocol and subsequently. The USDA does not do any sort of assessment of LMOs that remotely begins to match the requirements that we have developed under the Protocol. Between the willful non-regulation of LMOs by the US government and its capitulation to ever stringent notions of Commercial Business information (CBI), information which cannot be revealed to the public, the dossiers are next to useless.

I know that US representatives routinely say that "we have assesed the LMOs and find no problems" but that is (1) because the government relies on on industry data, generating none of its own; (2) because the government relies on industry self-assessments, doing none of its own; or (3) tells falsehoods under the well-exhibited ideological conditions of group-think, despite info to the contrary (eg, the failure to do public follow up studies to ANY of the research indicating possible problems with LMOs).

I urge the participants on this discussion to commit to following what Hillary Clinton called an "evidence-based reality" instead of repeating cliches grounded in wishful thinking.

Phil Bereano
***************************
Philip L. Bereano
Professor Emeritus
HCD&E Department
Sieg Hall
University of Washington
Seattle, Wash. 98195
***************************
posted on 2013-04-22 14:56 UTC by Ms. Manoela Miranda, UNEP/SCBD