Evaluation report on EFSA gives the wrong message

The European Food Safety Authority (EFSA) has published an ‘external evaluation’ report of the organisation carried out by international consultancy and auditing company Ernst & Young. EFSA proudly presented the conclusions of the report at a press conference in Brussels. EFSA is obliged by its founding regulation to commission an external evaluation of its achievements every six years.

EFSA summed up the main findings as follows: the report “notes the high quality of EFSA’s scientific outputs and risk communication activities”, and “highlights the Authority's culture of transparency and robust systems to ensure the impartiality of its scientific advice”. Among the recommendations are that EFSA ought to “enhance transparency in some decision-making processes”, “build better links with Member States” and “improve the clarity of its communications”. Yet CEO finds the report (price tag: 246,000 euro) – as well as EFSA's interpretation of it - highly flawed, as explained below:

The report is, to a large extent, based on a series of questionnaires and interviews that E & Y held with stakeholders [1]. E & Y wrote to stakeholders including CEO saying that their participation would help them to evaluate EFSA’s performance “in an effective and objective manner”.  The consultants refused to disclose a list of other interviewees to CEO. We received only a breakdown of respondents by different categories, as included in the final report.

This shows that EFSA’s most vocal critics had little weight (NGOs 4%, media 2%, MEPs 5%). By contrast, industry has a weight of 13% and EFSA's own Management Board and the Scientific Committee count together for 17%. The bulk of responses were from national risk managers and risk assessors.

Three MEPs participated in the questionnaire and four were interviewed. However, when CEO asked several who have been prominent in the debate around EFSA (Bart Staes (Greens), José Bové (Greens), Corinne Lepage (Liberals) and Monica Macovei (Conservatives)), they all confirmed that they had not been approached.. These four MEPs are all strong advocates of far reaching measures to make EFSA a more credible and independent institution.

Not all respondents participated in both the questionnaire and the interview. CEO’s responses on the questionnaire were not for example taken into account [2]. However, we did provide a telephone interview. The E&Y report wrongly claims that CEO did respond to the questionnaire, however we are not listed with the organisations that were interviewed. If this happened to our input, how reliable was  the processing of the other respondents’ inputs?

The basis for this report leaves ample scope for questions.

Does E & Y's report prove the quality of EFSA’s scientific output?

Essential to the scientific quality of EFSA's work is of course the nature of the data and studies used. Very often, the data comes solely from industry and is not replicated by independent laboratories. This simple fact, a fundamental flaw in the EU food safety system, is not identified by E & Y as a problem, it is just listed as a criticism “limited to NGOs”.

E & Y does not seem to have even checked the scientific quality of EFSA’s work, but simply looked at other reviews of whether certain quality procedures had been followed. E & Y make reference to a Quality External Review conducted in 2011 by an independent and external group of experts who, based on a sample of 49 EFSA scientific outputs, concluded that (only) 40% are well constructed, transparent and easily understandable. So what is wrong with the other 60%? These experts said – as quoted in the E&Y report - that the quality of EFSA’s outputs was limited by:

-       lack of clarity on databases used for the identification of reference material;

-       weak conclusions, presented without concrete support;

-       deficiencies in referencing and availability of original documentation;

-       deficiencies in synthesis and analysis;

-       limited consideration of uncertainties both at the level of parameter estimates and integrated final risk estimates;

-       inadequate summaries including missing out important criticial parameters.

Another aspect of the scientific quality not addressed by the E & Y report is the amount of time experts have available for EFSA work. EFSA panel experts are volunteers who are not paid and who only receive a reimbursement for travel and accommodation costs. These people assess scientific dossiers and help writing reports in their free time, unless they have employers that allow them to do this in work time.

Take the nutrition panel for example. EFSA says this panel has assessed 3000 industry health claims in three years. How much time then has each expert actually, realistically, given to examining each dossier? Three thousand over three years means nearly three per day, every day of the year, in their free time? Other panels perhaps face fewer dossiers, but not necessarily less work. The GMO panel for instance produced one to three scientific opinions per month in recent months, often including over 10 pages of references.

E & Y does not see this as a problem, and therefore does not suggest solutions, such as a publicly-managed fund to support independent research, financed by industry which would pay for the risk assessment (but not directly pay the risk assessor). The E & Y report only touches on industry fees in relation to the European Commission's  current proposal that EFSA could deliver extra services to industry applicants in exchange for a fee. Industry is favourable to this idea, as it would basically mean industry paying EFSA to help get them through their risk assessment!

Is EFSA independent?

On conflicts of interest, the report says that EFSA “operates in an impartial way” and the report describes EFSA as “having one of the most advanced and robust systems in place for ensuring independence.” The latter statement is literally copied-and-pasted from another consultancy’s report on EFSA’s independence published by Milieu in January 2011. A time when EFSA’s system to deal with conflicts of interest was becoming untenable, and was being swamped by criticism from all sides.

E & Y assesses EFSA’s old policy of dealing with conflicts of interest as being generally effective. With respect to the fact that EFSA has never checked any Declaration of Interest for correctness and completeness (as that was regarded as the individual expert’s responsibility), the consultancy just notes that this has “exposed the Authority to criticisms and attacks on its independence”. There is not a word in the report about the potentially grave conflicts of interest that could have gone unnoticed, with an unknown impact on food safety.

Regarding the (in)dependence of the Management Board, E & Y reports that ‘few stakeholders’ complained about the fact that more board members have links to industry than is allowed by the rules. This “perception” – as it is called by E & Y – “should however be counterbalanced by the recent case of resignation of the Chair of the Management Board that resigned to take up a position in ILSI [the International Life Sciences Institute] thus demonstrating the links with this Institute are not compatible with EFSA’s activities”. However, Banati’s resignation was her own initiative, knowing full well that these roles were incompatible, as in 2010 she had been forced to resign from an earlier ILSI position that she had failed to declare. In any case, the four management board members with industry links is already one too many according to EFSA’s own rules, even without counting Banati, who evidently had not fully abandoned her attachment to ILSI. Moreover, E & Y fails to bring up the question of why industry should be represented on the management board in the first place, since EFSA is supposed to be the ‘independent voice of science’.

The largely ‘in-crowd’ nature of the group of respondents is also reflected in comments cited in the report that EFSA could better defend its experts who are accused of conflicts of interest, “correcting the misleading image that external stakeholders may have on experts”, and that many of the accusations of conflicts with industry “have political roots and damage the image of the Authority, reducing the effectiveness of the efforts accomplished over the years”. A few pages later, E & Y this claim is repeated, saying that “the Authority should distinguish political criticisms from scientific-based ones [..] Indeed, as pointed out by many stakeholders, criticisms on EFSA’s independence are not always scientifically sound.”

It would have befitted Ernst & Young (“Quality in Everything We Do”) to come up with a single example of a ‘scientifically unsound’ criticism of EFSA’s independence – as well as to explain what exactly they mean by ‘political criticisms’. Is the way EFSA or any institution deals with conflicts of interest not a political issue?

E & Y even goes so far as to claim that further strengthened controls on experts “may reduce the number of experts compliant with EFSA’s requirements and can gradually risk to undermine the scientific quality of EFSA’s outputs”. With respect to this claim, E & Y ought seriously to be challenged to come up with any evidence backing up their assertion.

Regarding the new independence rules, EU Food Policy reported that last June, EFSA banned a leading scientist from the pesticides panel, after finding out that he had not disclosed his industry interests. While this is a positive step, this case also highlights the problem that until now - over its 10 years of existence - EFSA has not checked a single Declaration of Interest, from any panel member, for either completeness or correctness.

Corporate Europe Observatory (CEO) and Earth Open Source (EOS) have shown that several members of EFSA's recently renewed panels are still not free from conflicts of interest - based solely on the interests that were actually declared by the new panel members.

Another case of undeclared conflict of interest in the Food Packaging panel, repeatedly raised by CEO and EOS, has not led to any action from EFSA. Roland Franz failed to declare his membership of the scientific committee of the ILSI Food Packaging international symposium, while being a member of EFSA’s food packaging (CEF) panel for two years. ILSI has been shown to be an industry lobby group, funded by agri?business, biotech and pesticide companies, including MacDonald's and Monsanto. EFSA said that its new rules would ban most ILSI activities, but this has now been shown not to be the case. In fact Franz only updated his Declaration of Interest following an official complaint from CEO and EOS earlier this summer. EFSA nonetheless concluded there was no conflict of interests.

Selective research, selective reading: the wrong conclusions

The E & Y authors conclude that “Despite criticisms, no major changes in EFSA structures, governance and procedures are needed and the current situation is considered a satisfying infrastructure in itself and as compared to other European agencies and relevant international standards such as the OECD”. EFSA’s outputs “can be considered as of good quality” even though “some concerns persist”.

Executive Director Catherine Geslain-Lanéelle – who launched the report – after some apparently selective reading said in a press release: “This external evaluation underlines that EFSA operates to the highest standards – particularly with regard to the quality of our scientific opinions and risk assessment methodologies.” Even given the flaws in the evaluation, to draw this conclusion takes some imagination.

E & Y has failed to critically assess the impact of conflicts of interest that have occurred over the last 10 years in EFSA and that still have not been banned. E & Y equally fails to present a serious debate around the fact that industry science is still the backbone of EFSA's risk assessment.

This mandatory external evaluation on the functioning of EFSA was meant to input into the upcoming review of EFSA's founding regulation by the European Commission, Parliament and Council. But there is no visible sign of such ambitions in the report. Pressure on the Commission to come forward with such a proposal must come from elsewhere, for instance the upcoming final debates in the European Parliament on the approval of EFSA's 2010 budget. EFSA’s budget was not approved by the Parliament last spring. It is likely to impose substantial conditions on EFSA, if it releases the money at all.

[1] The other two data collection tools mentioned are ‘observation of 8 EFSA key meetings’ and ‘a benchmark with selected organisations’.

[2] CEO filled in the questionnaire on the condition that its report “Conflicts on the menu” would be taken into account, since the questionnaire did not give opportunity to express our opinions in a satisfactory way. We also demanded to know who the other interviewees were. Both conditions were decided by E&Y to be unacceptable, and therefore the questionnaire was regarded as not sent. Only a telephone interview was made, which was  taken into account.