The final document in the cache declassified by the Office of the Director of National Intelligence (ODNI) on Wednesday is the ninth joint compliance assessment conducted by the ODNI and the Department of Justice’s National Security Division (NSD). The FISA Amendments Act of 2008 requires the Attorney General and the DNI to assess NSA compliance with Section 702 targeting and minimization procedures and to submit their findings to the Foreign Intelligence Surveillance Court (FISC) and certain congressional committees at least once every six months. The 52-page report, dated August 2013, examines agency compliance from June 1, 2012 through November 30, 2012.

The document is interesting both because it is a more exhaustive account of compliance matters than is the internal document leaked by Edward Snowden to the Washington Post and because it is more recent. It contains a large amount of new information about the rigors of the bureaucratic process surrounding both collection and collection oversight. The Joint Assessment proceeds in three main parts.

Overview of Compliance Oversight Regime

Section 2 reviews the oversight of Section 702 implementation with respect to the NSA, CIA, and FBI. These agencies use Section 702-acquired data differently and according to their own respective minimization procedures; these differences translate into corresponding differences between the agencies’ internal compliance programs, as well as in the external oversight programs conducted by the NSD and the ODNI. The NSD and ODNI’s joint oversight of the NSA's implementation of Section 702 comprises compliance reviews on a periodic basis and in response to specific incidents. During the 2012 reporting period, the NSD and ODNI conducted three onsite reviews of the NSA and submitted them to the congressional committees with the Section 707 Report, as required by FISA, the report notes. Prior to onsite review, the NSA electronically sends the tasking record for each selector tasked during the review period to the NSD and ODNI; these are reviewed by joint oversight team members. (According to Appendix A, a selector is "a specific communications identifier or facility tasked to acquired information that is to, from, or about a target." Selectors include telephone numbers and email addresses.) The NSD prepares a report of the findings, and NSD attorneys determine whether the tasking sheets meet the NSA targeting procedure documentation standards and provide enough information for the reviewers to ascertain the basis for NSA foreignness determinations. The joint oversight team identifies those tasking sheets that did not provide sufficient information and issues questions for each selector or requests that the NSA provide the cited documentation for review. The team examines the cited documentation underlying the identified tasking sheets, along with NSA Signals Intelligence Directorate (SID) Oversight and Compliance personnel, NSA attorneys and other required NSA personnel, for the purpose of asking questions, identifying issues, clarifying ambiguous entries and providing guidance on potential improvements. The joint oversight team investigates and reports incidents of noncompliance with either NSA procedures or with the AG's acquisition guidelines, which restrict collection to communications of non-U.S. persons who are not located in the U.S. These incidents may be identified during reviews, but most are identified by NSA analysts or through NSA internal compliance. The NSA is required to report events that may not constitute compliance incidents where the report may lead to discovery of some underlying compliance incident. All identified compliance incidents are reported to the congressional committees and to the FISC. The CIA, unlike the NSA, does not engage in targeting, but it does nominate potential Section 702 targets to the NSA. Until the release of these documents, the CIA’s role in the FISA 702 process was a carefully guarded secret. And details of the nominating process in which it engages, which appear in Appendix A, are mostly redacted. The joint oversight review team conducts onsite visits at the CIA, the results of which are included in the bimonthly NSA review reports. The CIA has established mechanisms and procedures for internal compliance with Section 702. Once every two months, the NSD and ODNI conduct periodic compliance reviews of the CIA's use of minimization procedures; they conducted three such onsite reviews at the CIA between June 1, 2012 and November 31, 2012. Again, reports for these reviews were provided to the congressional committees as required by FISA, the report notes. In its bimonthly onsite reviews, the joint oversight team examines a sample of communications acquired under Section 702 that is identified as containing U.S. person information and is minimized and retained by the CIA; it ensures proper minimization; it reviews all dissemination of information acquired under Section 702 identified by the CIA as potentially containing U.S. person information; and it reviews the CIA's written justifications for all queries using U.S. person identifiers of the content of unminimized Section 702-acquired communications. The joint oversight team also investigates and reports incidents of noncompliance with the CIA minimization procedures and the acquisition guidelines, in addition to conducting regular reviews. Finally, the FBI plays three separate roles in implementing Section 702: acquiring foreign intelligence domestically, processing foreign intelligence, and receiving and minimizing unminimized Section 702-acquired communications. The NSD and ODNI conduct reviews of FBI targeting on a monthly basis; six such reviews were conducted between June 2012 and November 2012. Again, reports for these reviews were provided to the congressional committees as required by the FISA, according to the Joint Assessment. The joint oversight team reviews the "targeting checklist" completed by the FBI analysts and supervisory personnel, as well as the supporting documentation. Specific details of this review, as well as details pertaining to the review of the FBI's application of its minimization procedures, have been redacted from the report. In addition to monthly reviews, the team investigates potential incidents of noncompliance with the FBI's targeting and minimization procedures, the AG's Acquisition Guidelines or other agency procedures in which the FBI plays a role. Any identified compliance incidents are reported to the congressional committees and to the FISC.

Trends in Targeting

Section 3 of the Joint Assessment identifies trends in NSA and FBI targeting and minimization, and trends in CIA targeting. Key metrics have been redacted from this section, which makes it difficult to determine more than a few basic facts about increases and decreases. Those facts, however, are important. With respect to NSA trends, selectors under collection have increased since the last reporting period. The report notes that the average number of tasked selectors is expected to "accelerate" because the FBI has opened up its nomination process to a larger swath of its field office personnel. For example, the report has redacted the chart of total monthly numbers of newly tasked selectors since Section 702 collection began in September 2008. (In a footnote, the report defines newly tasked selector as "any selector that was added to collection under a certification.) But the report does note that the number of serialized reports issued by the NSA without U.S. person information has grown at a greater rate than the number containing such information. With respect to FBI trends, the number of reports based at least in part on Section 702-acquired U.S. person information has increased from the previous reporting period. By contrast, the number of disseminations of Section 702-acquired data containing minimized U.S. person information that were identified by the CIA has decreased since the prior reporting period.

Causes of Compliance Incidents

Section 4 summarizes the oversight team's findings as to underlying causes of compliance incidents and its assessment of the involved agency's attempts to prevent recurrences. Again, numbers have been largely redacted from this section of the Joint Assessment. For example, the total number of compliance incidents that involved noncompliance with the targeting and/or minimization procedures of any particular agency remains classified. But the report reveals that the compliance incident rate is 0.49%, described as a "low" rate that nonetheless represents an increase from compliance incident rate in the prior reporting period. Many of these incidents are trivial, involving only the NSA's failure to notify the NSD and ODNI of certain facts within the timeframe stipulated in the NSA targeting procedures---a median delay of about one business day. Eliminating these particular compliance incidents yields what the reports describes as a "better measure of substantive compliance with the applicable targeting and minimization procedures." Thus adjusted, the compliance incident rate for this reporting period drops from 0.49% to 0.20%. The report lists six separate types of compliance incidents involving the NSA's targeting or minimization procedures. Tasking issues involve "incidents where noncompliance with the targeting procedures resulted in an error in the initial tasking of the selector."Detasking issues involves" errors in the detasking of the selector." Notification delays occur when "a notification requirement contained in the targeting procedures was not satisfied." Documentation issues involve "incidents where the determination to target a selector was not properly documented." Overcollection occurs when the NSA's collection systems acquire data regarding untasked selectors while in the process of acquiring communications of properly tasked selectors. Minimization issues simply refer to problems with NSA's compliance with its own minimization procedures. The vast majority of compliance incidents during the reporting period---212 out of 338---were notification delays. There were also 48 tasking incidents, 51 detasking incidents, 2 incidents of overcollection, 15 minimization incidents and 10 documentation incidents. The report redacts a chart that depicts the compliance incident rates of previous reporting periods. Most noncompliance incidents in the reporting period did not involve U.S. persons but involved, for example, typographical errors in tasking that resulted in no collection, detasking delays with respect to facilities used by non-U.S. persons who had entered the country, or notification errors regarding similar detaskings that were not delayed. However, several incidents involved U.S. persons. These involved (1) tasking errors that led to the tasking of facilities used by U.S. persons, (2) delays in tasking facilities after the NSA determined the user of the selector was a U.S. person, and (3) unintentional querying of Section 702 repositories using a U.S. person identifier. Some undisclosed portion of the tasking incidents described in the report "involved facilities where at the time of tasking the Government knew or should have known that one of the users of the selector was a United States person." Details of one cited example of such an incident have been redacted. In another incident, the NSA failed to prevent a pending Section 702 tasking request "from being effectuated," although DHS had informed the NSA that the target of the request was "an LPR" (likely stands for “lawful permanent resident”). All Section 702-acquired data was purged in the cited incidents. The Joint Assessment concludes that these incidents are "isolated instances of insufficient due diligence." Most detasking incidents in the reporting period involved non-U.S. persons who had traveled to the U.S.; only one of the detasking delays involved a U.S. person who had been erroneously assessed and targeted as a non-U.S. person. The NSA detasked several selectors used by the individual based on the revised assessment, but did not detask one of the individual's telephone numbers for three weeks due to a "miscommunication within an NSA targeting office.” The Joint Assessment concludes that better records and additional detasking procedures could prevent such detasking delays from occurring in the future. An undisclosed number of NSA noncompliance incidents involved using U.S. person identifiers to query Section 702 repositories. The Joint Assessment notes that the FISC's October 3, 2011 and November 30,2011 orders approved modifications to NSA's modification procedures permitting the agency to use U.S. person identifiers to query telephony and non-upstream acquired electronic communications data. The query terms must be approved according to internal NSA procedures, and must be designed to yield foreign intelligence information. In each of the incidents, an NSA analyst made one of two errors: either conducting a query that the NSA had previously determined was a U.S. person identifier, or forgetting to filter out Section 702-acquired data while conducting a federated query (“a query using the same term or terms in multiple NSA databases”) using a known U.S. person identifier. The Joint Assessment notes that none of the analysts involved in the incidents were unaware that only approved U.S. person identifiers may be used to query Section 702-acquired data. Although "few" compliance incidents resulted from technical issues in the reporting period, the Joint Assessment notes that "technical issues can have larger implications" because they frequently involve multiple selectors. Thus all Section 702 agencies devote "substantial resources" toward preventing, identifying and remedying such issues. Collection equipment and other systems are tested before deployment, monitoring programs are employed to detect anomalies, joint oversight team members participate in technical briefings to better understand the impact of technical system development on information collection and processing. Some undisclosed number of the compliance incidents during the reporting period caused the NSA systems to overcollect data in violation of what had been authorized under the Section 702 certifications. Further details on particular incidents involving overcollection have been redacted. Two system errors in the reporting period led to delays in detasking facilities. In one such incident, an adjustment made in the NSA's system during transition between certifications resulted in unauthorized targeting of users in the U.S. for up to three days. The Joint Assessment begins its subsection on the NSA's human errors with the observation that human errors are the cause of many compliance incidents. Some human errors are isolated events; others represent a pattern that suggests the need for new training or procedures. As in "the last several reporting periods," the report notes, one of the most common errors in this reporting period involved selectors that were missed in the detasking process, and which were among those used by a target discovered to be in or traveling to the U.S. Subsections on the causes of CIA and FBI noncompliance are heavily redacted. The Joint Assessment describe two types of FBI noncompliance incidents: those concerning errors in the processing of requests and those involving improper targeting. In one processing incident, involving an individual located in the U.S., an FBI supervisory agent intended to reject an acquisition instead accidentally approved it, and the system fail-safe malfunctioned. Information on incidents involving FBI noncompliance with targeting procedures has been substantially redacted. Lastly, an undisclosed number of incidents involved data overproduction by an electronic communication service provider with a Section 702(h) directive. The report states that all of the incidents had different causes, but in each case agency personnel identified the overproduction through automated systems or by properly reporting within their agencies that the acquired data did not match the authorized scope of collection. The Joint Assessment concludes with the general observation “that the agencies have continued to implement the procedures and to follow the guidelines in a manner that reflects a focused and concerted effort by agency personnel to comply with the requirements of Section 702." Although reiterating that no intentional violations or circumventions of the FISA were identified, and that the number of compliance incidents is small, particularly when compared to total collection efforts, the joint oversight team emphasizes the importance of and its own role in continued noncompliance investigation and monitoring of collection activity.