\documentclass{article}
\usepackage[section]{algorithm}
%\usepackage[section]{algorithm}
\usepackage[noend]{algorithmic}
\usepackage[english]{babel}
\usepackage[utf8]{inputenc}
\usepackage[margin=1in]{geometry}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{graphicx}
\usepackage[colorinlistoftodos]{todonotes}

\def\myboxA#1{\vbox{\hrule height0.8pt\hbox{\vrule width0.8pt\kern1pt
      \vbox{\kern1pt#1\kern1pt}\kern1pt\vrule width0.8pt}\hrule height0.8pt}}

\def\myboxB#1{\vbox{\hrule height0.8pt\hbox{\vrule width0.8pt\kern2pt
      \vbox{\kern2pt#1\kern2pt}\kern2pt\vrule width0.8pt}\hrule height0.8pt}}
\def\titlebox#1{\myboxA{\myboxB{\vbox{\hsize \textwidth \noindent #1}}}}
\def\AlgTitle#1#2{\centerline{\titlebox{%
CS6200 Information Retreival 
 \hfill #1}}}

\def \lecturetitle {IR Evaluation} 
 
\title{\lecturetitle}
\date{ \today}
\author{Sanjiv Kumar, Jesse Anderton, Virgil Pavlu}



\begin{document}
\AlgTitle{\lecturetitle}{}
\bigskip
\vspace{15ex}
\begin{centering}
\textsf{\huge{\lecturetitle}} \\
\vspace{3ex}
% \textsf{based on a document by Andrew Ng}
\begin {center} \today \end{center}
\end{centering}
\vspace{5ex}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{IR Ranking, Search Engine Output}
\subsection{comparing search engines}
Web search engines have their ancestors in the information retrieval (IR) systems developed during the last fifty years. IR methods include (among others) the Boolean search methods, the vector space methods, the probabilistic methods, and the clustering methods [BelCroft87]. All these methods aim at finding the relevant documents for a given query.

One of the primary distinctions made in the evaluation of search engines is
between effectiveness and efficiency.Effectiveness, loosely speaking, measures the
ability of the search engine to đnd the right information, and efficiency measures
how quickly this is done. For a given query, and a speciđc deđnition of relevance,
we can more precisely deđne effectiveness as a measure of how well the ranking
produced by the search engine corresponds to a ranking based on user relevance
judgments. Efficiency is deđned in terms of the time and space requirements for
the algorithm that produces the ranking.Carrying out this type of holistic
evaluation of effectiveness and efficiency, while important, is very difficult because of the many factors that must be controlled. For this reason, evaluation is
more typically done in tightly deđned experimental settings and this is the type
of evaluation we focus on here.

To measure ad hoc information retrieval effectiveness in the standard way,
we need a test collection consisting of three things:
\begin{enumerate}
\item A document collection
\item A test suite of information needs, expressible as queries
\item A set of relevance judgments, standardly a binary assessment of either
relevant or non-relevant for each query-document pair.
\end{enumerate}

Given these ingredients, how is system effectiveness measured? The two
most frequent and basic measures for information retrieval effectiveness are
precision(the number of relevant retrieved documents divided by the number of retrieved documents) and recall(the number of relevant retrieved documents divided by the number of relevant documents). One main use is in the TREC (Text retrieval conference, http://trec.nist.gov), where many research groups get their system tested against a common database of documents.



* We dont focus on these, but they matter in practice:\\
	* Measure: speed\\
	* Measure: user interface\\
	
* we focus on raking/retrieval performance, by analogy to accuracy for Machine Learning

* Document relevant grades

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{ Set measures}
* Confusion matrices: TP, FP, TN, FN

* accuracy

* precision

* recall

* F1


\section{Ranking Measures}
* precision, recall @k

* relevant ranks

* Average Precision

* R-precision

* Reciprocal rank

\subsection{ROC and Precision-recall curves}


\subsection{nDCG}
* gains - transform grade in usefulness/benefit. What do grades mean? Essentially a benefit model

* discounts - transforms ranks into utility. How much gains still matter as we go down the list? Essentially a user model

* DCG = dot product between gains and discounts

* nDCG = DCG normalized




\section{Test Collections}
* why we ned them

* how do we create them

* QREL files

* utility of datasets

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Significance tests}

* why we need them

* popular tests

\section{Manual Assessment}
* create your own QREL

* assessment disagreemnts, fatigue

* experts vs users vs random people

\subsection{Crowdsourcing}
* cost vs benefit

* noise

* quality assurance 

\section{User Studies}

* users vs metrics

* selecting users

* IRB

* types of studies

* types of measurements
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


\end{document}


