org.egothor.duplicity.file

Class JaccardCoeficientsFile

java.lang.Object

org.egothor.duplicity.file.DuplicityCheckingFile

org.egothor.duplicity.file.JaccardCoeficientsFile

public class JaccardCoeficientsFile
extends DuplicityCheckingFile

Represents the file of Jaccard coeficients, or more exactly represents an aggregated "similar unit pairs" files of type AllSimilarUnitPairsFile. Just like the "similar unit pairs" files this file contains only pairs where a < b. The file contains instances of JaccardCoeficient class. That means it contains triples {first, second, num}, where first, second are identificators of units on which we check duplicity (can be document, paragraph or sentence) and num is the number of occurences of the pair in underlying AllSimilarUnitPairsFile, which is the number of permutations according which the two units are similar. The file is sorted - the main criteria is first field, in case of tie second field.

The file should be used as follows.

1. First it should be created by merging CommonSimilarUnitPairsFile using AllSimilarUnitPairsFile.mergeToJaccardCoeficientsFile(java.util.ArrayList<org.egothor.duplicity.file.CommonSimilarUnitPairsFile>) or read from filesystem using constructor(String location).
2. It can be merged with another files of Jaccard coeficients by a call to merge(org.egothor.duplicity.file.JaccardCoeficientsFile, org.egothor.duplicity.file.JaccardCoeficientsFile) method.

Author:

Kateřina Dufková

Constructor Summary

Constructors

Constructor and Description
JaccardCoeficientsFile(String location)

Method Summary

Modifier and Type	Method and Description
String	dump() Dumps the file with its content to String.
Map<TextUnitID,JaccardCoeficient>	filterRelevantForDocument(DocumentUnitID doc) Filter from the file only the entries relevant for given document.
String	getFilename() Returns the filename corresponding to this file.
boolean	hasTheSameContent(DuplicityCheckingFile file) Checks if two files has the same content.
Map<DocumentUnitID,Double>	markDuplicates(List<DocumentData> docs)
void	merge(JaccardCoeficientsFile jcf1, JaccardCoeficientsFile jcf2) Merges files externally on filesystem.
void	remove(Set<DocumentUnitID> toRemove) Removes all occurences of documents given in the set from the file.

Methods inherited from class org.egothor.duplicity.file.DuplicityCheckingFile

delete, getLocation, getOut, hasTheSameContent, toString

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

JaccardCoeficientsFile

public JaccardCoeficientsFile(String location)
                       throws IOException,
                              DuplicityCheckingException

Parameters:

location -

Throws:

IOException

DuplicityCheckingException

Method Detail

merge

public void merge(JaccardCoeficientsFile jcf1,
                  JaccardCoeficientsFile jcf2)
           throws IOException

Merges files externally on filesystem. Only convenience wrapper for mergeAll(org.egothor.duplicity.file.JaccardCoeficientsFile, java.util.ArrayList<org.egothor.duplicity.file.JaccardCoeficientsFile>).

Parameters:

jcf1 - a file to be merged into this

jcf2 - a file to be merged into this

Throws:

IOException - if temporary file could not be created

See Also:

mergeAll(org.egothor.duplicity.file.JaccardCoeficientsFile, java.util.ArrayList<org.egothor.duplicity.file.JaccardCoeficientsFile>)

markDuplicates

public Map<DocumentUnitID,Double> markDuplicates(List<DocumentData> docs)
                                          throws IOException,
                                                 DuplicityCheckingException

Parameters:

docs -

Returns:

Throws:

IOException

DuplicityCheckingException

getFilename

public String getFilename()

Returns the filename corresponding to this file. The location field MUST be already set. The filename is created in directory given in location field and is in form Constants.JACCARD_COEFICIENTS_FILE_NAME.

Specified by:

getFilename in class DuplicityCheckingFile

dump

public String dump()

Dumps the file with its content to String.

Returns:

String representation of the file with its content

remove

public void remove(Set<DocumentUnitID> toRemove)
            throws IOException

Removes all occurences of documents given in the set from the file.

Parameters:

toRemove - set of document ids to remove

Throws:

IOException

filterRelevantForDocument

public Map<TextUnitID,JaccardCoeficient> filterRelevantForDocument(DocumentUnitID doc)
                                                            throws IOException

Filter from the file only the entries relevant for given document. Relevant are considered the entries about the document, but only the one with the highest Jaccard coeficient for each text unit. Only entries for Jaccard coeficient above threshold value are returned (see Constants.SIMILARITY_RELEVANT_TRESHOLD).

Parameters:

doc - document for which the relevant entries are requested

Returns:

map containing the relevant entries

Throws:

IOException - on error while reading filesystem file

hasTheSameContent

public boolean hasTheSameContent(DuplicityCheckingFile file)

Checks if two files has the same content.

Parameters:

file - the second file to be tested

Returns:

true, if the file contents are the same