CLIR Scores

This is a task that calculates the probability of a query word given a foreign word P(q|f).
The probability measures the similarity of two words semantically.
To get the probability, we could use cosine similarity or take the dot product of two word vectors.
P(q|f) = exp(lambdacosine(q, f))/sum(exp(lambdacosine(q, f')))
in which f' takes values from the vocab of foreign doc.
lambda is a scaling factor that decides the shape of distribution of the softmax function.

How to run

1. Ensure to download the word embeddings before running the scripts.
   This can be done by running:

curl -Lo wiki.multi.ar.vec https://dl.fbaipublicfiles.com/arrival/vectors/wiki.multi.ar.vec

curl -Lo wiki.multi.en.vec https://dl.fbaipublicfiles.com/arrival/vectors/wiki.multi.en.vec

The embedding files should be saved to the embeddings folder.

2. Ensure all packages in requirements.txt are installed. This can be done by running:

pip install -r requirements.txt

3. To export the scores to files, run:

python3 scores.py --query_file [query_file] --foreign_file [foreign_file] --contextualized_only

Following flags help specify file paths and model hyper-parameters

--query_file specifies the input file containing query sentences/docs, required

--foreign_file specifies the target file containing sentences/docs, required

--query_embedding_file specifies the file containing static embeddings in query language, required when using static embeddings

--foreign_embedding_file specifies the file containing static embeddings in foreign language, required when using stsatic embeddings

--static_only outputs scores using only static embeddings, store true

--contextualized_only outputs scores using only contextualized embeddings, store true

For instance, if you only want to use contextualized embeddings, run:

python3 scores.py --query_file ../data/queries --foreign_file ../data/foreign_docs --contextualized_only

if you only want to use static embeddings, run:

python3 scores.py \
--query_file ../data/queries \
--foreign_file ../data/foreign_docs \
--query_embedding_file ../embeddings/wiki.multi.en.vec \
--foreign_embedding_file ../embeddings/wiki.multi.ar.vec \
--static_only

if you want to output scores using both embeddings, run:

python3 scores.py --query_file ../data/queries \
--foreign_file ../data/foreign_docs \
--query_embedding_file ../embeddings/wiki.multi.en.vec \
--foreign_embedding_file ../embeddings/wiki.multi.ar.vec 

Output

The scores are stored in a pytorch tensor before they are saved to a .pt file generated by pytorch.
The shape of the tensor is (number_of_query_sents * number_of_target_sents * query_sent_max_len * target_sent_max_len).

Example:

Sentences in the query file:
North Korea launched a series of short-range missiles in March.
vaccine

Sentences in the target file:
hello, world!
hello
Vaccines typically require years of research and testing before reaching the clinic.

In the above example, number_of_query_sents = 2
number_of_target_sents = 3
query_sent_max_len = 11
target_sent_max_len = 13

In order to process the data in batch, we pad all the sentences to the length of the longest sentence in the file. Hence, query_sent_max_len is 11, which is the tokens number of the first sentence in the query file.

The shape of output tensor is 2 X 3 X 11 X 13 in the above exmple.