Given a sequence of source code, the task is to predict the correct type for a particular variable, parameter, or function. Type prediction is an important task for software developement, especially in dynamically-typed environments, which can benefit from stronger type checking while maintaining the advantages of dynamic typing. We treat the task as sequence tagging task, similar to Named-Entity Recognition (NER) in NLP.

The dataset originates from the MSR '22 paper ManyTypes4TypeScript: A Comprehensive TypeScript Dataset for Sequence-Based Type Inference. The dataset is available for download on Zenodo.

1. Download dataset either using wget per below or by clicking on the link. Unzip and run the preprocess script with chosen parameters like type vocabulary size. To use the default type vocabulary size of 50k types, train#.jsonl, test.jsonl, valid.jsonl are already available in the main directory.

cd CodeXGLUE/Code-Code/TypePrediction-TypeScript/dataset
wget https://zenodo.org/record/6387001/files/ManyTypes4TypeScript.tar.gz?download=1 -O ManyTypes4TypeScript.tar.gz
tar -xvzf ManyTypes4TypeScript.tar.gz

2. Install required packages. Use

pip install -r requirements.txt

3. Preprocess dataset Use the process_datafiles.py in dataset folder.

python process_datafiles.py -v <vocab-size> 

A vocabulary of 50,000 was used in ManyTypes4TypeScript paper.

After preprocessing dataset there should be five jsonl files. train0.jsonl, train1.jsonl, train2.jsonl, test.jsonl, valid.jsonl. Train is split to accomodate Git LFS.

Each line represents a file. The data fields are the same among all splits.

We provide a script to evaluate predictions for this task, and report accuracy score. It can be found at evaluator/evaluator.py. The train.py in the code directory also has a built in evaluator and reports scores before generating a list of predictions.

python evaluator/evaluator.py -a evaluator/gold_labels.txt -p evaluator/example-codebert-predictions.txt

{'Acc': 0.6280417975625515}

A prediction file that has predictions in the form of

index	type

See example-codebert-predictions.txt.

We also provide a pipeline that finetunes CodeBERT on this task; see train.py. For models with torch weights, the model state_dict can be loaded traditionally with torch.load()

The train.py file will train and eval by default. The validation dataset can be replaced in the evalulation. We provide a pipeline for finetuning huggingface models like CodeBERT; this can be expanded to any huggingface pretrained encoding model.

CUDA_VISIBLE_DEVICES=0 accelerate launch train.py --output_dir type-model --train_batch_size=36 --eval_batch_size=16

CUDA_VISIBLE_DEVICES=0 accelerate launch train.py --output_dir type-model --do_train=False --eval_batch_size=16

python evaluator/evaluator.py -a evaluator/gold_labels.txt -p evaluator/example-codebert-predictions.txt

{'Acc': 0.6280417975625515}

The results from the paper are listed below. Top 100 indicates the performance across the top 100 most frequently occuring types.

Please cite the original paper.

@article{jesse2022manytypes4typescript,
  title={ManyTypes4TypeScript: A Comprehensive TypeScript Dataset for Sequence-Based Type Inference},
  author={Jesse, Kevin and Devanbu, Premkumar T},
  year={2022}
}