There is a lot of controversy about whether aseptic/clean or sterile technique should be used when accessing implanted ports, including the use of a mask, mostly because there is very little research or other evidence to indicate one best practice. With that said, below are several resources (including the Access Device Guidelines published by ONS) that give some of the current information. 
 
The ONS Access Device Guidelines book (2017) include procedures to access and deaccess implanted ports (p. 69-70). In regard to use of sterile gloves, the text states:  “Current research is insufficient to support routine use of sterile gloves during accessing and deaccessing procedures.” 

 
Additionally, there are two journal articles for additional information:  
Accessing Implanted Ports: Still a Source of Controversy 
Accessing and Deaccessing Ports: Where is the Evidence?
 

References: 

Camp-Sorrell, D., & Matey, L. (Eds.). (2017). Access device standards of practice for oncology nursing. Pittsburgh, PA: Oncology Nursing Society