This pilot clinical trial studies the side effects of anti-ESO (cancer/test antigen) murine T-cell receptor (mTCR)-transduced autologous peripheral blood lymphocytes and combination chemotherapy with cyclophosphamide and fludarabine phosphate in treating patients with cancer that has spread to other places in the body (metastatic) and expresses the gene NY-ESO-1. Donor white blood cells that are treated in the laboratory with anti-cluster of differentiation (CD)3 may help treat metastatic cancer. Drugs used in chemotherapy, such as cyclophosphamide and fludarabine phosphate, work in different ways to stop the growth of cancer cells, either by killing the cells, by stopping them from dividing, or by stopping them from spreading. Giving more than one drug (combination chemotherapy) may kill more cancer cells. Aldesleukin may stimulate white blood cells, including natural killer cells, to kill metastatic cancer cells. Giving anti-ESO (cancer/test antigen) mTCR-transduced autologous peripheral blood lymphocytes together with combination chemotherapy and aldesleukin may kill more cancer cells.