Inclusion Criteria:

          -  PHASE II ONLY:

          -  Histologically or cytological confirmed diagnosis of Rb-deficient HNSCC for which no
             standard curative therapy is available

          -  Rb deficient HNSCC includes Clinical Laboratory Improvement Act (CLIA)-certified
             testing confirming one of the following:

               -  Human papillomavirus (HPV) positive as determined by any one of the following:
                  p16 immunohistochemistry (IHC), HPV ribonucleic acid (RNA) in situ hybridization
                  (ISH), RNAscope (messenger RNA [mRNA] ISH), deoxyribonucleic acid (DNA) ISH, DNA
                  polymerase chain reaction (PCR), or quantitative reverse transcriptase PCR (qRT
                  PCR)

               -  No Rb protein expression in the tumor as determined by IHC

          -  No prior treatment with an anti-PD-1 antibody (e.g., nivolumab, pembrolizumab,
             cemiplimab), as well as anti-PD L1, anti-PD-L2, anti-CTLA-4 antibody, or any other
             antibody or drug specifically targeting T-cell co-stimulation or immune checkpoint
             pathways

          -  Appropriate for single agent pembrolizumab:

               -  Front line therapy for those whose tumors express PD-L1 (Control Preference Scale
                  [CPS] >= 1)

               -  Front line therapy for those who cannot tolerate chemotherapy per the judgement
                  of the treating physician

               -  As second line or greater line of therapy

          -  PHASE I ONLY:

          -  Histologically or cytological confirmed diagnosis of an invasive solid tumor
             malignancy, for which no standard curative or life prolonging therapy is available

          -  PHASE I AND PHASE II:

          -  Eastern Cooperative Oncology Group (ECOG) performance status of =< 2

          -  Absolute neutrophil count (ANC) > 1500/mm^3 (within 22 days before the first dose of
             study drug)

          -  Platelets > 100,000/mm^3 (within 22 days before the first dose study drug)

          -  Hemoglobin (Hgb) > 9 g/dL (within 22 days before the first dose of study drug). Values
             must be obtained without need for myeloid growth factor or platelet transfusion
             support within 14 days, however, erythrocyte growth factor is allowed as per published
             American Society of Clinical Oncology (ASCO) guidelines

          -  Total bilirubin =< 1.5 x upper limit of normal (ULN) (within 22 days before the first
             dose of study drug)

          -  Serum glutamic-oxaloacetic transaminase (SGOT) (aspartate aminotransferase [AST]) and
             serum glutamate pyruvate transaminase (SGPT) (alanine aminotransferase [ALT]) < 2.5 x
             ULN (within 22 days before the first dose of study drug). AST and/or ALT may be up to
             5 X ULN if with known liver mets or total 3 x ULN with direct bilirubin =< ULN in
             patients with well-documented Gilbert syndrome

          -  Adequate renal function as defined by calculated creatinine clearance >= 30 ml/min
             (Cockcroft-Gault formula) (within 22 days before the first dose of study drug)

          -  Measurable disease according to Response Evaluation Criteria in Solid Tumors (RECIST)
             version 1.1

          -  Voluntary written consent must be given before performance of any study related
             procedure not part of standard of care, with the understanding that consent may be
             withdrawn by the patient at any time without prejudice to future medical care

          -  Patients must be able to either swallow alisertib enteric coated tablets, swallow
             alisertib oral solution formulation, or administer alisertib oral solution formulation
             via a feeding tube that terminates in the stomach

          -  Willing to provide blood and tissue for correlative research purposes

          -  Female patients who:

               -  Are postmenopausal for at least 1 year before the screening visit, OR

               -  Are surgically sterile, OR

               -  If they are of childbearing potential, agree to practice 1 highly effective
                  method of contraception and 1 additional effective (barrier) method at the same
                  time, from the time of signing the informed consent through 120 days after the
                  last dose of study drug, OR

               -  Agree to practice true abstinence, when this is in line with the preferred and
                  usual lifestyle of the subject (Periodic abstinence [e.g, calendar, ovulation,
                  symptothermal, postovulation methods], withdrawal, spermicides only, and
                  lactational amenorrhea are not acceptable methods of contraception. Female and
                  male condoms should not be used together)

          -  Male patients, even if surgically sterilized (i.e., status postvasectomy), who:

               -  Agree to practice effective barrier contraception during the entire study
                  treatment period and through 120 after the last dose of study drug, OR

               -  Agree to practice true abstinence, when this is in line with the preferred and
                  usual lifestyle of the subject (Periodic abstinence [e.g., calendar, ovulation,
                  symptothermal, postovulation methods], withdrawal, spermicides only, and
                  lactational amenorrhea are not acceptable methods of contraception. Female and
                  male condoms should not be used together)

        Exclusion Criteria:

          -  Radiation therapy to more than 25% of the bone marrow. Whole pelvic radiation is
             considered to be over 25%

          -  Prior allogeneic bone marrow or organ transplantation

          -  Known gastrointestinal (GI) disease or GI procedures that could interfere with the
             oral absorption or tolerance of alisertib. Examples include, but are not limited to
             partial gastrectomy, history of small intestine surgery, and celiac disease

          -  Inability to swallow (or use a feeding tube to administer) oral medication or
             inability or unwillingness to comply with the administration requirements related to
             alisertib

          -  Known history of uncontrolled sleep apnea syndrome and other conditions that could
             result in excessive daytime sleepiness, such as severe chronic obstructive pulmonary
             disease; requirement for supplemental oxygen

               -  Requirement for constant administration of proton pump inhibitor, H2 antagonist,
                  or pancreatic enzymes throughout the study. The intermittent use of
                  H2-antagonists and antacids (including carafate) is only allowed within these
                  guidelines:

                    -  H2 antagonists until D-1 and after the dosing of alisertib is done

                    -  Antacid formulations until 2 hours before doing and after 2 hours following
                       dosing

                    -  Proton pump inhibitor (PPI) is allowed until D-5 of first alisertib dose.
                       PPIs are prohibited throughout the study

          -  Myocardial infarction within 6 months prior to enrollment or has New York Heart
             Association (NYHA) class III or IV heart failure, uncontrolled angina, severe
             uncontrolled ventricular arrhythmias, or electrocardiographic evidence of acute
             ischemia or active conduction system abnormalities. Prior to study entry, any
             electrocardiogram (ECG) abnormality at screening has to be documented by the
             investigator as not medically relevant

          -  Female subject who is pregnant or breast-feeding. Confirmation that the subject is not
             pregnant must be established by a negative urine or serum beta-human chorionic
             gonadotropin (beta-hCG) pregnancy test result obtained during screening. Pregnancy
             testing is not required for post-menopausal or surgically sterilized women

          -  Female patient who intend to donate eggs (ova) during the course of this study or 120
             days after receiving their last dose of study drug(s)

          -  Male patients who intend to donate sperm during the course of this study or 120 days
             after receiving their last dose of study drug(s)

          -  Other severe acute or chronic medical or psychiatric condition, including uncontrolled
             diabetes, malabsorption, resection of the pancreas or upper small bowel, requirement
             for pancreatic enzymes, any condition that would modify small bowel absorption of oral
             medications, or laboratory abnormality that may increase the risk associated with
             study participation or investigational product administration or may interfere with
             the interpretation of study results and, in the judgment of the investigator, would
             make the patient inappropriate for enrollment in this study

          -  Diagnosed or treated for another invasive malignancy within 2 years of enrollment,
             with the exception of complete resection of basal cell carcinoma or squamous cell
             carcinoma of the skin, an in situ malignancy, or low-risk prostate cancer after
             curative therapy

          -  Has received any anti-cancer treatment including investigational agents within 21 days
             prior to first dose of alisertib

          -  Patients who receive gamma knife radiosurgery for brain metastases or whole brain
             radiation are eligible if gamma knife radiosurgery was completed > 2 weeks before
             treatment is started or whole brain radiation was performed > 4 weeks before treatment
             is started, and are clinically stable (not requiring steroids or anti-epileptic drugs)

          -  Known hypersensitivity to any of the excipients of alisertib enteric coated tablets or
             severe reaction to any human monoclonal antibody

          -  Patients with a prior history of clinically significant metabolic acidosis (exclusion
             only for patients receiving alisertib oral solution)

          -  Major surgery within 28 days prior to first dose of alisertib or persisting side
             effects that have not improved to National Cancer Institute - Common Terminology
             Criteria for Adverse Events (NCI-CTCAE) grade 1 or better

          -  Patients who are on (or will require) prolonged systemic corticosteroid treatment
             during the study except for replacement dosing for adrenal insufficiency

          -  Concurrent severe and/or uncontrolled medical conditions that would, in the
             investigator's judgment, contraindicate patient participation in the clinical study or
             require concomitant anti-cancer drugs (e.g. active or uncontrolled severe infection,
             chronic active hepatitis, immuno-compromised, acute or chronic pancreatitis,
             uncontrolled high blood pressure, interstitial lung disease)

          -  Patients with active, known, diagnosed or suspected autoimmune disease. Patients
             suffering from vitiligo, type I diabetes mellitus, residual hypothyroidism due to
             autoimmune thyroiditis only requiring thyroid hormone replacement therapy, or
             psoriasis not requiring systemic treatment can be enrolled

          -  Patients diagnosed with active interstitial lung disease (ILD)/pneumonitis or a
             history of ILD/pneumonitis or another condition requiring immunosuppressive doses of
             systemic medication such as systemic corticosteroids or absorbed topical
             corticosteroids (doses >= 10 mg/day prednisone or equivalent) or other
             immunosuppressive medications within 14 days of study drug administration. Inhaled or
             topical corticosteroids, adrenal replacement doses, or < 10 mg daily prednisone or
             equivalent are permitted

          -  Administration of any live vaccine within 30 days before first dose of study drug

          -  Uncontrolled infection with human immunodeficiency virus (HIV), hepatitis B virus or
             hepatitis C virus, except for: Patients with HIV who have controlled infection
             (undetectable viral load and CD4 count above 350 cells/mm3 either spontaneously or on
             a stable antiviral regimen) are permitted; Patients with hepatitis B (HepBsAg+) virus
             who have controlled infection (serum hepatitis B virus DNA PCR that is below the limit
             of detection AND receiving antiviral therapy for hepatitis B) are permitted; Patients
             who are hepatitis C virus antibody positive (HCV Ab +) who have controlled infection
             (undetectable HCV RNA by PCR either spontaneously or in response to a successful prior
             course of anti-HCV therapy) are permitted

          -  Participating in another therapeutic clinical trial