This phase I/II trial studies the side effects and best dose of liposomal irinotecan and rucaparib when given together with fluorouracil and leucovorin calcium and to see how well they work in treating patients with pancreatic, colorectal, gastroesophageal, or biliary cancer that has spread to other places in the body (metastatic). Drugs used in chemotherapy, such as liposomal irinotecan, fluorouracil, and leucovorin calcium, work in different ways to stop the growth of tumor cells, either by killing the cells, by stopping them from dividing, or by stopping them from spreading. PARPs are proteins that help repair DNA mutations. PARP inhibitors, such as rucaparib, can keep PARP from working, so tumor cells can't repair themselves, and they may stop growing. Giving liposomal irinotecan and rucaparib together with fluorouracil and leucovorin calcium may work better in treating patients with pancreatic, colorectal, gastroesophageal, or biliary cancer.