So I was going to try out some DOS games on CDROM and looking for TEAC drivers. Reading the information on vogondrivers for the TEAC CD532E I have, it strongly recommends from daisy chaining the CDROM and hard drive. I have indeed done that because I only have one long IDE cable at the moment. I have my IDE2SD adapter on the same IDE port but I never complained about the speed.

NOTE: Industry-wide, it is not recommended to daisy chain either a CD-ROM or TAPE BACKUP drive with your hard disk drive. The performance of the hard disk drive will be degrated to the maximum speed of the other device since the data transfer rate is as fast as the transfer rate of the slowest device. The slowest device in this case will be either the CD-ROM or TAPE BACKUP drive.
It is recommended to connect your desktop CD-ROM drive as a MASTER in the next available IDE port in your system. Please reference to your system’s manual to identify the number of IDE ports available in your machine.
You can guide yourself by identifying where in the system your hard disk drive C: is physically connected to. In older style PCs, normally you will find the hard disk drive connected to an Interface Card. Check to see if there is a second (unused) 40-pin, male IDE connector on that card. On most current PCs, you will find the hard disk drive connected directly into the motherboard. Is there another 40-pin IDE connector on the motherboard?

To be honest it sounds weird that the CD-ROM would influence in this way. Is it only during reading and writing to both drives simultaneously?

My motherboard is the 5ali61 (Ultra DMA66 version).