Should I be isolating with little one in nurseru?

My wife and I are returning to IVF after a successful birth a couple of years ago. Our little one is in nursery now and picking up bugs left right and center; for the past few months all of us have been really ill with bad colds at least once a month. As we start to go through the IVF process the risk of illness has be concerned; do we go on with life as usual and accept that IVF has to work around us and continue with normal life, or do we take our little one out of nursery and reduce contact and in effect isolate a little? We really only get a couple more shots at this so want to make it count, but at the same time it feels like isolating and being on germ patrol creates its own stress.

I'm curious what others feel who have been in a similar position. Thanks