I had a corpus luteum cyst on my ovary with both of my pregnancies. My ob explained it's actually a good sign in early pregnancy because that little cyst produces the progesterone needed to support the placenta until around 12-14 weeks. By the time my dating scan came around it had shrunk on its own.
I did feel the occasional tug or ache on that side, and they kept an eye on it at my scans, but it never affected baby. The only time it can become an issue is if it ruptures or twists, which causes sudden severe pain and needs medical attention. Otherwise, most corpus luteal cysts resolve by the second trimester. If you're worried or the pain increases, give your caregiver a call, but chances are it will disappear quietly as your pregnancy progresses.