The reason it took us 3 months to get around to doing this was because one of the things you need to have to apply is 3 months' payslips, so we waited until we had been working for three months. Despite the obvious logic of this, the man at reception berated us for not having come earlier.
After waiting for another half hour, we were passed on to a second employee whose job it was to compile our dossiers. I had been sent a list of things that we needed and we had everything apart from the things that, well, we didn't have because you only get them if you're French. None of these things turned out to be a problem. What she did ask for, however, was a form E104, which is mysteriously different from a European Health Card and somehow certifies that you have "rights" when you move to another EU country. And you can only get it in your home country.
Both my colleague and I have moved around the EU several times, including 3 times to France between us, and never been asked for these forms. We are both EU citizens employed by a French company on French contracts and paying French social security contributions. Luckily, when we looked extremely doubtful of our ability to get the form, the lady (who was actually extremely nice despite being a fonctionnaire) decided it was worth trying to apply without it, so we got our paperwork completed ready to send off.
Feeling fairly sure that nobody else in France had heard of this form either, I didn't worry about it. Then today, I was bored or curious enough to look up what it was on the HMRC website and found the following information:
The foreign authority may, exceptionally, ask you to get either a Certificate E104 which shows insurance for sickness and maternity purposes or an E301 which shows insurance for unemployment purposes.
Exceptionally, aka "in France".