I used to work with a woman who, in her previous job, taught phys ed (including swimming) in a disadvantaged inner city elementary school. I remember her telling me that black children generally had a lot more trouble in her swimming classes. She thought there were actual physical differences that made it harder for the black children to learn to swim. That there were racial differences in bone density, leg length, size of large motor muscles, amount of fat vs muscle, etc, which caused black children to weigh more than white children of similar size and shape, which made it harder for them to float.
I have no idea whether there was any factual basis for this, whether they weighed the children or did body fat testing or anything. I felt uncomfortable with this topic of conversation and I didn't encourage her to take it any farther. But I do remember her having interesting things to say about specific people's individual body proportions, weight distributions, gaits, the way that they moved, etc, that explained why certain sports came more easily to one individual than to another. So I suppose if there were something to observe there, she'd be the one to spot it.
I've had some large weight fluctuations in my lifetime and I have noticed that it was surprisingly easy to swim when I was overweight, compared to how it felt when I've been thinner. But if this was enough to account for the difference in children's swimming abilities, I'd think it would also mean men were less likely to be able to swim, than women are, and I've never noticed anything like that. OTOH any increased aptitude due to relative fat/muscle ratios are probably more than masked by the social pressures on overweight women to never appear in public in a swimsuit.