The world champion took a tight victory in Normandy after the peloton caught breakaway rider Jasper Stuyven in the final 500 metres of an uphill finish.
Cavendish finished one minute 45 seconds behind Sagan in what was always going to be a tough stage for him.
Defending champion Chris Froome and rival Nairo Quintana finished together, but Alberto Contador lost 48 seconds.
Contador crocked
Two-time winner Contador, who has geared his entire season around trying to win this year's Tour, crashed for a second time in two days as early rain made for treacherous conditions on the roads from Saint-Lo to Cherbourg-en-Cotentin.
He bruised the right side of his body in a heavy crash during Saturday's opening stage and hurt the other side on Sunday and looked in discomfort as he crossed the finish line.
"I lost more time than I had hoped to," said the 2007 and 2009 winner.
"The Tour has really started on the wrong footing. I'm not happy at all. I must see if I can make up some time in the Pyrenees and the Alps.
"I'm physically hampered. I cannot pedal as I would like as a result of the crashes. The important thing is to keep my morale, not fall apart, which is sometimes complicated."
Two-time winner Contador, who has geared his entire season around trying to win this year's Tour, crashed for a second time in two days as early rain made for treacherous conditions on the roads from Saint-Lo to Cherbourg-en-Cotentin.
He bruised the right side of his body in a heavy crash during Saturday's opening stage and hurt the other side on Sunday and looked in discomfort as he crossed the finish line.
"I lost more time than I had hoped to," said the 2007 and 2009 winner.
"The Tour has really started on the wrong footing. I'm not happy at all. I must see if I can make up some time in the Pyrenees and the Alps.
"I'm physically hampered. I cannot pedal as I would like as a result of the crashes. The important thing is to keep my morale, not fall apart, which is sometimes complicated."
