James confirmed her resignation and tweeted: "For personal and professional reasons, I will not take the election process further."
She added that she does not "have sufficient authority, nor the full support" of her MEP colleagues and party officers to implement changes she believes necessary.
"I will not be formalising my recent nomination to become the new leader of the party with the election commission," she tweeted.
The surprising move came less than three weeks after James took over from Nigel Farage as UKIP's leader after a leadership contest in September.
James, a 56-year-old businesswoman, said she would campaign for UKIP to be battle ready for the next general election, due in 2020, after she was elected to lead the third largest British party.
UKIP was formed in 1993, but still only has one MP in the House of Commons. It has 22 MEPs sitting in the European Parliament and more than 400 local councilors.
