WASHINGTON (CNN) -- Supreme Court nominee Samuel Alito said Thursday that there was no conflict of interest over his role in a 2003 ruling involving a financial giant where he had large amounts of money invested.In April 2003, Alito and two other judges upheld a lower court's dismissal of a New Jersey woman's lawsuit against the Vanguard Group Inc. Alito later recused himself when the plaintiff, in her appeal, questioned his involvement. The 3rd Circuit later reaffirmed the initial ruling.In his response to the chairman of the Senate Judiciary Committee, Sen. Arlen Specter, R-Pennsylvania, Alito wrote he had been "unduly restrictive" when he pledged in his 1990 appellate court nomination hearing to skip cases involving the mutual fund company Vanguard and that he had no conflict of interest in hearing the 2003 matter."To the best of my knowledge, I have not ruled on a case for which I had a legal or ethical obligation to recuse myself during my 15 years on the federal bench," he wrote.Specter said his committee consulted with two experts who found nothing improper about Alito's decision to hear the case as a federal appellate judge in New Jersey. In a letter to Alito, Specter had urged the nominee "to make a full public response" on the issue."I know this issue will be responded to when you complete your questionnaire," Specter wrote in his letter to the nominee. "But I think it is not advisable to wait until that time, which would allow columnists, radio/TV talk shows and your adversaries to speculate on this issue to the detriment of your nomination."Possible problem?Sen. Edward Kennedy, a member of the Judiciary Committee, urged Alito on Tuesday to explain his decision to hear the case against the firm while he had investments ranging between $455,000 and $1 million with the company at the time.The Massachusetts Democrat and Sen. Russ Feingold, another Judiciary Committee member, noted that Alito told senators before his 1990 confirmation as a federal judge that he would recuse himself from any cases involving the firm. Feingold, D-Wisconsin, told CNN the issue raises questions about that promise, and that Alito's answers about the issue were not "consistent."Asked whether Vanguard issue could endanger Alito's nomination, Specter said, "I'm concerned that it has potential. It may."Geoffrey Hazard, a University of Pennsylvania law professor, advised Specter that Alito had no reason to step aside because the amount of money involved in the case was relatively small."The claim against Vanguard was in the thousands of dollars, whereas the Vanguard investment management companies are worth millions," Hazard wrote to the Judiciary Committee. "Hence, there was no practical possibility that the claim could create risk to Judge Alito's mutual fund investment."Feingold said he doesn't think the issue will be enough to scuttle the nomination, but he wants more details. He said Alito's explanation -- that a computer system that warns judges about upcoming cases that would require their recusal failed to alert him -- is "not adequate."In his letter to Specter, Alito explained his participation in the case as "an oversight." He offered no details, but said he voluntarily stepped aside "once my participation was called into question."Bush named Alito as his pick to replace retiring Supreme Court Justice Sandra Day O'Connor on October 31. Alito is his third pick for that seat: The first, John Roberts, was renominated for chief justice after the death of William Rehnquist in September, and the second, White House counsel Harriet Miers, withdrew in October after sharp criticism from Bush's conservative allies.When Bush announced Alito's nomination, he said, "I'm confident that the United States Senate will be impressed by Judge Alito's distinguished record, his measured judicial temperament, and his tremendous personal integrity."Hazard wrote that Alito's 1990 pledge was incautious because it was so sweeping, and his stance in the Vanguard case should not raise questions about his ethics."On the contrary, when the situation was called to his attention, he recused himself even though he was not, in my opinion, required to do so," Hazard wrote.The unusual call came the same day President Bush formally sent Alito's nomination to the Senate. CNN's Ted Barrett contributed to this report.