Of course all rights have limits, including rights of religious practice.
The common saying which describes this principle is as follows: "the right to swing your fist ends at my face." I.e. you have a right to swing your fist, by my right to security in my person trumps your right to swing your fist. Much constitutional law involves conflicts of interest that mimic this example.
However, the the only rights, other than the plaintiffs right to religious practice, in question in Oregon v. Smith, are the rights of the government to deny someone unemployment benefits. This interest is not substantial enough to deny someone a fundamental right that is clearly enshrined in the text of our constitution. In other words, it doesn't meet the fist-to-face test. The governments security is not threatened by our plaintiffs religious practice (nor is anyone's security threatened by this practice).
Furthermore, the very first amendment to our Constitution states: "Congress shall make no law respecting an establishment of religion, or prohibiting the free exercise thereof." A basic reading of the constitution makes it clear that Congress cannot make a law that "prohibits the free exercise" of religion.
However it is clear that the government policy at question in Oregon v. Smith did indeed "prohibit the free exercise" of the plaintiffs' religious practice. This is conceded in the majority opinion. The plaintiff's were denied unemployment benefits because they were fired "for cause" after a drug test revealed traces of peyote. The plaintiff's are/were active members of a native american religion and they consumed (smoked) the peyote as part of a religious ritual. The law designating firing on this basis as "for cause" was designed to restrict the practice of smoking peyote, which is in, the case of the plaintiff's, a religious practice. Therefore, the first amendment is implicated.
Furthermore, The plaintiffs weren't harming anyone via their religious practice. (They were smoking peyote, in a controlled environment similar to a sweat lodge.) And there is never any mention of the possibility that they're religious practice might have affected the rights of other people in the majority opinion. Therefore, the plaintiffs' constitutionally guaranteed right to religious practice should trump the governments interest in denying them money.
However, Oregon vs. Smith, chose to ignore the plain meaning of the Constitution. The majority in the Supreme Court instead chose to proscribe limits to the right of religious practice that subject religious practice to basically any whim of government policy.
Oregon v. Smith establishes that the government only needs a reasonable rationale in order to restrict religious practice. Furthermore, the "reasonable rationale" need not be very clearly fleshed-out - as it never very clear what Oregon's rationale is in restricting the plaintiff's religious practice. Therefore, SCOTUS essentially ruled that it would default to government opinion, everytime government policy "accidentally" restricts the practice of someone's religion.
Having a "reasonable" government interest as the only test to determine the constitutionality of laws that restrict religious practice is not enough protection for a foundational constitutional right. In this case (and any other case where it is solely a governments interest that conflicts with religious practice) the religious practice should win out. I think this is facially obvious and the only plausible way to interpret the meaning of the "free exercise" clause of the first amendment.