Murray Rothbard in 1989
Libertarianism is a system for resolving conflict. In other words, it is an ethical system. Libertarianism simply tells you not to commit crimes like theft and murder. So, it only applies to how you interact with other people, and even then only sets some bare minimum of acceptable behavior.
This invariably leads people to ridicule libertarianism because it does not give any guidance on activities that are not crimes. Should you donate to charity? Libertarianism doesn’t say. People who don’t like libertarianism phrase this as, “libertarianism does not support charity, ” which is technically true but very misleading. Many libertarians give to charity, but they do not do it because they are libertarian. They have other codes of behavior that motivate them.
These other codes are called moral systems. They help people decide what is good and what is bad. For example, Jainism says that drinking alcohol is bad. Like libertarianism, a moral code might dictate how to interact with others. On the other hand, moral codes can also deal with how to behave when you are all alone. Some moral codes are compatible with libertarianism and some are not. So even though libertarianism does not tell you what is good and bad, if your moral system says that it is good to murder, then you cannot also be a libertarian.
Some people think that libertarianism is a political theory, and so it relates only to how government should function. Yet, libertarianism can apply to any society, even ones without government. So it is important not to confuse the political efforts of libertarians for the fundamentals of libertarianism. There are certainly libertarian political theories, but they are derived from the libertarian ethical system. Libertarianism is not defined by its political conclusions.
As mentioned, libertarianism is a system for evaluating conflict. When people have a conflict, libertarianism says that the person in the wrong is whoever caused the conflict to happen. That’s why libertarianism can be defined by a single rule for human interaction: do not cause conflict. Though it is a simple idea, it has enormous implications: no war, no taxes, and no limits on lives of peaceful people.
Powerful though it is, libertarianism is not the end of the journey for those who are looking for personal codes of conduct. Everyone needs to decide for themselves if they value family, wealth, technology, art, and all the other things that can be a part of life. Anarcho-capitalism offers a good set of values to adopt and is not only compatible with libertarianism, but actually incorporates it. Yet, even ancaps still have many choices to make before arriving at a fully fleshed-out personal philosophy.
The stakes get even higher when discussing codes of conduct for entire societies. One thief can ruin a family. Government orchestrated theft can ruin a nation. One murderer can devastate a town. Government orchestrated murder can devastate an entire continent. All the more reason to continue to educate people on what libertarianism is, and why it is a wonderful ethical system to bring into your life.
Rothbard.com 
Unfortunately, the initiation of force is in the eye of the beholder. The example I like to use is when a restaurant owner places a “Whites Only” sign in the restaurant window. A black man walks in, sits at a table, and orders a cup of coffee. Question: Who initiated force? The white man placing the sign in the window or the black man coming in to buy a cup of coffee?
Typically, Libertarians claim that the black man initiated force, and they use that to justify having him forcefully removed. But denying a race the ability to participate in commerce is a threat to his livelihood and well-being if denied necessary services.
Thus we now have laws that require businesses to do business without discrimination by race.
The problem with Libertarian philosophy is that they do not understand where rights come from.
The question that libertarianism is interested in is: who caused the conflict? If I am having a dinner party and my neighbor shows up uninvited, who is responsible? Libertarianism does not tell you whether or not you should invite your neighbor to dinner. In the same way, libertarianism does not tell you whether or not you should be racist, but that does not mean it endorses racism.
The question is not about private homes. It is about restaurants and any other business accessible to the public. We all agree that, in general, your home is a private matter, unless it presents some hazard to the public (for example, it’s on a crowded street and it is on fire, where the fire might spread to your neighbors home). But we no longer agree that that a restaurant owner can be selective on the basis of a private prejudice he holds against one race or another.
All practical rights arise by agreement.
Is it unethical to open a gym that only caters to women? How about a golf league for people between the ages of 40 and 55? Special movie ticket prices for seniors? Private scholarships for minorities? Would all of these be illegal in a libertarian society?
I can’t tell you what is legal or illegal in a Libertarian society. There is an underlying question as to whether their philosophy allows them to constitute a city, state, or nation to begin with.
However, in any democratically constituted society, all practical rights arise by agreement.
And the first agreement is the one that constitutes the state or nation. The U.S. Constitution, for example, is an agreement between each citizen and every other citizen to constitute a nation and its form of government. Further formal agreements, as to the legal rights that we will respect and protect for each other, are then reach by their representatives in a legislature, in the laws they pass.
For example, to establish and protect a right to property, laws prohibiting theft are passed. These are then enforced through citizens actions (for example, calling the police when they see someone trying to steal your car), a police force that arrests the offender, courts that hold trials to determine guilt, and penalties that repair the harm, correct the offender, and protect society by confining the offender until corrected.
The people arrive at a consensus on rights by their shared moral intent. This intent may be summed up as “to achieve the best possible good and least possible harm for everyone”.
But to directly answer your question:
Since everyone loves their freedom, there is a natural inclination to avoid creating unnecessary laws. Due to the harm done to individual liberty, by denying them the right to participate in commerce solely by their race, religion, and other special categories of prejudice by which harm was inflicted upon specific groups, the right to participate in commerce is guaranteed by law.
Specific forms of injurious discrimination are outlawed by different laws. But, again, to avoid unnecessarily limiting anyone’s liberties, only specific forms of discrimination are outlawed.
So, benefits to seniors, for example, are not illegal.
The more types of discrimination that are legal, the more liberty people enjoy. It is not clear to me how you would decide whether any particular type of discrimination should or should not be legal. In a libertarian society they would all be legal.
How would it not be an initiation of force to seem armed enforcers to force any private business to serve those it doesn’t not wish to for whatever reason? It is not an initiation of force to merely have and ignorant abhorrent opinion and act on it without initiating physical force or fraud on everyone. No one has a right to tell people what to think and to try to force them to think like they would like them to think. That would be completely unethical.
The black man walked in, alone, sat down like everyone else, and, like everyone else, ordered a cup of coffee. He did not initiate force.
As to an army invading the restaurant and forcing the owner to serve the black man, that would never happen. If the restaurant owner does not comply with the law then his license to operate a restaurant will be revoked. The only way that force would escalate would be if the owner initiated it.
The type of discrimination that would be illegal would be unnecessarily harmful in a way that we would find intolerable if imposed upon us.