You should always attend your panel hearing. For one thing, you cannot rely on the Business Agent to adequately represent you. It may be a case where you actually have a good case according to the contract, but, for whatever reason, the BA does not feel like pushing your case. I went to a panel hearing once, and my goofy BA (who was immediately hired by UPS as a supervisor when he later lost his BA election) did not adequately represent me and even kept interrupting me when I was given a minute to state my case.
No one knows the details of your case as well as you do, and you need to be present to give testimony on your behalf, just as you would in a court of law. In my experience, the Union appeared to have an inherent disadvantage at the panel hearings to start with. The company will likely be represented on the panel by executives with masters degrees while the union will likely be represented on the panel by truck drivers who perhaps do not have the advantage of having advanced degrees. (Please be assured that I am not criticizing the Union representation on the panel or truck drivers, only pointing out that there may be an unequal match regarding education and business experience.)
The panel will probably conclude that you did not consider your case very important if you don't show up. Can you imagine going asking for a variance from your local zoning board and not bothering to show up?