
151 Wis.2d 419 (1989)
444 N.W.2d 738
STATE of Wisconsin EX REL. Ricky JONES, Petitioner-Appellant,
v.
Richard FRANKLIN, Superintendent, Kettle Moraine Correctional Institution, Respondent.
No. 88-1995.
Court of Appeals of Wisconsin.
Submitted on briefs May 8, 1989.
Decided June 15, 1989.
*420 For the petitioner-appellant the cause was submitted on the brief of Ricky Jones, pro se, of Winnebago.
For the respondent the cause was submitted on the brief of Donald J. Hanaway, attorney general, and E. Gordon Young, assistant attorney general.
Before Gartzke, P.J., Eich and Sundby, JJ.
EICH, J.
Ricky Jones appeals from an order affirming a decision of the Kettle Moraine Correctional Institution Adjustment Committee finding him guilty of violating several prison disciplinary rules.
*421 Two conduct reports are involved. The dispositive issue with respect to the first report is whether the committee's failure to hold a hearing on the allegations within twenty-one days of the issuance of the report, as required by Wis. Adm. Code, sec. HSS 303.76(3) deprives the committee of competency or jurisdiction to proceed in the matter. We hold that it does, and that, given the language of the rule, we may not engage in a "harmless error" analysis of the failure to hold a timely hearing. We therefore reverse on that issue. With regard to the second report, we see no error in the manner in which the committee proceeded, or in its ultimate decision, and we affirm.
The basic facts are not in dispute. Jones received the first conduct report on February 1, 1988. It charged him with violating three prison rulesdisobeying orders, disruptive conduct and "punctuality and attendance." The report was issued after a prison officer complained that Jones disobeyed an order to end a telephone conversation in order to attend an inmate "count," and that the resulting "incident" created a mild disturbance among other inmates in the vicinity. The hearing on the report was held on February 26, 1988, and the committee found Jones guilty of disobeying orders and the "attendance" violation and imposed disciplinary sanctions.
Jones had received a second report on February 5, 1988, charging him with lying to an officer. This charge was also considered by the committee at the February 26 hearing.
Jones did not appear personally at the hearing. He appeared through an advocate who informed the committee that he was relying on the evidence in the officers' incident reports. Jones was found guilty on this report as well. On certiorari, the circuit court affirmed *422 both decisions and Jones appealed. Other facts will be referred to below.

I. THE UNTIMELY HEARING ON THE FIRST REPORT
Wisconsin Administrative Code, sec. HSS 303.76(3) provides that any inmate charged with violating prison disciplinary rules is entitled to a hearing before the committee and states that the hearing "shall be held no sooner than 2 working days or later than 21 days after the inmate receives a copy of the conduct report and hearing notice." The rule also provides that the inmate "may waive these time requirements in writing if the security director agrees...." There is no question that the committee did not convene the hearing on the February 1, 1988, conduct report until after expiration of the twenty-one day period.
The department does not dispute the time-limit violation. It argues, however, that because there has been no showing by Jones as to how he was prejudiced by the delay, the "harmless error" provisions of sec. HSS 303.87 apply, and that those provisions require us to affirm the committee's decision. Section HSS 303.87 provides as follows:
If a procedural requirement under this chapter is not adhered to by staff, the error may be deemed harmless and disregarded if it does not substantially affect the rights of the inmate. Rights are substantially affected when a variance from a requirement prejudices a fair proceeding involving an inmate.
The department contends that where, as here, there is no proof that the inmate's rights were "substantially affected" by the committee's failure to follow the administrative *423 rules, enforcing them would be a "needless formality." We disagree.
[1]
The department is bound by its own rules. State ex rel. Meeks v. Gagnon, 95 Wis.2d 115, 119, 289 N.W.2d 357, 361 (Ct. App. 1980). Those rules define the boundaries of the committee's authority, and when it abandons or exceeds them, it acts beyond its authority. Id.
[2]
As indicated, sec. HSS 303.76(3) gives the inmate the right to a hearing which "shall be held no ... later than 21 days" after receipt of the notice of charges. Such language implies that the time is intended to be a limitation, as opposed to language simply providing that the act "shall" be done within a specific time. In Interest of R.H., 147 Wis.2d 22, 26, 433 N.W.2d 16, 18 (Ct. App. 1988). In R.H., we held that a statute stating that, in actions to terminate parental rights, the trial court "shall set a date for the dispositional hearing which ... is ... no more than 30 days from the fact-finding hearing" was mandatory, and that failure to comply with its terms deprived the court of competency to entertain the action. Id. at 25, 433 N.W.2d at 17-18. The language of sec. HSS 303.76(3) requires a similar construction. Consequently, the committee lost competency to proceed when it failed to hold the hearing within twenty-one days.
As to the department's harmless error claim, whatever validity that claim may have with respect to other "procedural requirements" found in the disciplinary rules, we believe the plain language of sec. HSS 303.76(3) precludes application of the harmless error provisions of sec. HSS 303.87 to violations of the hearing time limitations. The inmate's right to a timely hearing may be waived only by the inmate. Sec. HSS *424 303.76(3). The committee has no authority to waive it any more than it has the authority to disregard it.
If we were to apply the harmless error provisions of sec. HSS 303.87 to violations of sec. HSS 303.76(3), we would be allowing the committee to disregard its own rules by refusing to hold the hearing as required and by placing the burden on the inmate to gather and present evidence of harm or prejudice in order to vindicate his or her right to the hearing. We would thus be rewriting the rule to state that not only may the inmate waive the time limits, but that: "the department may also waive the time requirements unless the inmate is able to prove that such waiver will affect his or her substantial rights." The construction sought by the department would effectively nullify the plain statement in sec. HSS 303.76(3) that compliance with the mandatory time limits may only be waived by the inmate.
Administrative rules are subject to the same rules of construction we apply to statutes. Brooks v. LIRC, 138 Wis.2d 106, 110, 405 N.W.2d 705, 706 (Ct. App. 1987). The primary source of the meaning of a rule is its language. St. ex rel. Smith v. Oak Creek, 139 Wis.2d 788, 795, 407 N.W.2d 901, 904 (1987). And where, as here, provisions of a rule or rules which appear to conflict must be construed harmoniously, giving effect to the whole. State v. Fouse, 120 Wis.2d 471, 477, 355 N.W.2d 366, 369 (Ct. App. 1984). We decline to apply the provisions of sec. HSS 303.87 to 303.76(3) so as to defeat the plain language of the latter rule restricting the ability to waive the hearing time limits to the inmate.
We conclude, therefore, that the "harmless error" provisions of sec. HSS 303.87 are inapplicable to violations of the twenty-one day time limitation for hearings in disciplinary proceedings under sec. HSS 303.76(3). We thus reverse the decision on the conduct report *425 issued to Jones on February 1, 1988, and remand to the circuit court with directions to enter an order directing the department to expunge from Jones's record all reference to the conduct report dated February 1, 1988, as well as any changes in his status that may have resulted from the issuance of that report.

II. THE FEBRUARY 5, 1989, REPORT
[3, 4]
Jones raises several challenges to the disposition of his second conduct report. In certiorari actions we are confined to the record and our review is limited to determining: (1) whether the committee kept within its jurisdiction; (2) whether it acted according to law; (3) whether its action was arbitrary, oppressive or unreasonable; and (4) whether the evidence was such that it might reasonably make the order of determination it did. State ex rel. Hoover v. Gagnon, 124 Wis.2d 135, 140, 368 N.W.2d 657, 659 (1985). The facts found by the committee are conclusive if supported by "any reasonable view" of the evidence, and we may not substitute our view of the evidence for that of the committee. Nufer v. Village Bd. of Village of Palmyra, 92 Wis.2d 289, 301, 284 N.W.2d 649, 655 (1979), quoting State ex rel. B'nai B'rith F. v. Walworth County, 59 Wis.2d 296, 303-04, 208 N.W.2d 113, 117 (1973); Van Ermen v. H&SS Department, 84 Wis.2d 57, 64, 267 N.W.2d 17, 20 (1978).
Jones claims that the committee refused to permit him to call witnesses at the hearing, including the reporting officers, and that the committee, the prison and the officers wrote, approved and ruled upon the conduct reports in retaliation for his use of the inmate complaint system in the past. As indicated, Jones did not attend the hearings despite the opportunity to do so. In *426 addition, he was advised of the due process regulations applicable to committee hearings and was asked on two separate occasions if he wanted to appear. He refused in each instance.
[5]
By his own actions, thenhis "boycott" of the hearingsJones has made it impossible to review his claims of error, for there is no evidence in the record that he ever requested to call any witnesses at the hearing or any evidence relating to his claim of retaliatory action on the part of prison officials. Given these facts, we tend to agree with the state that his claims with respect to the conduct report issued on February 5, 1989, constitute little more than "recreational litigation," and we reject them. We affirm the decision on that report in all respects.
By the Court.Order reversed in part, affirmed in part and cause remanded for further proceedings consistent with this opinion.
SUNDBY, J. (dissenting).
Prisoners retain rights under the due process clause. Wolff v. McDonnell, 418 U.S. 539, 556 (1974). By Wis. Adm. Code, ch. HSS 303 the department has established procedures for imposing punishment on an inmate which implicate the inmate's due process rights. For example, if the offense is a "major" offense, segregation, extension of the inmate's mandatory release date or loss of good time are possible punishment. Where the state threatens sanctions of such magnitude, the inmate's interest "has real substance and is sufficiently embraced within Fourteenth Amendment `liberty' to entitle him to those minimum procedures appropriate under the circumstances and required by the Due Process Clause to insure that the state-created right is not arbitrarily abrogated." Id. at 557. These minimum *427 procedures include adequate notice and a meaningful opportunity to be heard. Id. at 557-58. Inmates' due process rights may, however, be "subject to restrictions imposed by the nature of the regime to which they have been lawfully committed." Id. at 556.
Thus, the hearing procedure for major violations prescribed by sec. HSS 303.76 may be subject to restrictions imposed because of the inmates' status as prisoners in a correctional institution. I conclude, therefore, that no constitutional right of an inmate is implicated if harmless error, sec. HSS 303.87, is applied to the hearing procedure under sec. HSS 303.76.
The majority, by its failure to discuss the time limit of sec. HSS 303.76(3) in constitutional terms, recognizes that the state is free to place restrictions on the hearing procedure as long as the inmate's right to due process is not adversely affected. The majority, nevertheless, refuses to apply the harmless error provision to the committee's four-day delay in holding the hearing on Jones's first conduct report.
The majority's holding may be summarized as follows: (1) The department is bound by its own rules. (2) The use of the word "shall" in sec. HSS 303.76(3) is "mandatory," not "directory." (3) Section HSS 303.76(3) gives only the inmate the right to waive the time limit thereunder.
First, it is no answer that the department is bound by its own rules. The harmless error provision in sec. HSS 303.87 is as much a part of the department's rules as is sec. HSS 303.76(3). Section HSS 303.87 provides:
If a procedural requirement under this chapter is not adhered to by staff, the error may be deemed harmless and disregarded if it does not substantially affect the rights of the inmate. Rights are substantially *428 affected when a variance from a requirement prejudices a fair proceeding involving an inmate.
Certainly the adjustment committee is bound by sec. HSS 303.76(3). It erred when it held Jones's hearing four days after the time limit. But the committee's acts are also subject to sec. HSS 303.87. Our review should focus on whether, under sec. HSS 303.87, the committee's error was harmless or whether it affected Jones's right to present defense in a meaningful way.
Second, whether to construe "shall" as mandatory or directory is not governed by a per se rule. The question is one of intent. In the case relied on by the majority, In Interest of R.H., 147 Wis.2d 22, 433 N.W.2d 16 (Ct. App. 1988), aff'd, 150 Wis.2d 432, 441 N.W.2d 233 (1989), we were construing a statute. We sought to discover the legislative intent. Similarly, we should seek to determine the department's intent in using the word "shall" in sec. HSS 303.76(3). In view of sec. HSS 303.87 I have no trouble concluding that the department did not intend that the committee's failure to hold a major-violation hearing within the time limit will, in every case, vitiate the committee's action. If the committee fails to hold a major-violation hearing within the time prescribed in sec. HSS 303.76(3), the committee's action will be vitiated only if the inmate's right to present a defense in a meaningful way is adversely affected by the delay.
Finally, the fact that an inmate may waive the hearing time requirement is not significant. The right to waive is obviously for the benefit of the inmate. This is not a case where the expression of the one excludes the other. I am willing to concede that the department did not confer upon the committee the power to waive the time limits of sec. HSS 303.76(3). If the committee had that power, there would be no error. Section HSS 303.87 *429 assumes that there may be error but makes clear that technical, nonsubstantive errors on the part of staff may, if harmless, be disregarded. I find it significant that the example situation set forth in the Note to sec. HSS 303.87 involved staff error in complying with a time limit.[*] The majority precludes what sec. HSS 303.87 permits.
The department has recognized that too much formality in disciplinary proceedings may be counter-productive. In its Note to sec. HSS 303.64 the department states:
These objectives [maintaining order, rehabilitation], as well as the objectives of punishment and deterrence, can also be served in the less formal process. Unnecessary formality may in fact detract from some of these objectives. For example a formal adversary procedure may make it impossible to counsel an inmate about misbehavior, when counseling is more important than punishment. But increasingly, there has been pressure to rely on formal procedure. Sometimes, this detracts from fairness and other values served by the system. This is not to say that inmates should not be treated fairly. [Emphasis added.]
Wis. Adm. Code, ch. HSS 303, Appendix, Note: HSS 303.64.
*430 The Supreme Court in Wolff, 418 U.S. at 570, also noted the possible detrimental effect of giving inmate disciplinary proceedings "a more adversary cast." The Court suggested that such a result might tend to reduce the utility of disciplinary proceedings as a means to further correctional goals. Our decision today emphasizes the "adversarial cast" of inmate disciplinary procedures. Because I do not believe that this result is justified or required under the department's rules, I respectfully dissent.
NOTES
[*]  The Note reads as follows:

This rule is to make clear that technical, non-substantive errors on the part of staff in carrying out the procedures specified in this chapter, may, if harmless, be disregarded. For example, if an inmate is not served with an approved conduct report within the time specified, this would be harmless unless it affected the inmate's right to present a defense in a meaningful way. This rule conforms to present practices.
Wis. Adm. Code, ch. HSS 303, Appendix, Note: HSS 303.87.
