
736 So.2d 514 (1999)
Linda Juawice (Stringer) SULLIVAN, Appellant,
v.
Rodney Lane STRINGER, Appellee.
No. 97-CA-01645-COA.
Court of Appeals of Mississippi.
April 20, 1999.
Rehearing Denied June 22, 1999.
R.K. Houston, Bay Springs, Attorney for Appellant.
W. Terrell Stubbs, Mendenhall, Attorney for Appellee.
BEFORE THOMAS, P.J., LEE, AND SOUTHWICK, JJ.
SOUTHWICK, J., for the Court:
¶ 1. This is an appeal from an order of the Simpson County Chancery Court changing child custody from the mother to the father. On appeal the mother argues that the chancellor erred in finding a material change in the child's circumstances justifying a change in custody. She also seeks attorney's fees. We find that the chancellor's primary reason for changing *515 custody is not a justification permitted by controlling precedent. Therefore we reverse and remand for further proceedings.

FACTS
¶ 2. On July 7, 1995, Linda Juawice Stringer Sullivan and Rodney Lane Stringer were granted a divorce on the grounds of irreconcilable differences. By mutual agreement, primary custody of the couple's three-year-old daughter, Hillarie Lane Stringer, was given to the mother, while the father was to have liberal visitation.
¶ 3. After Rodney Stringer remarried, he filed on March 31, 1997, a petition for modification. The portion of the petition relevant to this appeal requested that primary custody of Hillarie Stringer be granted to Rodney Stringer. Trial was held on October 14, 1997. The court ruled from the bench to transfer custody to Rodney Stringer effective December 25, 1997, at 5:00 p.m.
¶ 4. Sullivan filed several motions to stay both with the trial court and the supreme court, but each was denied.

DISCUSSION

Issue 1: A material change in circumstances justifying change of custody
¶ 5. Several witnesses testified as to improper conduct by the mother, Linda Sullivan. These included alcohol abuse at a number of parties, giving alcohol to minors including the minor daughter, offering drugs to minors, making romantic advances towards a teenage boy, physical abuse of the child, staying a couple of nights with one man while having a different live-in male friend, driving while intoxicated, letting the child stay up until early in the morning, threatening suicide, allowing the child to sleep in the bedroom with Linda and a male companion, and having sex with a man while the child was in the bed. The truth of much of this was contested. The chancellor did not make factfindings as to most of these events.
¶ 6. The chancellor principally relied in his opinion on the multiple sexual relationships that the mother maintained. It was the existence of those relationships, not any finding that fornication or other sexual activities occurred in view of the daughter (with one possible exception), that underlie his decision. He never found that drug abuse, propositioning a teenager, or many of the scandalous charges were true. In fact, on several occasions the chancellor stated that he believed that the mother was a fit parent. If that is to be taken as a finding, that should mean that the chancellor rejected a large portion of the charges.
¶ 7. We will accept a chancellor's fact-findings if they are supported by substantial evidence, unless there has been an abuse of discretion or manifest error. In addition, application of an erroneous legal standard to the facts may cause reversal. Cummings v. Benderman, 681 So.2d 97, 100 (Miss.1996).
¶ 8. Regarding abuse of alcohol at a number of parties, the chancellor determined that while Sullivan had on previous occasions consumed too much alcohol, it was neither illegal nor terribly significant to his decision. Neither was the court convinced that physical abuse had occurred given that there were only one or two instances of marks on the child and the parties disputed the cause.
¶ 9. Other than the events of cohabitation that will be discussed below, the chancellor made no other findings. The supreme court has held that "[a]s to issues of fact where no specific findings have been articulated by the chancellor, this Court proceeds upon the `assumption that the chancellor resolved all such fact issues in favor of the appellee.'" Anderson v. Anderson, 692 So.2d 65, 72 (Miss.1997) (quoting Love v. Barnett, 611 So.2d 205, 207 (Miss.1992)). We interpret that only to mean that to the extent other facts are needed to support the chancellor's analysis of the law, those facts would be implicitly *516 found. However, if the incorrect legal standard was applied, we will not infer the facts necessary to support other law that the chancellor did not analyze. That becomes important in this case. The inference is that in the written or spoken opinion, the judge expressed some findings and failed to articulate the other findings that he must have made to reach the result under the law that he applied. There would be no such logical inference if the judge did not even expressly consider certain law. Here, the relevant law is what has to be proven in order to change custody.
¶ 10. What the court relied upon here was a concern "that this mother has lived with three different people prior to marrying the last person." The chancellor found:
There was no question for six months [Linda Sullivan] lived with one man without the benefit of marriage with the child present, with the child sleeping in the bedroom. Likewise, with her present husband she has done that. This court finds that in and of itself would justify a material change in circumstances that would adversely affect the child. In doing this, the court with a five year old being subjected to seeing either of the parent, and in this case the mother, living with a person that she is not married [to] has got to be affected. These first years with the child at her age has got to leave impressions.
¶ 11. The chancellor was also of the apparent belief that Ms. Sullivan would not be married to that third man if not for the proceedings regarding custody of the daughter. Parts of the chancellor's opinion suggest that he rejected the other allegations. The chancellor pointed out that Ms. Sullivan had married, thus stopping what he perceived as the principal harm that she was causing her child. The chancellor noted that she was attending school in an effort to improve herself. Most importantly, he declared that "[t]his Court specifically is not finding the mother unfit. I want to make sure that that is clear on the record. The court finds that both parties are fit and proper persons." The chancellor later determined that while the father had the best parenting abilities, he also felt the mother was a good parent. Moreover, the trial court acknowledged "[t]he fact that the mother has had problems, she is correcting those problems and, in fact, evidently about back to where she needs to be." He felt "the mother [was] on the right track."
¶ 12. A finding that the mother was a fit parent is logically inconsistent with a unstated finding that she had sex with a man while her child was in the bed, offered drugs and alcohol to minors, or engaged in other of the charged conduct. Nonetheless, we are loath to imply that he found the other charges were baseless, especially when the absence of findings generally supports the lower court decision, not weakens it. We conclude that no inference can be made about the other allegations.
¶ 13. The primary factor governing the chancellor's decision was the relationships Linda Sullivan had out of marriage. It is undisputed that she carried on consecutive relationships with at least three different men over the course of approximately 18 months. Whether such conduct can constitute a material change in circumstances justifying a change of custody is our issue. This is a question of law and as such it is reviewed de novo.
¶ 14. Modification of custody should not occur unless there "has been a material change in circumstances which adversely affects the child's welfare." Cheek v. Ricker, 431 So.2d 1139, 1143 (Miss.1983). We note first that cohabitation between persons not married to each other is against the law in Mississippi. Miss.Code Ann. § 97-29-1 (Rev.1994). That this law is frequently broken has been recognized by the supreme court, but still cohabitation "remains a crime against public morals and decency...." Davis v. Davis, 643 So.2d 931, 935 (Miss.1994). *517 Commission of crimes by a custodial parent, even if they are only about sex, is properly the concern of a chancellor.
¶ 15. The chancellor's bench opinion left little doubt as to his feelings about the impact of a live-in relationship between a custodial parent and her paramour both on society as a whole and a child in particular. Several excerpts from the chancellor's opinion make this quite clear:
The court however is very concerned with the fact that this mother has lived with three different people prior to marrying the last person.
* * *
[T]his court, of course, setting forth it will not tolerate people living outside marriage while the children are present.
* * *
The mother since she has corrected what this Court feels is a major problem with society nowadays and, of course, I can't correct all of what this court considers problems with people living together outside the scope of marriage. She has, in fact, corrected that and therefore the court is further going to give her the child during the summer months....
* * *
And when I say that I think that the mother is a good parent and maybe she is growing of age and realizing what she has done as far as having these other men in the house with the children, and the Court cannot stress enough that this greatly concerns the court.
* * *
[P]rior to the filing of the petition her [the mother's] stability as far as her relationship with men after the divorce of the parties lacks something....
Any other factor relevant, the court is satisfied that I have addressed what in this case to this Court was probably the controlling factor and that is the relationships with these men outside the sanctity of marriage with the child present.
* * *
[I]f either of these parties should experience marital difficulties, they are not to have any member of the opposite sex to stay overnight in the presence of this child. If I find out that they have, I will take whatever appropriate steps are necessary.
* * *
I am not going ... to put up with anybody living and having a member of the opposite sex present that they are not married to when a child, especially a female child, is present.
¶ 16. The impact of these relationships, either by themselves or in conjunction with other factors, must rise to a level justifying a change in custody. The supreme court has made plain, despite the criminal nature of these acts, that cohabitation is relevant only to the extent it can be shown to affect the child adversely. Cheek v. Ricker, 431 So.2d 1139, 1144 (Miss.1983). In effect the supreme court has indicated that the mores of modern society are such that moral conduct by a custodial parent is difficult to make a requirement in a divorce. Absent a finding of some conduct harmful in a more specific sense than the certain knowledge of sexual relations between unmarried adults, one of whom is the custodial parent, the court has held that custody cannot be withheld on that basis.
¶ 17. Regarding custody in a divorce proceeding, the Mississippi Supreme Court has held:
The question addressed by this child custody appeal is whether the fact of adultery precludes, per se, the award of custody to the offending spouse. This *518 Court holds that the fact of adultery alone does not disqualify a parent from custodianship but that the polestar consideration in original custody determinations is the best interest and welfare of the minor child.
Carr v. Carr, 480 So.2d 1120, 1121 (Miss. 1985).
¶ 18. Building on this basic theme, the state high court has determined that "[i]f the mother's indiscretions with another man do not, in a divorce action, constitute a per se barrier to an award of child custody, it makes no sense that such a barrier should be erected in custody modification proceedings." See Cheek, 431 So.2d at 1144-45.
¶ 19. Six years after Cheek, the supreme court stated that maintaining a sexual relationship outside of marriage is not, by itself, sufficient grounds for modification of child custody. Phillips v. Phillips, 555 So.2d 698, 701 (Miss.1989).
¶ 20. What we are left with is that the existence of the relationship is insufficient, but if the relationship is coupled with other conduct that indicates the custodial parent's behavior is harmful in additional ways, custody can be changed. The chancellor was reluctant to make findings on the other allegations other than to say that "[t]here was no question for six months she lived with one man without the benefit of marriage with the child present, with the child sleeping in the bedroom. Likewise, with her present husband she has done that." Although Ms. Sullivan stated that on perhaps six or seven occasions when her daughter got scared, the child had slept in the bed with Linda Sullivan and Robbie Jordan, Linda never acknowledged having sex in the child's presence with any of her boyfriends. Mr. Jordan, however, testified that he and Linda had actually had sex about seven or eight times while her daughter was in the bed with them. In addition to whatever history Linda actually had with Robbie, she admitted that her daughter spends some nights sleeping on a pallet on the floor by Linda's and Eddie Wayne's bed, although both adults denied having sex while the child was in the bedroom. We do not know what the chancellor concluded, but it is evident he made no finding that sexual conduct occurred while the daughter was in the room.
¶ 21. Since the chancellor was so adamant that he changed custody because of live-in relationships with three different men, only the last of which had the "benefit of marriage" and that perhaps only because of the custody proceedings, we find his changing of custody to be unsupported by current case law. We therefore reverse and remand. What the supreme court has held is that the relationships themselves are not "per se" a basis to deny or change custody. Should the chancellor find truth in other charges, including the obviousness of sexual acts, or other matters unrelated to the live-in boyfriends, then those could be the additional acts necessary to change custody.
¶ 22. Though we are reversing, we find that it is in the child's best interest to maintain the father's custody until the further proceedings are held. Case v. Stolpe, 300 So.2d 802, 804 (Miss.1974) (not in child's best interest "to be shifted from parent to parent").

Issue 2: Attorney's fees.
¶ 23. The awarding of attorney's fees in a domestic relations matter rests with the trial court's sound discretion and will not be disturbed on appeal absent manifest error or abuse of that court's discretion. Suess v. Suess, 718 So.2d 1126, 1129 (Miss.Ct.App.1998). Fees were not awarded here.
¶ 24. We are pointed by Ms. Sullivan to Gregg v. Montgomery, as support for a finding that she was entitled to attorney's fees. 587 So.2d 928 (Miss.1991). Gregg can certainly be read to support such a conclusion.
When the husband files a petition against his former wife in which he is *519 unsuccessful, he is generally assessed with attorney's fees. This Court has said that when a former husband brings his former wife into court:
[s]eeking, without justification, alteration of his liability to her from a Court decree fixing it, [he] should pay for her an attorney's fee. Otherwise, he could sue her so often as to impose an oppressive burden on her allowance in resisting his repeated applications.
Id. at 934 (quoting Cumberland v. Cumberland, 564 So.2d 839, 845 (Miss.1990)).
¶ 25. Since Ms. Sullivan has so far been the unsuccessful party, she would not be entitled to attorney's fees. However, if after remand the chancellor determines to return custody of the child to her, then he can reconsider the issue of fees.
¶ 26. Since we are not holding that custody necessarily should be returned to Ms. Sullivan, but only concluding that based on the findings made by the chancellor a change has not been justified, we order that the custody that has been maintained during the pendency of this appeal continue until the chancellor has had an opportunity to address the case on remand.
¶ 27. THE JUDGMENT OF THE SIMPSON COUNTY CHANCERY COURT IS REVERSED AND REMANDED ON THE ISSUE OF CHILD CUSTODY AND ATTORNEY'S FEES FOR FURTHER PROCEEDINGS CONSISTENT WITH THIS OPINION. COSTS OF THIS APPEAL ARE ASSESSED TO THE APPELLEE.
McMILLIN, C.J., KING, P.J., BRIDGES, COLEMAN, DIAZ, IRVING, LEE, PAYNE, AND THOMAS, JJ., CONCUR.
