Update: The workaround below isn't needed if you use the Bootstrap4Xpage plugin: you can use the standard Dialog control from
the
Extension Library:
Bootstrap has a component to create a modal dialog, but it doesn't play well with XPages. The Extension Library has a modal
dialog component that plays well with XPages, but doesn't look like Bootstrap.

Luckily, by changing some properties and little jQuery magic, we can let thexe:dialog controlfrom the Extension Library look like a Bootstrap dialog, so you'll have the best of both. Here's the sample code.
<xe:dialog id="dialog1" title="Dialog title" styleClass="modal" style="margin-left: inherit;">
<xe:this.onShow>
<![CDATA[//make the dialog look like a Bootstrap dialog using some jQuery magic
//add the modal-header class
to the title bar
var titleBar = $(".modal .dijitDialogTitleBar").addClass("modal-header");
//replace title node (by default it's a span)
var titleNode = $(".dijitDialogTitle", titleBar);
var title = titleNode.text();
titleNode.remove();
//add a class to the close icon
$(".dijitDialogCloseIcon", titleBar).removeClass("dijitDialogCloseIcon").addClass("close");
//add a new h3 node with the title
titleBar.append("<h3>" + title + "</h3>");
]]>
</xe:this.onShow>
<xe:dialogContent id="dialogContent1"
styleClass="modal-body">
dialog content here
</xe:dialogContent>
<xe:dialogButtonBar id="dialogButtonBar1"
styleClass="modal-footer">
<xp:button value="Close" id="button2"></xp:button>
</xe:dialogButtonBar>
</xe:dialog>
The dialog shown on this page is the standard Dialog
component from the Extension Library. It has been
configured to reflect a
Bootstrap style dialog.
Show dialog