You should be able to connect to an Esri MapServer directly (i.e. without going through the WMS interface) using ArcGisMapServerImageryProvider. What problem did you run into? The Imagery Layers Manipulation example in Sandcastle includes a MapServer layer:
I've couple of ESRD shape file with ArcGIS LYRs. I want to visualize them in Cesium. I tried ArcGisMapServerImageryProvider, but it requires some URL of mapserver. Not sure what is that.
I've .lyr, .shp files etc.