here are my comments:
What should VEDIT do with many(?, eventually more than one) input
files but only one output file?
I think your line is expanded to (as example, with 3 files in the
vpw.exe -y -X ebcdic-3.vdm file1 file2 file3 -A MEX.RPT MEX.lay
and is interpreted as:
OPEN 4 files: file1 file2 (file3 -A MEX.RPT) MEX.lay
and save the third one with the name MEX.RPT.
What do you want "ebcdic-3.vdm" to do? Maybe that:
Work with two input files: one data and one layout file.
Save the data file after a conversion or so as MEX.RPT then.
If so you have to specify no wildcards but the exact input file name.
Maybe WILDFILE.VDM or a DOS batch script can help you further...
You see: it's not easy to help if one has to guess the details.