The preferred way of changing the thumbnail resolution and quality is by changing the appropriate values in the TranscodePresetDocument in a shape tag.

To adjust resolution;

<thumbnailResolution>
<width>1200</width>
<height>800</height>
</thumbnailResolution>
CODE

To adjust quality;

<setting>
 <key>thumbnailcompression</key>
 <value>75</value>
</setting>
CODE

Where the value 75 is default and lower value gives you a more reduced quality.

NOTE: Quality adjustment is only supported for images.

More information on Thumbnails;

http://apidoc.vidispine.com/latest/ref/item/thumbnail.html?highlight=thumbnail