The absolute maximum voltage on any pin is <VDD+0.3V, the maximum VDD is 3.6V. If you have supplied 5V to an input, then you may have damaged the chip.
If you use internal reference as reference (0.6V), then the measured signal must be between 0V and 0.6V. You have a gain control register that can divide the input voltage by 1/6 to for instance support up to 3.6V analog input.
Example:
If you measure on a 3V input signal, using internal reference (0.6) and 1/6 gain, then the measurement should be about (3/6)/0.6 = 0.833 of the max ADC value. The max ADC value depend on how many bit resolution you have configured, but for instance with 10-bit resolution the max value is 2^10 = 1024. So in that case the ADC measurement will be 0.833 * 1024 = 853 decimal value when a 3V input is applied.