Energia button debounce interrupt

Sorry if this has been asked and answered before, but the results from searches didn't explain the reason not working on my device and code..

Basically trying to do a very simple debounce in an interrupt for an Energia code sketch that paint a 1602 LCD:

const int buttonPin = PUSH1;     // the number of the pushbutton pin
int buttonState;             // the current reading from the input pin
volatile int lastButtonState = LOW;   // the previous reading from the input pin
volatile long lastDebounceTime = 0;  // the last time the output pin was toggled
volatile long debounceDelay = 500;    // the debounce time; increase if the output flickers


void setup() {
...
  pinMode(buttonPin, INPUT_PULLUP);

  attachInterrupt(buttonPin, switchDisplayMode, FALLING);
}

...


void switchDisplayMode() {
  int reading = digitalRead(buttonPin);
  if (reading != lastButtonState) {
    // reset the debouncing timer
    lastDebounceTime = millis();
  } 
  if ((millis() - lastDebounceTime) > debounceDelay) {
    // whatever the reading is at, it's been there for longer
    // than the debounce delay, so take it as the actual current state:
    buttonState = reading;
    
    // start
    displayMode++;
    if (displayMode > 5) {
      displayMode = 0;
    }
//    delayMicroseconds(6000000);
    refreshDisplay();
    // end
    lastButtonState = reading;
  }
}

In the above code, debounce did not work all the times, like 3 in 5 times it seem to work, the rest are not. The displayMode variable incremented twice for each button clicked event. The target device is F5529 LP.

I have tried the delayMicroseconds which I believed to be irrelevant to the debounce logic if implemented correctly, but that doesn't seem to do the trick either.. Am I doing it wrongly? Any advice will be much appreciated. Thanks in advance.