const vs non-const of container and its content

With a container class, such as std::vector, there are two different concepts of constant-ness: that of the container (i.e. its size) and that of the elements. It seems that std::vector confuses these two, such that the following simple code won't compile:

struct A {
  A(size_t n) : X(n) {}
  int&x(int i) const { return X[i]; }    // error: X[i] is non-const.
private:
  std::vector<int> X;
};

Note that even though the data members (the three pointers to the begin&end of the data and the end of the allocated buffer) of std::vector are not changed by a call to its operator[], this member is not const -- isn't this a strange design?

Note also that for raw pointers, these two concepts of constant-ness are neatly separated, such that the corresponding raw-pointer code

struct B {
  B(size_t n) : X(new int[n]) {}
  ~B() { delete[] X; }
  void resize(size_t n);                 // non-const
  int&x(int i) const { return X[i]; }    // fine
private:
  int*X;
};

works just fine.

So what is the correct/recommended way to deal with this when using std::vector (without using mutable)?

Is a const_cast<> as in

int&A::x(int i) const { return const_cast<std::vector<int>&>(X)[i]; }

deemed acceptable (X is known to be non-const, so no UB here)?

EDIT just to prevent further confusion: I do want to modify the elements, i.e. the contents of the container but not the container itself (the size and/or memory location).