Element is pushed away in list move transition when content is scrolled - vue.js

Question

I'm trying to display a one line list of elements.
This list is browsed using the scrollbar.
Each element can be removed on click with a transtion making it go up and fade.
I came up with the following code:
new Vue({
el: '#container',
data: { blocks: [1, 2, 3, 4, 5, 6, 7] }
})
#container {
position: relative;
height: 100px;
width: 200px;
border: 1px solid black;
overflow-x: scroll;
}
.block {
position: absolute;
bottom: 0px;
height: 40px;
width: 40px;
background-color: blue;
cursor: pointer;
}
.rise-fade-leave-active,
.rise-fade-move {
transition: all 0.5s ease;
}
.rise-fade-leave-to {
transform: translateY(-40px);
opacity: 0;
}
<script src="https://unpkg.com/vue#2.6.11/dist/vue.min.js"></script>
<div id="container">
<transition-group name="rise-fade" tag="div">
<div v-for="(b, i) in blocks"
v-bind:key="b"
v-bind:style="{ left: i*50 + 'px' }"
v-on:click="blocks.splice(i, 1)"
class="block">
</div>
</transition-group>
</div>
But if I scroll to the most right and click on any visible element except the last one, the element seems to be pushed to the right during the list move transition.
Is there a way to prevent this?
I would like to have my element rising just vertically no matter the scroll value.