Summary and additional thoughts
Part 1: Why did God not get rid of Lucifer when he rebelled? -> To get rid of evil forever, while still keeping free will.
Before Genesis and before the Earth was created, Lucifer chose to rebel against God.
One third of the angels sided with Lucifer, which indicates that two-thirds did not. The angels had a choice to make. If God had finished off Lucifer and his angels at once, His loyal angels might think that God is a tyrant and maybe Lucifer would have made a better leader than God. They would also feel compelled to follow God out of fear, rather than love. That's why God is letting his angels see how unfit Lucifer is and what a mess he makes on planet Earth in contrast to all that Lucifer boasted.
This planet has become a testing ground where the character of Satan and the nature of his kingdom are contrasted with the character of God and the nature of His kingdom.
God decreed a set amount of time for Satan to rule the earth. It is not an unending stretch of time. It has almost been 6000 years since the beginning of creation. And Satan’s time is almost up. Even the demons know they have a set amount of time and an appointed end. When Jesus cast the demons out of the demon possessed man, the demons shouted, “What do you want with us son of God? Have you come here to torture us before the appointed time? The demons knew this was not the "appointed time" in history for Jesus to overthrow their rule and cast them into the Lake of Fire.
It has taken time for Everyone (man and spiritual beings) to really grasp how disastrous Satan's alternative is. “Time” is the one element that is absent from the Epicurean Paradox. There is a coming end to the existence of evil.
After everyone understands the deadly nature of sin and the destructive nature of Satan's philosophy, God can, and will, destroy Satan and sin. He will also have to destroy those who stubbornly resist his grace and cling to Satan's alternative. God is just as eager to resolve the problem of sin and suffering as we are to have him do so, but He is waiting until He can do it on a permanent basis and until He can both preserve our free will and prevent evil from ever appearing again. From the beginning of Genesis, God set a decreed the amount of time Satan would have to rule this world. God cannot simply stop everything before that time is complete. He is fair and just. However, God will make sure that the plan of salvation, our escape from Satan’s domain, is always available. (This was the reason for Noah’s Flood – the blood line of the Messiah was in danger, and so God had to step in earlier than the decreed time. Only when the plan of salvation is threatened does God intervene in history to stop evil.) From the very first word in the Bible, even before sin had entered the world, God had a plan for us to escape from the evil that was coming. He knew what would happen to a world ruled by Satan. The first Hebrew word in the Bible translates into English as “In the beginning”. The Hebrew letters of this word mean: “Son of God”, “destroyed”, ” hand”, and “cross.” Basically the message of the entire Bible.
We each play a part in the controversy between Christ and Satan. As the controversy ends, every soul will fully understand the principles of both kingdoms and will have chosen to follow either Christ or Satan.
Part 2: Why create beings, such as Lucifer, with the ability to do wrong? -> To create “Love”
In order for “love” to exist, there must be two independent beings who choose to love one another. So in order to create “love”, God had to create beings with the choice to love or not to love. Without choice, there could be no love.
At the end of time, only those who have chosen freely to follow God, and chosen to love Him, will live in the new Earth.
Maybe this is how "love" is created, without being forced or robotic.
And this is how free will can continue to exist without the presence of evil.
Part 3: Why do humans have to be part of this test? -> To create a Bride for Yahshua
Why are we caught in the middle of this? Maybe there is something greater that awaits us because we have endured this test. No angels will become the Bride of Christ. Angels long to look into these things – our salvation which is not only for attaining eternal life, but most importantly, to become the Bride of Christ. The ultimate expression of love between two people is marriage; thus the marriage of the Bride to Yahshua will happen at the end of this testing time.
And maybe God also wanted to create “faith’ and “hope”. Angels cannot experience these because they are in the presence of God and thus do not need either hope or faith.
If we see something, we do not need faith that it exists; if we have obtained something, then we do not need to hope for it.