Combine cubes of goat, chilli pepper, curry powder, garlic, salt and black pepper in a bowl. Cover and refrigerate to allow flavours to blend, 1 hour to overnight.
Remove goat meat mixture from bowl and pat dry, reserving marinade. Heat vegetable oil in a large pot over medium-high heat. Cook meat in batches, browning on all sides, 5 to 6 minutes per batch. Transfer meat to a plate. Add onion and celery to the pot; cook and stir until onion begins to brown, 4 to 6 minutes.
Stir browned goat meat into onion mixture. Add reserved marinade, vegetable stock and bay leaf. Bring to the boil, cover, reduce heat to low, and simmer for 1 hour. Stir in potatoes; simmer until potatoes and meat are tender, 35 to 45 minutes more.
Remove pot from heat, skim off surface fat and remove bay leaf.