Setprop message are activated when a custom property is modified. If you don't explicity modify it in the SetProp or pass the message, you will not modify the custom property. A setprop must be defined outside any message. Example:

setProp myCustomProperty NewValue
 answer "Watch out! You are modifying myCustomProperty to " & NewValue
 pass myCustomProperty 
end myCustomProperty

setProp myCustomProperty2 NewValue
 set myCustomProperty2 of me to NewValue + 2
end myCustomProperty2

on MouseUp
 set the myCustomProperty of me to 11
 set the myCustomProperty2 of me to 12
 answer "Cutomproperty is " & the myCustomProperty of me & return & "Cutomproperty2 is " & the myCustomProperty2 of me 
end MouseUp

Notice that you can use even not existent custom properties, they will be created automatically when set the respective custom property.