A value function in machine learning estimates the expected total reward an agent can achieve starting from a specific state or by taking a particular action, considering future rewards. It guides decision-making in reinforcement learning by helping the agent identify the most beneficial actions, thereby optimizing its policy to maximize cumulative rewards over time.