Tuning filters to refine Ads eligibility to surface in search results emerges as a pivotal problem. It often necessitates a nuanced approach to cater to diverse requirements from the customers. Adjusting these filters must judiciously balance the preferences of both advertisers and users in the online marketplace. Hence, it requires a multi-objective optimization which often turns out to be hard due to the conflicting nature of the objectives from these customers. In this paper we present AFA: Auto-tuning Filters for Ads - a novel application of Bayesian Optimization for auto-tuning these filters. We specifically develop AFA to employ a probabilistic model to navigate the intricate trade-offs between multiple objectives. It iterates over a feasible solution space and quickly converges to an operating point which ensures showing well performing ads while increasing their scale. This offers a substantial advancement in the automation for digital advertising campaigns. Our approach significantly reduces the reliance on manual adjustments and expensive A/B testing, as demonstrated by empirical results from a large-scale e-commerce platform.