This is nothing but Windows Authentication. If any web application is created with Classic Mode Authentication then it cannot applicable to configure forms based authentication. But, to configure this application to use Forms Based Authentication then you can convert from Classic Mode to Claims Based.
But, there is no UI exist for doing this conversion. The only way around is through PowerShell.
$app = get-spwebapplication "URL"
$app.useclaimsauthentication = "True"
But, if an application is created using Claims based and if you want to convert from Claims based to Classic Mode then it is not possible either through UI or through PowerShell. So, remember this.
In SharePoint 2010 for a web application we can enable both windows, forms authentication. In earlier implementation to do this, we have to create two web applications which has different zones and different authentication. But, with the new claims based authentication a single application can have the capability to configure both windows and forms under single URL. All this is possible because of the authentication framework is built on Microsoft Identify Foundation. And it uses “Geneva” framework to handle this authentication.