I use Filezilla as my FTP client to connect to my server. Here is a quick video showing you how to connected to your server to manage your wordpress files 

Once you find and delete the new plugin folder (or file) it will reset the site and automatically deactivate the plugin.

If you are updating an existing plugin and the site breaks, follow the above procedure by deleting the plugin from your wordpress wp-content/plugins folder and then re-upload the older working version of the plugin you were using.

You can see all the old versions of a wordpress plugin in the wordpress repository.

Once you upload the older version of the plugin back to your site, you can simply head into the plugins settings inside of wordpress and activate the older version.

The plugin will continue to prompt you to upgrade, DO NOT DO THAT. (unless you want to go through these steps again)

Until the conflicting plugin/code is fixed, you are going to want to hold off on upgrading it.

Typically plugin updates are done to add features, but if the update is concerning a security issue, you will want to consult with a wordpress ninja (or the plugin developer) to get the issue resolved.

This post is purely meant to get your site back up and running as quickly as possible.

To troubleshoot exactly where and what is causing the conflict lies you will want to consult this post.

I also highly recommend you have daily/weekly backups running on your server and via a backup plugin,  just in case all hell breaks lose and you completely fry your wordpress installation.

Getting a fresh wordpress installation up and running is a pretty quick and simple process but without a backup of your database and your wp-content folder, you could be in trouble.